ORESoftware/live-mutex

EADDRINUSE

Closed this issue · 11 comments

I'm getting an error like this. The code below is a small example but should show the setup I'm using. I noticed around 5k/s connections I'm seeing the error in 1/5000 connections.

(node:7495) UnhandledPromiseRejectionWarning: Error: listen EADDRINUSE 127.0.0.1:6970
    at Object._errnoException (util.js:1022:11)
    at _exceptionWithHostPort (util.js:1044:20)
    at Server.setupListenHandle [as _listen2] (net.js:1367:14)
    at listenInCluster (net.js:1408:12)
    at doListen (net.js:1517:7)
    at _combinedTickCallback (internal/process/next_tick.js:141:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)
import { LMXBroker, LMXClient } from 'live-mutex';

const wrappedLog = (name: string, level = 'debug') => (...args: any[]) => (log[level] || log.debug)(name, ...args);

let mutexClient;

const getMutex = async () => {
    if (mutexClient) {
        return mutexClient;
    }

    const [broker, client] = await Promise.all([
        new LMXBroker().ensure(),
        new LMXClient().connect()
    ]);

    broker.emitter.on('warning', wrappedLog('broker', 'warn'));
    client.emitter.on('warning', wrappedLog('client', 'warn'));

    mutexClient = client;

    return client;
};

// This function will be called by multiple requests
// I've just simplified to show the issue
const main = async () => {
    // Connect to mutex broker
    const mutex = await getMutex();

    // Lock mutex
    const lock = await mutex.acquireLock('registered').catch(error => {
        log.error(error);
        server.close();
    });

    if (!lock) {
        return;
    }

    // Unlock mutex
    await mutex.releaseLock('registered', { id: lock.id });
};

main().catch(error => {
    console.error(error);
});

you shouldnt (must not) cluster the broker - there needs to be only one broker.

the docs arent that clear on this, i can make that clearer.

if you run only one broker, does the problem occur?

or maybe thats not the problem.
let me take a look at your code tomorrow, i gtg to sleep. are you using the docker version of the broker? how are you running it?

Yeah the docs seem hit and miss.

Currently I just need a setup where I can use await and get a lock, then once finished unlock it. I’m using this 100% in node. No docker. I’m also only using a single process but may need to scale out later on.

do you have a full repro? something where I can get it to 5000k/s?

so this is likely what's happening:

const lock = await mutex.acquireLock('registered').catch(error => {
    log.error(error);
    server.close();   // when a lcck cannot be acquired, it will reach here, so the server is restarted
});

I put your code here:
https://github.com/ORESoftware/live-mutex-examples/blob/master/src/dual/repro.ts

the code style in the above is probably better than what you have, which is fine, I can improve the docs

i think the main problem was calling:

   server.close();

you should only call that if you are shutting down the process.

I made a patch, it's here:

  • live-mutex@0.2.23

i fixed a bug with regard to client connection

so try your code here:

https://github.com/ORESoftware/live-mutex-examples/blob/master/src/dual/repro.ts

with version live-mutex@0.2.23

and lmk know if it works for you

@ORESoftware server.close() is the websocket connection I had open. It needs to happen if the lock isn't acquired.

So this seems to work apart from this. I've added catch handlers to all promises I'm using. I'm thinking this may be an error that's thrown in live-mutex itself?

(node:2662) UnhandledPromiseRejectionWarning: lmx client err: client connection timeout after 3000ms.
(node:2662) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 548)

Fixed the issue above, I missed the Promise.all.

I still get this though on mass connections. I'm trying to handle 5000 ws connections a second with each needing to use a mutex.

client warning: Connection was closed (and/or a client connection error occurred.)
Couldn't acquire lock for registered:nqkgk LMXClientLockException {
  id: '626bdccc-d587-47ea-9027-96e102d9cf7c',
  key: 'registered:nqkgk',
  code: 'connection_closed',
  message: 'Connection was closed (and/or a client connection error occurred.)',
  stack: 'Connection was closed (and/or a client connection error occurred.)'
}

sure can you share a full repro?

I can try but I'm not 100% sure I'll be able to. I'm going to try what you've suggested in the other issue.