squaremo/rabbit.js

How best to implement distributed semaphores?

Opened this issue · 3 comments

gRoberts84 commented

I'm currently trying to find the best way of structuring my project and I'm currently using Rabbit.js to pub/sub a queue to pass jobs between processes.

We have a process that hosts a web application which allows our users to initiate a process. Upon clicking "Go", the process parameters are inserted into the database and then a message is queued.

Now it gets complicated... We have a process that is clustered, which listens to the queue and initiates a connection to a data stream. As each stream is unique, we only want to initiate it once and if that stream dies for any reason, a message is automatically replaced in the queue until it is removed or expires.

Something similar to: https://www.rabbitmq.com/blog/2014/02/19/distributed-semaphores-with-rabbitmq/

Using Rabbit.js, is it possible to implement this?

Thanks

gRoberts84 commented

Ok, sorry I should have RTFM...

PUSH / WORKER: a WORKER socket is similar to a PULL socket, but requires that you call #ack on it to acknowledge that you have processed each message. Any messages left unacknowledged when the socket closes, or crashes, will be requeued and delivered to another connected socket (should there be one).

D'oh. Sorry all.

squaremo commented

Yes, that ought to do it. Beware of relying on RabbitMQ for locking though, as noted in the edits at the end of that blog post. Something like zookeeper may be more reliable, if you need strong consistency guarantees.

gRoberts84 commented

Hey Michael,

Thanks for getting back to me and sorry for the delay, I've been trying to find the time to work on this around all of my other work. Fun huh.

Still not sure how best to tackle this... At first I thought PUSH/WORKER was the best solution, however now looking at it, it "may" be REQ/REP.

Ultimately I need a non-blocking way of ensuring that only one recipient processes a message at a time and should that recipient fail, the message needs requeuing to allow another recipient to pick it up.

Just to explain how I've done things so far:

  1. REST Server sends PUSH message to Address 1 to initiate data stream.
  2. DATA Connector receives PUSH message on Address 1 and creates a new data stream.
  3. DATA Connector receives data from data stream and sends PUSH message to Address 2 with data.
  4. QUEUE Processor receives PUSH message on Address 2, processes and acknowledges message.

Currently 3 won't receive any messages until 1 is acknowledged, however by acknowledging it, should the process fail after acknowledging it, it won't be requeued.

I still think message queuing is the best solution, it's just more how to implement it.

Do you have any suggestions on how best to use RabbitMQ/Rabbit.js to support this, or anything else for that matter?

Thanks again

Gavin