Big queue (BQ) is a multi-consumer topic system designed to be very simple and very scalable, the main idea of is that each topic can be consumed asynchronously by many consumer groups, we’ll call consumer group to a pool of process which will receive the same bunch of messages, using this method we can send a group of messages to a topic and each consumer group will receive this messages (like a kestrel fanout queue) then we can read it in a paralelized fashion using many process reading from the same consumer group
BQ is based on top of redis http://redis.io/ as their main storage (in memory) but could be implemented over any persistent system
One of the very atractive things of BQ is the idea of be asynchronous this enable us to create an http api (could be any other faster protocol if we want) that implements a similar Amazon SQS comunication pattern being very easy integrate and platform/languaje agnostic
curl -d'{"name":"test_topic"}' 127.0.0.1:8081/topics -H"Content-Type: application/json"
curl -d'{"name":"test_consumer"}' 127.0.0.1:8081/topics/test_topic/consumers -H"Content-Type: application/json"
curl -XPUT 127.0.0.1:8081/topics/test_topic/consumers/test_consumer -H"Content-Type: application/json"
curl -d'{"msg":{"test":"message"},"topics":["test_topic"]}' 127.0.0.1:8081/messages -H"Content-Type: application/json"
The rest interface can be runned using the bigqueue-http command installed at the module install time, this command will run in default mode (using a pre instaled redis at default port), if you want to configure it to run in cluster mode you could use the config example at example/clusterconfig.js
If you want to run with the cluster configuration you should run
./node_modules/.bin/bigqueue-http ./node_modules/BigQueue/examples/clusterconfig.js