Resque-brokered is an effort at allowing you to inject different strategies for picking resque jobs off queues. It was initial created to meet the requirement of having workers reading from multiple queues, but only having one job from each queue being worked on at any one time.
If you're using bundler, then something like this will be required in your Gemfile:
gem 'resque-brokered'
If you're not using bundler, then you should take a good hard look at yourself, then if you're still not using bundler, then you'll need to do something like this:
$ gem install resque-brokered
Then somewhere in your code (before you do any queueing stuff):
require 'resque'
require 'resque-brokered'
Brokered queues are defined by having a two-part queue name, the
group-name, then the queue name. Thus you could have a queue system with
queues big_process:user1, big_process:user2 ... big_process:usern then you
can have as many big_process workers as you need to cater to your user
base, but only handle one big_process per user at any one time.
To start a worker to consume from all the big_process queues, simply
specify the first part of the queue name like so:
QUEUES=big_process: rake resque:work
Note the colon. It's the special sauce. The worker you just started will consume
from all the queues which start with big_process:. If you start more workers with
the same QUEUES environment variable, they will do the same, but they won't pick jobs
up from queues which are already active.
Normally named queues and queues without colons work as per usual. It's
suggested that you don't use the catch-all QUEUES=* in conjunction
with resque-brokered as that will ignore all the limiting and
consistency work which resque-brokered brings.
Right now resque-brokered only supports one strategy, so it isn't really a broker in the truest sense. Moving forward, we would like to support additional strategies, such as weighted/priority queues, rate limiting and more advanced concurrency limitation.
We also need to improve our test coverage and documentation
James Sadler and Andrew Harvey
You know the deal:
- Fork the project
- Write code with tests
- Submit a pull request
- ???
- Profit.