BBQue

BBQue is an opinionated ruby gem to queue and process background jobs. Other gems for this purpose usually don't work with ruby objects, serialize method arguments only and/or have strong reliability options only in paid/pro versions.

BBQue jobs are simple ruby objects:

MyQueue.enqueue MyJob.new

BBQue jobs need to fulfill the following interface:

The object contains an instance method #work without any arguments
The object (instance) must be serializable via Marshal.dump and Marshal.load

Installation

Add this line to your application's Gemfile:

gem 'bbque'

And then execute:

$ bundle

Or install it yourself as:

$ gem install bbque

Usage

BBQue is using Redis. Therefore, you have to install Redis first.

Producer

To enqueue a job, you need a Producer instance:

SomeQueue = BBQue::Producer.new("default")

where default is the queue name. You can additionally pass a delay, priority (-512 ... 512, higher is better, default is 0), logger and/or the Redis instance:

SomeQueue = BBQue::Producer.new("default", delay: 2.hours, pri: 20, redis: Redis.new, logger: Logger.new(...))

Then enqueue a job:

SomeQueue.enqueue MyJob.new("Some argument")

where MyJob looks like:

class MyJob
  attr_accessor :argument

  def initialize(argument)
    self.argument = argument
  end

  def work
    # Do some work
  end
end

Consumer

To process the enqueued jobs, create a file, e.g. jobs.rb:

BBQue::Consumer.new("default").run

and run it via:

$ bbque jobs.rb

BBQue will loop through all the jobs, run them, and will then wait for new ones (no polling). You can pass the queue name, the Redis instance and a logger.

BBQue::Consumer.new(["default", "important"], logger: ..., redis: ...).run

Forking Consumers

By default, BBQue does not fork a process for every job, but it already ships with the ability to do so.

To make BBQue fork, create a custom consumer:

class ForkingConsumer < BBQue::Consumer
  def fork?
    true
  end

  def before_fork
    # ...
  end

  def after_fork
    # ...
  end
end

Then, start your custom consumer in favor of BBQue's default one:

ForkingConsumer.new(["queue1", "queue2", ...], ...).run

Job Limit

You can specify a limit for a job, such that no more than limit instances of the same job can be concurrently enqueued. You need to pass a job_key as well, such that BBQue can determine duplicate jobs:

MyQueue.enqueue MyJob.new, limit: 1, job_key: "my_singleton_job"

Delay/Scheduler

You can enqueue jobs while specifing a delay:

MyQueue.enqueue MyJob.new, delay: 60

When using delay, jobs are added to a special queue, such that you need to start a scheduler process, which regularly checks for delayed jobs, checks whether the delay has passed and subsequently moves the jobs from the delay queue to its target queue.

BBQue::Scheduler.new(redis: Redis.new, logger: Rails.logger).run

Safety

Regarding Redis issues, BBQue implements job safety measures similar to sidekiq pro's super_fetch. BBQue is using BRPOPLPUSH and when a job is fetched, it is added to a processing queue, such that lost jobs get rescued on worker restart.

BBQue, howoever, doesn't offer any retry mechanisms for failed jobs, ie jobs raising exceptions. You are responsible for rescuing exceptions within your jobs and taking appropriate measures.

Graceful Termination

You can stop a worker gracefully by sending a QUIT, TERM, INT or USR2 signal to it. The worker will finish its current job and terminate afterwards.

Contributing

Fork it ( https://github.com/mrkamel/bbque/fork )
Create your feature branch (git checkout -b my-new-feature)
Commit your changes (git commit -am 'Add some feature')
Push to the branch (git push origin my-new-feature)
Create a new Pull Request

mrkamel/bbque