Delayed Job Heartbeat Plugin

By default Delayed Job uses the ClearLocks plugin to unlock jobs when a worker shuts down. Unfortunately this only works if a worker shuts down cleanly. If the worker crashes, the job won't be unlocked until max_run_time elapses which can cause unacceptable delays processing background jobs. Enter the Delayed Job Heartbeat Plugin...

The Delayed Job Heartbeat Plugin adds a heartbeat to all Delayed Job workers. After a configurable timeout jobs from unresponsive workers can be unlocked either via a Ruby API or a rake task.

Installation

Add this line to your application's Gemfile:

gem 'delayed_job_heartbeat_plugin'

And then execute:

$ bundle

Or install it yourself as:

$ gem install delayed_job_heartbeat_plugin

Run the required database migrations:

$ rails generate delayed_job_heartbeat_plugin:install
$ rake db:migrate

Usage

There are two parts to using the Delayed Job Heartbeat Plugin:

Configuring the worker's heartbeat options.
Periodically unlocking jobs from unresponsive workers.

Worker Heartbeat Options

The worker heartbeat can be configured in an initializer (e.g. config/initializers/delayed_job_heartbeat.rb) follows:

Delayed::Heartbeat.configure do |configuration|
  configuration.enabled = Rails.env.production?
  configuration.heartbeat_interval_seconds = 60
  configuration.heartbeat_timeout_seconds = 180
  configuration.worker_termination_enabled = true
end

The plugin supports the following options (all of which are optional):

enabled - enables/disables the plugin entirely (defaults to true in production and false in other environments)
worker_label - a label for the worker. Consider setting this to ENV['DYNO'] if running in Heroku to get the dyno's friendly name (defaults to the Delayed Job worker's name)
heartbeat_interval_seconds - how often workers should send a heartbeat (defaults to 60 seconds)
heartbeat_timeout_seconds - theshold after which workers are considered dead if they haven't heartbeated (defaults to 180 seconds)
worker_termination_enabled - controls whether a worker that detects it has not heartbeated within the timeout period (e.g. due to severe memory swapping) should shut itself down (defaults to true in production and false in other environments)
on_worker_termination - a callback proc that accepts a Delayed::Heartbeat::Worker and an Exception if the heartbeat fails. This can be useful for reporting to an error monitoring system.
worker_version - a version number of the worker's source code that is only used if you want to cleanup workers from old source code versions (defaults to nil)

Unlocking Unresponsive Workers

Jobs from unresponsive workers can be unlocked by calling Delayed::Heartbeat.delete_timed_out_workers or running the delayed:heartbeat:delete_timed_out_workers rake task. This should be integrated with a scheduler like cron, clockwork or the Heroku Scheduler (if running in Heroku).

Jobs from workers running an old version of the source code can be unlocked by calling Delayed::Heartbeat.delete_workers_with_different_version or running the delayed:heartbeat:delete_workers_with_different_version rake task.

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/salsify/delayed_job_heartbeat_plugin.

License

The gem is available as open source under the terms of the MIT License.