Sidekiq::Status

An extension to Sidekiq message processing to track your jobs. Inspired by resque-status and mostly copying its features, using Sidekiq's middleware.

Installation

Add this line to your application's Gemfile:

gem 'sidekiq-status'

And then execute:

$ bundle

Or install it yourself as:

gem install sidekiq-status

Usage

Configuration

Configure your middleware chains, lookup Middleware usage on Sidekiq wiki for more info.

require 'sidekiq'
require 'sidekiq-status'

Sidekiq.configure_client do |config|
  config.client_middleware do |chain|
    # accepts :expiration (optional)
    chain.add Sidekiq::Status::ClientMiddleware, expiration: 30.minutes # default
  end
end

Sidekiq.configure_server do |config|
  config.server_middleware do |chain|
    # accepts :expiration (optional)
    chain.add Sidekiq::Status::ServerMiddleware, expiration: 30.minutes # default
  end
  config.client_middleware do |chain|
    # accepts :expiration (optional)
    chain.add Sidekiq::Status::ClientMiddleware, expiration: 30.minutes # default
  end
end

After that you can use your jobs as usual and only include Sidekiq::Status::Worker module if you want additional functionality of tracking progress and passing any data from job to client.

class MyJob
  include Sidekiq::Worker

  def perform(*args)
  # your code goes here
  end
end

To overwrite expiration on worker basis and don't use global expiration for all workers write a expiration method like this below:

class MyJob
  include Sidekiq::Worker

  def expiration
    @expiration ||= 60*60*24*30 # 30 days
  end

  def perform(*args)
    # your code goes here
  end
end

But keep in mind that such thing will store details of job as long as expiration is set, so it may charm your Redis storage/memory consumption. Because Redis stores all data in RAM.

What is expiration time ?

As you noticed you can set expiration time for jobs globally by expiration option while adding middleware or writing a expiration method on each worker this expiration time is nothing but

Redis expire time, also know as TTL(time to live)
After expiration time all the info like status, update_time etc. about the worker disappears.
It is advised to set this expiration time greater than time required for completion of the job.
Default expiration time is 30 minutes.

Retrieving status

Query for job status any time later:

job_id = MyJob.perform_async(*args)
# :queued, :working, :complete, :failed or :interrupted, nil after expiry (30 minutes)
status = Sidekiq::Status::status(job_id)
Sidekiq::Status::queued?      job_id
Sidekiq::Status::working?     job_id
Sidekiq::Status::complete?    job_id
Sidekiq::Status::failed?      job_id
Sidekiq::Status::interrupted? job_id

Important: If you try any of the above status method after the expiration time, will result into nil or false

Tracking progress, saving, and retrieving data associated with job

class MyJob
  include Sidekiq::Worker
  include Sidekiq::Status::Worker # Important!

  def perform(*args)
    # your code goes here

    # the common idiom to track progress of your task
    total 100 # by default
    at 5, "Almost done"

    # a way to associate data with your job
    store vino: 'veritas'

    # a way of retrieving said data
    # remember that retrieved data is always String|nil
    vino = retrieve :vino
  end
end

job_id = MyJob.perform_async(*args)
data = Sidekiq::Status::get_all job_id
data # => {status: 'complete', update_time: 1360006573, vino: 'veritas'}
Sidekiq::Status::get     job_id, :vino #=> 'veritas'
Sidekiq::Status::at      job_id #=> 5
Sidekiq::Status::total   job_id #=> 100
Sidekiq::Status::message job_id #=> "Almost done"
Sidekiq::Status::pct_complete job_id #=> 5

Unscheduling

scheduled_job_id = MyJob.perform_in 3600
Sidekiq::Status.cancel scheduled_job_id #=> true
# doesn't cancel running jobs, this is more like unscheduling, therefore an alias:
Sidekiq::Status.unschedule scheduled_job_id #=> true

# returns false if invalid or wrong scheduled_job_id is provided 
Sidekiq::Status.unschedule some_other_unschedule_job_id #=> false
Sidekiq::Status.unschedule nil #=> false
Sidekiq::Status.unschedule '' #=> false
# Note: cancel and unschedule are alias methods.

Important: If you try any of the status method after the expiration time for scheduled jobs, will result into nil or false. But job will be in sidekiq's scheduled queue and will execute normally, once job is started on scheduled time you will get status info for job till expiration time defined on Sidekiq::Status::ServerMiddleware.

Deleting Status by Job ID

# returns number of keys/jobs that were removed
Sidekiq::Status.delete(job_id) #=> 1
Sidekiq::Status.delete(bad_job_id) #=> 0

Sidekiq web integration

Sidekiq::Status also provides an extension to Sidekiq web interface with a /statuses.

Setup Sidekiq web interface according to Sidekiq documentation and add the Sidekiq::Status::Web require:

require 'sidekiq/web'
require 'sidekiq-status/web'

Testing

Drawing analogy from sidekiq testing by inlining, sidekiq-status allows to bypass redis and return a stubbed :complete status. Since inlining your sidekiq worker will run it in-process, any exception it throws will make your test fail. It will also run synchronously, so by the time you get to query the job status, the job will have been completed successfully. In other words, you'll get the :complete status only if the job didn't fail.

Inlining example:

You can run Sidekiq workers inline in your tests by requiring the sidekiq/testing/inline file in your {test,spec}_helper.rb:

require 'sidekiq/testing/inline'

To use sidekiq-status inlining, require it too in your {test,spec}_helper.rb:

require 'sidekiq-status/testing/inline'

Contributing

Bug reports and pull requests are welcome. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.

Fork it
Create your feature branch (git checkout -b my-new-feature)
Commit your changes along with test cases (git commit -am 'Add some feature')
If possible squash your commits to one commit if they all belong to same feature.
Push to the branch (git push origin my-new-feature)
Create new Pull Request.

Thanks

Pramod Shinde
Clay Allsopp
Andrew Korzhuev
Jon Moses
Wayne Hoover
Dylan Robinson
Dmitry Novotochinov
Mohammed Elalj
Ben Sharpe

cuongn265/sidekiq-status