bokeh (pronounced boh-kay) is a simple, scalable and blazing-fast task queue built on Node.js and ZeroMQ. It allows you to offload tasks from your main application process and distribute them among a pool of workers. Workers can be running on the same host as your application, or scaled out onto multiple machines for greater processing power.
When you want a worker to run a task, just submit it to the broker using the client API. A task is simply any class in your application which responds to the run method.
Bokeh consists of three components:
- The client library which your application uses to submit tasks to the broker.
- The broker process which manages the pool of workers.
- The worker processes which are responsible for running tasks.
The only prerequisite is that you have ZeroMQ installed.
OS X
Install ZeroMQ using brew:
$ brew install zeromq
Ubuntu 10.04 LTS
Install ZeroMQ using Chris Lea's PPA:
$ sudo add-apt-repository ppa:chris-lea/zeromq
$ sudo apt-get update
$ sudo apt-get install libzmq-dbg libzmq-dev libzmq1
Install Bokeh using npm:
$ npm install bokeh
A task is a class which responds to the run method. A task is dealt to a worker and executed.
Once the task has been completed, you must call the callback with any data you want to pass back to your application.
class Reverse
run: (data, callback) -> callback null, data.split("").reverse().join("")
The client is used by your application to submit tasks to the broker and monitor their progress.
bokeh = require "bokeh"
handle = bokeh.getClient().submitTask "Reverse", "hello world"
handle.on "complete", (data) -> console.log "Task %s completed: %s", handle.id, data
handle.on "error", (error) -> console.error "Task %s failed: %s", handle.id, error
The broker is responsible for routing messages from clients, persisting them to the data store and dealing them to workers.
bokeh = require "bokeh"
broker = new bokeh.Broker
Bokeh supports pluggable data stores for persisting tasks, currently in-memory, Redis and Riak are supported.
A worker is a process which receives tasks from a broker and executes them. You must register all your task classes with the worker.
bokeh = require "bokeh"
worker = new bokeh.Worker
worker.registerTask "Reverse", require("./tasks/reverse")
Bokeh is released under the MIT license.