joealcorn/huey

a little task queue for python

huey - a little task queue

a lightweight alternative.

• written in python (2.7+, 3.4+)
• optional dependency on the Python Redis client

supports:

• multi-process, multi-thread or greenlet task execution models
• schedule tasks to execute at a given time, or after a given delay
• schedule recurring tasks, like a crontab
• retry tasks that fail automatically
• task result storage
• task locking
• task pipelines and chains

Huey's API

from huey import RedisHuey, crontab

huey = RedisHuey('my-app', host='redis.myapp.com')

@huey.task()
def add_numbers(a, b):
    return a + b

@huey.periodic_task(crontab(minute='0', hour='3'))
def nightly_backup():
    sync_all_data()

To run the consumer with 4 worker processes:

$ huey_consumer.py my_app.huey -k process -w 4

To enqueue a task to add two numbers and print the result:

res = add_numbers(1, 2)  # Enqueues task.
print(res.get())  # Prints "3".

To schedule two numbers to be added in 10 seconds:

res = add_numbers.schedule(args=(1, 2), delay=10)

# Attempt to get result without blocking.
print(res.get(False))  # returns None.

# Block until result is ready and print.
print(res.get())  # after 10 seconds, prints "3".

Brokers

To use Huey with Redis (recommended):

from huey import RedisHuey

huey = RedisHuey()

To use Huey with SQLite (docs):

from huey.contrib.sqlitedb import SqliteHuey

huey = SqliteHuey('my-app-queue.db')

To run Huey within the parent process using background greenlets (docs):

from huey.contrib.minimal import MiniHuey

huey = MiniHuey()
huey.start()  # Spawns scheduler background thread and returns immediately.

To run Huey with a simple Python broker (should not be used in production):

from huey.contrib.simple_storage import SimpleHuey

huey = SimpleHuey()

# Be sure to run the Python broker process, e.g.:
# $ python huey/contrib/simple.py  # Starts Python broker.

Documentation

See Huey documentation.

Project page

See source code and issue tracker on Github.

Huey is named in honor of my cat: