celery-director: A Python repository from dmcclory

Director is a simple and rapid framework used to manage tasks and build workflows using Celery.

The objective is to make Celery easier to use by providing :

a WebUI to track the tasks states,
an API and a CLI to manage and execute the workflows,
a YAML syntax used to combine tasks into workflows,
the ability to periodically launch a whole workflow,
and many others.

See how to use Director with the quickstart and guides in the documentation.

Installation

Install the latest version of Director with pip (requires Python 3.6 at least):

pip install celery-director

Usage

Write your code in Python

# tasks/orders.py
from director import task
from .utils import Order, Mail

@task(name="ORDER_PRODUCT")
def order_product(*args, **kwargs):
    order = Order(
      user=kwargs["payload"]["user"],
      product=kwargs["payload"]["product"]
    ).save()
    return {"id": order.id}

@task(name="SEND_MAIL")
def send_mail(*args, **kwargs):
    order_id = args[0]["id"]
    mail = Mail(
      title=f"Your order #{order_id} has been received",
      user=kwargs["payload"]["user"]
    )
    mail.send()

Build your workflows in YAML

# workflows.yml
product.ORDER:
  tasks:
    - ORDER_PRODUCT
    - SEND_MAIL

Run it

You can simply test your workflow in local :

$ director workflow run product.ORDER '{"user": 1234, "product": 1000}'

And run it in production using the director API :

$ curl --header "Content-Type: application/json" \
  --request POST \
  --data '{"project": "product", "name": "ORDER", "payload": {"user": 1234, "product": 1000}}' \
  http://localhost:8000/api/workflows

Read the documentation to try the quickstart and see advanced usages of Celery Director.

Project layout

.env                # The configuration file.
workflows.yml       # The workflows definition.
tasks/
    example.py      # A file containing some tasks.
    ...             # Other files containing other tasks.

Commands

director init [path] - Create a new project.
director celery [worker|beat|flower] - Start Celery daemons.
director webserver - Start the webserver.
director workflow [list|show|run] - Manage your project workflows.

License

See https://github.com/ovh/celery-director/blob/master/LICENSE

dmcclory/celery-director