KQ (Kafka Queue) is a lightweight Python library which provides a simple API to process jobs asynchronously in the background. It uses Apache Kafka and is designed primarily for ease of use.
- Apache Kafka 0.9+
- Python 2.7+ 3.4+ or 3.5+
First, ensure that your Kafka instance is up and running:
# This command is just an example
~$ ./kafka-server-start.sh -daemon server.properties
Let's say you want to run the following function asynchronously:
import time
def my_func(foo, bar, baz=None):
"""This is a blocking function."""
time.sleep(10)
return foo, bar, baz
Start a KQ worker:
~$ kq worker --verbose
[INFO] Starting Worker(topic=default) ...
Enqueue the function call as a job:
# Import the blocking function
from my_module import my_func
# Initialize a queue
from kq import Queue
q = Queue()
# Enqueue the function call
q.enqueue(my_func, 1, 2, baz=3)
Sit back and watch the worker process it in the background:
~$ kq worker --verbose
[INFO] Starting Worker(topic=default) ...
[INFO] Processing Record(topic=default, partition=5, offset=3) ...
[INFO] Running Job 1b92xle0: my_module.my_func(1, 2, baz=3) ...
[INFO] Job 1b92xle0 returned: (1, 2, 3)
Check out the full documentation for more details!
To install a stable version from PyPI (recommended):
~$ pip install kq
To install the latest version directly from GitHub:
~$ pip install -e git+git@github.com:joowani/kq.git@master#egg=kq
You may need to use sudo
depending on your environment setup.
This project was inspired by RQ and built on top of kafka-python.