joagonzalez/async-scheduler

Async task scheduler through rest API using flask, celery and rabbitMQ

Celery using rabbitMQ as broker

Using producer-consumer approach allows to decouple task scheduling from workers that actually execute tasks. This also allows to run code in workers regardless of their OS and having load balacing for task execution.

Late acknowledge tasks are recommended when the code that workers must run is idempotent.

Celery + two workers Run celery workers

cd single_worker

celery -A tasks worker --loglevel=info -n worker1
celery -A tasks worker --loglevel=info -n worker2

Celery + flask

cd flask_worker

python3 app.py
celery -A app.celery worker --loglevel=info -n worker1
celery -A app.celery worker --loglevel=info -n worker2
celery -A services.workerService.linux worker --loglevel=info -n worker1 --queue linux

Celery + windows workers

Install requirements

pip install eventlet
pip install gevent
pip install pymsteams
pip install celery==4.4.5

Considerations regarding celery version and windows compatibility

https://stackoverflow.com/questions/37255548/how-to-run-celery-on-windows

celery -A <module> worker -l info -P gevent

window 10 + celery 4.2 + python 3.6

celery -A windows_worker.celery worker --loglevel=Info -n worker3-win -P gevent
celery -A windows_worker.celery worker --loglevel=Info -n worker2-win -P gevent

/process/<word>

jgonzalez@godel:~$ curl http://127.0.0.1:5000/process/joaquin | python -mjson.tool

{
    "buffer_status": "{ '5a636196-ba86-4e58-a770-9ae46df2199c': <AsyncResult: a19712ea-ac8a-4523-89d6-09c89b753250>}",
    "name": "joaquin",
    "scheduled": true,
    "status": "will be ready in 10 sec",
    "task_state": "PENDING",
    "taskid": "5a636196-ba86-4e58-a770-9ae46df2199c"
}

/status/<taskid>

jgonzalez@godel:~$ curl http://127.0.0.1:5000/status/5a636196-ba86-4e58-a770-9ae46df2199c |  python -mjson.tool

{
    "result": "Not ready",
    "status": "PENDING"
}

jgonzalez@godel:~$ curl http://127.0.0.1:5000/status/5a636196-ba86-4e58-a770-9ae46df2199c |  python -mjson.tool

{
    "result": "niuqaoj",
    "status": "SUCCESS"
}

/buffer/

jgonzalez@godel:~$ curl http://127.0.0.1:5000/buffer | python -mjson.tool
{
    "result": "{'b9810946-ee45-4bd6-bae2-7eedea36d54e': <AsyncResult: b9810946-ee45-4bd6-bae2-7eedea36d54e>, '303a2d28-66ac-471f-bb61-6653d3950b58': <AsyncResult: 303a2d28-66ac-471f-bb61-6653d3950b58>, '654fcfa0-cc9a-4110-bb3b-13e5d6aae882': <AsyncResult: 654fcfa0-cc9a-4110-bb3b-13e5d6aae882>, 'faf161ae-b9b8-48fe-97c6-2251419ab896': <AsyncResult: faf161ae-b9b8-48fe-97c6-2251419ab896>, 'a24f9ed7-966f-46f8-943d-630fe2c40f3f': <AsyncResult: a24f9ed7-966f-46f8-943d-630fe2c40f3f>, 'bd9832de-c3ec-4738-b0c5-6d514249bf4a': <AsyncResult: bd9832de-c3ec-4738-b0c5-6d514249bf4a>, '4733d56e-1c91-4f1b-a72a-1901ce6e8e15': <AsyncResult: 4733d56e-1c91-4f1b-a72a-1901ce6e8e15>}"
}

/powershell

jgonzalez@godel:~$ curl http://127.0.0.1:5000/powershell | python -mjson.tool

Configure rabbitMQ

Port 8080 will expose RabbitMQ monitoring GUI

docker run -d --hostname rabbitmq-poc --name rabbitmq-poc -p 5672:5672 -p 8080:15672 rabbitmq:3-management

Configure flower

Flower allows monitor Celery worker and also expose an API to manage the cluster

docker run -d --hostname flower-poc --name flower-poc -p 5555:5555 -p 8888:8888 mher/flower "flower --broker=pyamqp://guest@localhost// --port=8888" 

celery -A app.celery flower --port=5555 --address=0.0.0.0 --basic_auth=admin:admin --broker=pyamqp://guest@localhost// --broker_api=http://guest@localhost.com:8080/api