django-celery Commands Inconsistency

[JeffPaine]

The django-skel Procfile

web: newrelic-admin run-program gunicorn -c gunicorn.py.ini wsgi:application
scheduler: python manage.py celery worker -B -E --maxtasksperchild=1000
worker: python manage.py celery worker -E --maxtasksperchild=1000

The django-celery docs claim it should read:

web: newrelic-admin run-program gunicorn -c gunicorn.py.ini wsgi:application
scheduler: python manage.py celery beat --maxtasksperchild=1000
worker: python manage.py celery worker --maxtasksperchild=1000

I don't presume to know if this has already been considered, just noticed it today as an inconsistency, so I thought I'd mention it. Cheers!

[rdegges]

Hi Jeff,

Thanks for checking it out! I'm actually doing it this way (notice the -B flag on the scheduler definition) so that the scheduler dyno does both beat AND a normal worker process. This makes it cheaper to run on Heroku, eg: instead of running a web dyno, a scheduler dyno, and a worker dyno, you can just run a web / scheduler dyno and still have task processing without any worker dynos present.

[JeffPaine]

Ahhh, gotcha, thanks!

you can just run a web / scheduler dyno and still have task processing without any worker dynos present

So with this Procfile you can have web, scheduler and task processing all with just heroku ps:scale web=1 scheduler=1 correct?

And then, if you need more web or worker dynos do a say heroku ps:scale web=3 scheduler=1 worker=3 correct?

Thanks much for your time.

[rdegges]

Yep, exactly! You'll never want more than 1 scheduler (otherwise you'll have duplicate tasks). So you can scale all the 'workers' freely >:)

[JeffPaine]

Fascinating, thanks!

For folks finding this in the future, here's the documentation of this feature. Essentially, you can run a celery beat (scheduler) from within a worker process.