Integrate Celery with flasky

[harrywang]

Hi Miguel,

I was able to get this celery example working - thanks for a great tutorial. However, I have trouble to integrate Celery with Flasky from your book. I am just not sure where to put various information into Flasky to make it work. For example, I have put the following line in config.py:

# Celery configuration
CELERY_BROKER_URL = 'redis://localhost:6379/0'
CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'

And in init.py:

from celery import Celery

but I don't know where to add:

# Initialize Celery
celery = Celery(app.name, broker=app.config['CELERY_BROKER_URL'])
celery.conf.update(app.config)

will the following work?

celery = Celery()

And then what to do? Please help and thanks.

[harrywang]

I have this:

celery.init_app(app)

got an error when running:

python manage.py runserver
Traceback (most recent call last):
 File "manage.py", line 21, in <module>
app = create_app(os.getenv('FLASK_CONFIG') or 'default')
File "../app/__init__.py", line 34, in create_app
celery.init_app(app)
AttributeError: 'Celery' object has no attribute 'init_app'

[miguelgrinberg]

@harrywang yes, it's a bit tricky because the Celery class is not a Flask extension.

I haven't tried this, but what probably works is to decouple the celery instance creation from the Flask app:

celery = Celery(__name__, broker=Config.CELERY_BROKER_URL)

Then in create_app() you can apply the configuration, once you have an app instance:

def create_app():
    # ...
    celery.conf.update(app.config)
    # ...
    return app

[harrywang]

I just tried and got an error:

$ python manage.py runserver
Traceback (most recent call last):
File "manage.py", line 16, in <module>
from app import create_app, db
File "/flasky/app/__init__.py", line 16, in <module>
celery = Celery(__name__, broker=config.CELERY_BROKER_URL)
AttributeError: 'dict' object has no attribute 'CELERY_BROKER_URL'

[miguelgrinberg]

It's Config (uppercase C), the base config class.

[harrywang]

Just tried:

$ python manage.py runserver
Traceback (most recent call last):
File "manage.py", line 16, in <module>
from app import create_app, db
File ".../app/__init__.py", line 16, in <module>
celery = Celery(__name__, broker=Config.CELERY_BROKER_URL)
NameError: name 'Config' is not defined

I have the following lines:

from celery import Celery
from config import config

[miguelgrinberg]

Try:

from config import Config

The problem here is that you need this item from the config, but there is no instantiated app yet, so the trick is to get it direct from the configuration class. The disadvantage is that you can't use different brokers for development/production/testing.

[harrywang]

I found this: https://github.com/Robpol86/Flask-Celery-Helper

I am trying this and will report soon. Thanks, Miguel.

[harrywang]

from config import Config does not work. I really hope I can make Celery work with Flasky but run into trouble.

I tried to use the https://github.com/Robpol86/Flask-Celery-Helper with Flasky but cannot get it work:

this is what I have:

config.py: https://gist.github.com/harrywang/886f71d4dfc62648b2c3
init.py: https://gist.github.com/harrywang/430ecd6d359bdd6a2cd1

I created
app/task.py: https://gist.github.com/harrywang/26a2f0706d5a5eddef5c
app/tasks/views.py: https://gist.github.com/harrywang/cb2cb397d2ddd6d9d6e7
app/templates/celery.html: https://gist.github.com/harrywang/70ee82aa56136c79a61a

When I start redis using run-redis.sh and then run celery worker -A app.celery --loglevel=info, got the following error:

[2015-05-12 16:04:12,581: ERROR/MainProcess] consumer: Cannot connect to   amqp://guest:**@127.0.0.1:5672//: [Errno 61] Connection refused.
Trying again in 2.00 seconds...

Please help and thanks a lot!!

[harrywang]

@miguelgrinberg a more generalized question related to this is: how to integrate services that are not Flask extensions, such as in this case the Celery class is not a Flask extension. Thanks again.

[miguelgrinberg]

@harrywang I'm working on a Flasky + Celery example that sends the emails as Celery tasks. I will upload in a day or two.

[harrywang]

@miguelgrinberg wow. Really looking forward to it. Thanks a lot!! :)

[miguelgrinberg]

Here is the project: https://github.com/miguelgrinberg/flasky-with-celery. The last commit has all the Celery related changes.

[harrywang]

@miguelgrinberg I have tried the new Flasky+Celery example and it worked great!! Thanks a lot for your help!