'Message' is not JSON serializable
astronaut0131 opened this issue · 8 comments
trying to run the first example but got the following error.
the code is cloned from the repository.
Traceback (most recent call last): File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/flask/app.py", line 1985, in wsgi_app response = self.handle_exception(e) File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/flask/app.py", line 1540, in handle_exception reraise(exc_type, exc_value, tb) File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise raise value File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app response = self.full_dispatch_request() File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request rv = self.handle_user_exception(e) File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception reraise(exc_type, exc_value, tb) File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise raise value File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request rv = self.dispatch_request() File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/Users/astronaut/PycharmProjects/using_celery_with_flask/flask-celery-example/app.py", line 75, in index send_async_email.delay(msg) File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/celery/app/task.py", line 413, in delay return self.apply_async(args, kwargs) File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/celery/app/task.py", line 536, in apply_async **options File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/celery/app/base.py", line 737, in send_task amqp.send_task_message(P, name, message, **options) File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/celery/app/amqp.py", line 554, in send_task_message **properties File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/kombu/messaging.py", line 169, in publish compression, headers) File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/kombu/messaging.py", line 252, in _prepare body) = dumps(body, serializer=serializer) File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/kombu/serialization.py", line 221, in dumps payload = encoder(data) File "/usr/local/Cellar/python3/3.6.4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/contextlib.py", line 99, in __exit__ self.gen.throw(type, value, traceback) File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/kombu/serialization.py", line 54, in _reraise_errors reraise(wrapper, wrapper(exc), sys.exc_info()[2]) File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/vine/five.py", line 178, in reraise raise value.with_traceback(tb) File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/kombu/serialization.py", line 50, in _reraise_errors yield File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/kombu/serialization.py", line 221, in dumps payload = encoder(data) File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/kombu/utils/json.py", line 72, in dumps **dict(default_kwargs, **kwargs)) File "/usr/local/Cellar/python3/3.6.4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 238, in dumps **kw).encode(obj) File "/usr/local/Cellar/python3/3.6.4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/local/Cellar/python3/3.6.4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/kombu/utils/json.py", line 62, in default return super(JSONEncoder, self).default(o) File "/usr/local/Cellar/python3/3.6.4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 180, in default o.__class__.__name__) kombu.exceptions.EncodeError: Object of type 'Message' is not JSON serializable
This example was written for Celery 3, which uses pickle as a default serialization format. In Celery 4, the default was changed to JSON, so the error here is that there is no way to serialize the Message object in JSON format.
You can fix the problem in a few ways:
- downgrade to Celery v3
- configure your Celery v4 to use pickle as serialization format
- don't construct a
Messageinstance, just send all the components of the message in a dictionary, then construct theMessageinstance in the task.
thank you very much!clear explanation :D @miguelgrinberg
It is highly recommended to not use pickle as it has a vulnerability. And I think that's why it has been replaced in version 4 by JSON. Check the official docs here. Also check out this blogpost discussing the security vulnerability found in pickle.
@Habush would you be interested in crafting a PR that changes this project to use the 3rd option I indicated above? Happy to merge a PR that upgrades to Celery 4 and uses JSON instead of pickle.
That said, the warning that you refer reads "Never unpickle data received from an untrusted or unauthenticated source". The key there is that the data needs to come from an untrusted source. In this application you are running the web server and the Celery workers in a protected environment of a production server. For this to be an issue you would need the attacker to be able to somehow post a malicious message to the message broker (Redis or Rabbit), which is not going to be accessible from the Internet (or at least it shouldn't be). So while I fully agree that it's best to not use pickle, in this particular case the risk is extremely low.
This example was written for Celery 3, which uses pickle as a default serialization format. In Celery 4, the default was changed to JSON, so the error here is that there is no way to serialize the
Messageobject in JSON format.You can fix the problem in a few ways:
- downgrade to Celery v3
- configure your Celery v4 to use pickle as serialization format
- don't construct a
Messageinstance, just send all the components of the message in a dictionary, then construct theMessageinstance in the task.
Hi Miguel,
I'm struggling with this error because of the app_context. I'm starting from the send_async_email and send_email from your wonderful Flask Tutorial. Because of this JSON serialisation thing, I do not find a way to rewrite the code for using celery...
Would you be so kind as to expand your Celery/Flask example with Celery 4/JSON and current_app._get_current_object() (or the right solution)?
I would be eternally grateful and probably other would be too.
@cdruet the code in this repository has been updated to work with Celery 4. Have you seen it?
@miguelgrinberg Just did ;-)
But, as I found out on my own late during the night, it only fixes the Message serialisation thing. I kept struggling with the app_context for a while after that. I ultimately found inspiration in your flasky-with-celery-example on how to structure things to make it work. In doing so, I struggleg with ENV variables when running celery_worker.py (I forgot that I was using dotenv as you suggested in your flasky tutorial, and that it should also be integrated in celery_worker.py).
I'm still convinced the whole thing is worth a post to illustrate where and how things should be declared and organised in a neat app structure.