PicklingError: Can't pickle <class 'tagulous.models.descriptors.TagRelatedManager'>
valentijnscholten opened this issue · 6 comments
I am integrating django-tagulous into an existing/legacy application. This application uses celery extensively to do background processing. It's passing (lists of) model instances to the celery tasks. For this reason, the serialization protocol for celery is still set to the "old and dirty" pickle method.
I am running into the traceback below as it seems parts of tagulous models/classes cannot be pickled. At some point we want to refactor our code to only send model instance ids to celery tasks, but that is no easy task. And may result on some changed behaviour as it effectivelty means the celery tasks will (re)load the model instances from the database. Our models currently are too complex to use json serialization with celery.
Is there any quickfix for the below? Anybody else running into this? The traceback is not exactly clear on what is preventing the pickling from happening.
Traceback (most recent call last):
File "/home/valentijn/venv/lib/python3.8/site-packages/kombu/serialization.py", line 50, in _reraise_errors
yield
File "/home/valentijn/venv/lib/python3.8/site-packages/kombu/serialization.py", line 221, in dumps
payload = encoder(data)
File "/home/valentijn/venv/lib/python3.8/site-packages/kombu/serialization.py", line 350, in pickle_dumps
return dumper(obj, protocol=pickle_protocol)
_pickle.PicklingError: Can't pickle <class 'tagulous.models.descriptors.TagRelatedManager'>: attribute lookup TagRelatedManager on tagulous.models.descriptors failed
Full traceback (click to expand)
``` Traceback (most recent call last): File "/home/valentijn/venv/lib/python3.8/site-packages/kombu/serialization.py", line 50, in _reraise_errors yield File "/home/valentijn/venv/lib/python3.8/site-packages/kombu/serialization.py", line 221, in dumps payload = encoder(data) File "/home/valentijn/venv/lib/python3.8/site-packages/kombu/serialization.py", line 350, in pickle_dumps return dumper(obj, protocol=pickle_protocol) _pickle.PicklingError: Can't pickle : attribute lookup TagRelatedManager on tagulous.models.descriptors failedDuring handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/valentijn/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/home/valentijn/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 115, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/home/valentijn/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 113, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/valentijn/dd/dojo/user/helper.py", line 77, in _wrapped
return view_func(request, lookup_value, *args, **kwargs)
File "/home/valentijn/dd/dojo/finding/views.py", line 776, in edit_finding
new_finding.save(push_to_jira=push_to_jira)
File "/home/valentijn/dd/dojo/models.py", line 2156, in save
do_apply_rules(self, *args, **kwargs)
File "/home/valentijn/dd/dojo/decorators.py", line 38, in __wrapper__
return func(*args, **kwargs)
File "/home/valentijn/dd/dojo/decorators.py", line 23, in __wrapper__
return func.delay(*args, **kwargs)
File "/home/valentijn/venv/lib/python3.8/site-packages/celery/app/task.py", line 426, in delay
return self.apply_async(args, kwargs)
File "/home/valentijn/venv/lib/python3.8/site-packages/celery/app/task.py", line 566, in apply_async
return app.send_task(
File "/home/valentijn/venv/lib/python3.8/site-packages/celery/app/base.py", line 741, in send_task
amqp.send_task_message(P, name, message, **options)
File "/home/valentijn/venv/lib/python3.8/site-packages/celery/app/amqp.py", line 552, in send_task_message
ret = producer.publish(
File "/home/valentijn/venv/lib/python3.8/site-packages/kombu/messaging.py", line 167, in publish
body, content_type, content_encoding = self._prepare(
File "/home/valentijn/venv/lib/python3.8/site-packages/kombu/messaging.py", line 252, in _prepare
body) = dumps(body, serializer=serializer)
File "/home/valentijn/venv/lib/python3.8/site-packages/kombu/serialization.py", line 221, in dumps
payload = encoder(data)
File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
self.gen.throw(type, value, traceback)
File "/home/valentijn/venv/lib/python3.8/site-packages/kombu/serialization.py", line 54, in _reraise_errors
reraise(wrapper, wrapper(exc), sys.exc_info()[2])
File "/home/valentijn/venv/lib/python3.8/site-packages/vine/five.py", line 194, in reraise
raise value.with_traceback(tb)
File "/home/valentijn/venv/lib/python3.8/site-packages/kombu/serialization.py", line 50, in _reraise_errors
yield
File "/home/valentijn/venv/lib/python3.8/site-packages/kombu/serialization.py", line 221, in dumps
payload = encoder(data)
File "/home/valentijn/venv/lib/python3.8/site-packages/kombu/serialization.py", line 350, in pickle_dumps
return dumper(obj, protocol=pickle_protocol)
kombu.exceptions.EncodeError: Can't pickle <class 'tagulous.models.descriptors.TagRelatedManager'>: attribute lookup TagRelatedManager on tagulous.models.descriptors failed
ERROR:django.request:Internal Server Error: /finding/72719/edit
Traceback (most recent call last):
File "/home/valentijn/venv/lib/python3.8/site-packages/kombu/serialization.py", line 50, in _reraise_errors
yield
File "/home/valentijn/venv/lib/python3.8/site-packages/kombu/serialization.py", line 221, in dumps
payload = encoder(data)
File "/home/valentijn/venv/lib/python3.8/site-packages/kombu/serialization.py", line 350, in pickle_dumps
return dumper(obj, protocol=pickle_protocol)
_pickle.PicklingError: Can't pickle <class 'tagulous.models.descriptors.TagRelatedManager'>: attribute lookup TagRelatedManager on tagulous.models.descriptors failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/valentijn/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/home/valentijn/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 115, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/home/valentijn/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 113, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/valentijn/dd/dojo/user/helper.py", line 77, in _wrapped
return view_func(request, lookup_value, *args, **kwargs)
File "/home/valentijn/dd/dojo/finding/views.py", line 776, in edit_finding
new_finding.save(push_to_jira=push_to_jira)
File "/home/valentijn/dd/dojo/models.py", line 2156, in save
do_apply_rules(self, *args, **kwargs)
File "/home/valentijn/dd/dojo/decorators.py", line 38, in __wrapper__
return func(*args, **kwargs)
File "/home/valentijn/dd/dojo/decorators.py", line 23, in __wrapper__
return func.delay(*args, **kwargs)
File "/home/valentijn/venv/lib/python3.8/site-packages/celery/app/task.py", line 426, in delay
return self.apply_async(args, kwargs)
File "/home/valentijn/venv/lib/python3.8/site-packages/celery/app/task.py", line 566, in apply_async
return app.send_task(
File "/home/valentijn/venv/lib/python3.8/site-packages/celery/app/base.py", line 741, in send_task
amqp.send_task_message(P, name, message, **options)
File "/home/valentijn/venv/lib/python3.8/site-packages/celery/app/amqp.py", line 552, in send_task_message
ret = producer.publish(
File "/home/valentijn/venv/lib/python3.8/site-packages/kombu/messaging.py", line 167, in publish
body, content_type, content_encoding = self._prepare(
File "/home/valentijn/venv/lib/python3.8/site-packages/kombu/messaging.py", line 252, in _prepare
body) = dumps(body, serializer=serializer)
File "/home/valentijn/venv/lib/python3.8/site-packages/kombu/serialization.py", line 221, in dumps
payload = encoder(data)
File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
self.gen.throw(type, value, traceback)
File "/home/valentijn/venv/lib/python3.8/site-packages/kombu/serialization.py", line 54, in _reraise_errors
reraise(wrapper, wrapper(exc), sys.exc_info()[2])
File "/home/valentijn/venv/lib/python3.8/site-packages/vine/five.py", line 194, in reraise
raise value.with_traceback(tb)
File "/home/valentijn/venv/lib/python3.8/site-packages/kombu/serialization.py", line 50, in _reraise_errors
yield
File "/home/valentijn/venv/lib/python3.8/site-packages/kombu/serialization.py", line 221, in dumps
payload = encoder(data)
File "/home/valentijn/venv/lib/python3.8/site-packages/kombu/serialization.py", line 350, in pickle_dumps
return dumper(obj, protocol=pickle_protocol)
```
I think it's related to this statement where class/type is being modified:
It looks like the existing serializers provided by tagulous avoid this by 'detagging' the model before serialization. But it looks like we can't provide our own pickle serializer module.
Yes it could well be that - the class would not be defined somewhere that would be easy for a generic pickler to find.
Can you write a custom pickler? If you can get it to pickle the tag field as a string eg str(instance.mytagfield) I would think the unpickling would sort it out automatically at the other end.
This isn't something I've tried - as you say, I'd normally pass a pk and do another lookup in celery, or use django's serializers.
Can you write a custom pickler? If you can get it to pickle the tag field as a string eg
str(instance.mytagfield)I would think the unpickling would sort it out automatically at the other end.
I tried to set a pickle entry in SERIALIZATION_MODULES, but it doesn't seem to do anything. Or do you mean a different construct? I was just looking at the detagging stuff to see if I could use that in some shape or form.
I took a quick look at kombu's docs and it sounds like you're using pythons standard pickle? I can't remember the details off-hand, but could start by looking at adding a __getstate__ method to the tagged model.
Interesting one. I won't be able to look at this for a while, but do let me know if you make any progress in the meantime.
I didn't have to time to find out how these serializers work, so went with a simple model_to_dict and list_of_models_to_dict. But this doesn't cover all cases, so looks like serializer would be needed after all.
Awesome, thanks. Never got around myself to dive into the details of pickling.