Override default pickling behavior
gpauloski opened this issue · 4 comments
I am trying to override the default pickling behavior of Proxy
to instead pickle the factory. This works for lazy_object_proxy.cext.Proxy
but not for lazy_object_proxy.slots.Proxy
.
from lazy_object_proxy.slots import Proxy
class MyProxy(Proxy):
def __reduce__(self):
return MyProxy, (self.__factory__,)
def __reduce_ex__(self, protocol):
return MyProxy, (self.__factory__,)
Then, pickling an instance of MyProxy
gives the following error:
Traceback (most recent call last):
File "test.py", line 37, in <module>
proxy_pkl = pkl.dumps(proxy_instance)
_pickle.PicklingError: Can't pickle <class 'MyProxy'>: import of module <property object at 0x7f42833a50b0> failed
The factory in this case is a callable, pickleable object that returns a large numpy array, but I do not want to include the large array in the pickling (as done with the default behavior).
Is there a workaround for pickling the slots version of a Proxy? I am not using the lazy_object_proxy.cext.Proxy
because I have run into some strange behavior with common numpy functions.
import pickle as pkl
from lazy_object_proxy.slots import Proxy
class Factory():
def __init__(self, obj):
self.obj = obj
def __call__(self):
return self.obj
class MyProxy(Proxy):
def __reduce__(self):
return MyProxy, (self.__factory__,)
def __reduce_ex__(self, protocol):
return MyProxy, (self.__factory__,)
p = MyProxy(Factory([1, 2, 3]))
p_pkl = pkl.dumps(p) # Raises error
p = pkl.loads(p_pkl)
I am using lazy-object-proxy 1.6.0 and Python 3.7.10.
Well yes it breaks cause slots.Proxy defines a property for __module__
and that confuses pickle a lot.
There are two ways to solve this:
- use simple.Proxy
- use a trampoline like so:
def proxy_trampoline(factory):
return MyProxy(factory)
class MyProxy(Proxy):
def __reduce__(self):
return proxy_trampoline, (object.__getattribute__(self, '__factory__'), )
def __reduce_ex__(self, protocol):
return proxy_trampoline, (object.__getattribute__(self, '__factory__'), )
That make sense, and the trampoline works well. Thanks for the help.