aio-libs/multidict

Enable pickling of MultiDictProxy objects

hashstat opened this issue · 2 comments

As an aiohttp user, I would like MultiDictProxy and CIMultiDictProxy objects to be pickle-able so that I can cache aiohttp.ClientResponseErrors to disk and/or use aiohttp with multiprocessing.

Currently, attempting to pickle MultiDictProxy instances produces the following error:

>>> import pickle
>>> import multidict
>>> d = multidict.MultiDict(a=1, b=2, c=3)
>>> p = multidict.MultiDictProxy(d)
>>> pickle.dumps(p)
TypeError: can't pickle multidict._multidict.MultiDictProxy objects

Attempting to pickle a CIMultDictProxy instance produces a similar error, thus preventing their use in scenarios that require pickling, such as on-disk caching and multiprocessing, just to name a couple that I am actually using.

I believe the alternative is to use custom Pickler classes and/or monkeypatch modules to convert the proxy objects into MultiDict or CIMultDict objects, which do support pickling.

I ended up monkeypatching around this issue for now:

import multidict

class MultiDictProxy(multidict.MultiDictProxy):
    def __reduce__(self):
        return self.copy().__reduce__()

class CIMultiDictProxy(multidict.CIMultiDictProxy):
    def __reduce__(self):
        return self.copy().__reduce__()

def monkeypatch_multidict():
    multidict.MultiDictProxy = MultiDictProxy
    multidict.CIMultiDictProxy = CIMultiDictProxy

Sorry, no.
multidict proxies are modeled after types.MappingProxyType.
MappingProxyType is not pickleable, multidict proxies don't support pickling for the same reason.

Your patch breaks consistency:

>>> import pickle
>>> import multidict
>>> d = multidict.MultiDict(a=1, b=2, c=3)
>>> p = multidict.MultiDictProxy(d)
>>> d1, p2 = pickle.loads(pickle.dumps((d, p)))
>>> d2['a'] = 4
>>> assert p2['a'] == 4

A proxy object is just a very bad type for pickling.
Trying to slide through the limitation opens a can of worms.
aiohttp, in turn, is not dedicated to be used in the multiprocessed environment.
Please send through the wire request and response data, not aiohttp objects themself.