Cloudpickle compatibility

[samimia-swks]

params 1.13.0
cloudpickle 2.2.1
python 3.11
Ubuntu 22.04

The docs recommend Cloudpickle as a way to pickle Parameterized classes with lambda functions, but even the most basic Parameterized object (with or without lambdas) cannot be pickled with cloudpickle :

import param
import cloudpickle

class MyParams(param.Parameterized):
    a = param.Integer()
    b = param.Integer()
    # sum_pull = Integer(lambda : p.a + p.b)

p = MyParams()
pkl = cloudpickle.dumps(p)
p2 = cloudpickle.loads(pkl)

Output

File [/param/parameterized.py:1685), in Parameters.__setstate__(self, state)
   1682 def __setstate__(self, state):
   1683     # Set old parameters state on Parameterized._parameters_state
   1684     self_or_cls = state.get('self', state.get('cls'))
-> 1685     for k in self_or_cls._parameters_state:
   1686         key = '_'+k
   1687         if key in state:

AttributeError: 'NoneType' object has no attribute '_parameters_state'

[maximlt]

Hi @samimia-swks,

cloudpickle is indeed mentioned in the docs, albeit in a pretty light way 🙃

lambdas cannot easily be pickled for saving and restoring state (though see cloudpickle for an alternative to pickle that does support lambdas)

Unfortunately it seems that Param doesn't support it out of the box, even for very simple cases like the one you tried above. It'd be nice to add support for cloudpickle (and fix the original support for pickle if it's been broken in some ways), contributions in the area of serializing/deserializing Param objects are welcomed :) !

[samimia-swks]

Hi @maximlt,

I was able to figure out that cloudpickle (or dill) support was broken by #386 which is between version 1.9.3 and 1.10.0.

If I remove the offending lines of code, which are the definition setstate() method for the Parameters class, I can get cloudpickle to pickle a Parametrized class with both lambdas and @param.depends() decorators even on the latest 1.13.0 version :

    def __setstate__(self, state):
        # Set old parameters state on Parameterized._parameters_state
        self_or_cls = state.get('self', state.get('cls'))
        for k in self_or_cls._parameters_state:
            key = '_'+k
            if key in state:
                self_or_cls._parameters_state[k] = state.pop(key)
        for k, v in state.items():
            setattr(self, k, v)

From what I understand, this is a method that's called by pickle after unpickeling. However, I can't tell what it is doing because things still seem to work without it. My knowledge of python and this project is not nearly good enough to figure this out in the short terrm :)

Perhaps @philippjfr, who authored this code can clarify.

[maximlt]

Thanks @samimia-swks I'm sure that's going to be very helpful!