How Come dill doesn't dill subobjects?

Question

How Come dill doesn't dill subobjects?

Closed this issue 2 years ago · 13 comments

radiantone commented 2 years ago

I have a class Customer with a property Address, another class.

dill doesn't serialize the Address property and its state. This seems like a basic feature it should be doing.

Answer 1 · 2023-01-18T17:50:52.000Z

Please add an example that demonstrates the issue you are seeing.

To my knowledge, dill serializes recursively, so any "sub-objects" should also be serialized.
For example:

Python 3.7.16 (default, Dec  7 2022, 05:04:27) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> class Address:
...   def __init__(self, street, city, zip):
...     self.street = street
...     self.city = city
...     self.zip = zip
...   def __call__(self):
...     return '\n'.join([self.street, self.city, self.zip])
...   def __repr__(self):
...     return self()
... 
>>> class Customer:
...   def __init__(self, name, address):
...     self.name = name
...     self.address = address
... 
>>> joe = Customer('Joe Smith', Address('11 Fish Lane','Yuma, AZ','85364'))
>>> joe.name
'Joe Smith'
>>> joe.address
11 Fish Lane
Yuma, AZ
85364
>>> 
>>> import dill
>>> customer = dill.loads(dill.dumps(joe))
>>> customer.name
'Joe Smith'
>>> customer.address
11 Fish Lane
Yuma, AZ
85364

Answer 2 · 2023-01-18T20:30:22.000Z

Yeah, I'm seeing something weird here with my classes, let me do up a test of it and share.

Answer 3 · 2023-01-18T20:30:39.000Z

It works with regular pickle, but not with dill. btw

Answer 4 · 2023-01-19T13:05:39.000Z

It works with regular pickle, but not with dill. btw

I'll be interested to see a minimal example that reproduces your issue, if that's the case. Which dill serialization variant are you using (e.g. dill.settings['recurse'] = True)?

Answer 5 · 2023-01-19T14:30:42.000Z

Here is an example to reproduce it. As you can see, works fine with pickle. But subobject not recreated with dill

from dataclasses import dataclass

from persistent import Persistent
import dataclasses
import datetime
import json
from typing import Any


@dataclass
class EmergeData(Persistent):

    id: str
    data: Any
    date: str = str(datetime.datetime.now().strftime("%b %d %Y %H:%M:%S"))

    def __str__(self):
        return json.dumps(dataclasses.asdict(self))


@dataclass
class EmergeObject(EmergeData):

    pass


@dataclass
class EmergeFile(EmergeObject):
    """Holds references to file"""

    name: str = ""
    path: str = "/"
    perms: str = "rwxrwxrwx"
    type: str = "file"
    data: str = ""
    uuid: str = ""
    node: str = ""

    def str(self):
        return self.to_json(self)

    def to_json(self):
        import json
        from types import ModuleType

        import persistent.list

        ser = {}

        for i in [
            v
            for v in dir(self)
            if not callable(getattr(self, v))
            and v[0] != "_"
            and isinstance(getattr(self, v), ModuleType) is False
        ]:
            if isinstance(getattr(self, i), EmergeObject):
                ser[i] = getattr(self, i)
            if isinstance(getattr(self, i), persistent.list.PersistentList):
                ser[i] = [str(o) for o in getattr(self, i)]
            else:
                ser[i] = getattr(self, i)

        return json.dumps(ser)


@dataclass
class Address(EmergeFile):

    street: str = ''
    city: str = ''
    state: str = ''


@dataclass
class Customer(EmergeFile):

    address: Address = None
    name: str = ''


customer = Customer(
    id="customer1",
    name="johndoe",
    path="/customers",
    address=Address(id="address1", street="Park Place", city="Quincy", state="MA")
)
import dill
import pickle

print(str(customer.address))
print("WITH DILL:")
print(dill.loads(dill.dumps(customer)))

print("WITH PICKLE:")
print(pickle.loads(pickle.dumps(customer)))

Answer 6 · 2023-01-19T16:02:04.000Z

When I go though your example, I get the same result from dill and pickle. You mentioned that the Address wasn't reproduced, but I'm seeing that it is:

Python 3.7.16 (default, Dec  7 2022, 05:04:27) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from dataclasses import dataclass
>>> Persistent = object
>>> import dataclasses
>>> import datetime
>>> import json
>>> from typing import Any
>>>
>>> @dataclass
... class EmergeData(Persistent):
...     id: str
...     data: Any
...     date: str = str(datetime.datetime.now().strftime("%b %d %Y %H:%M:%S"))
...     def __str__(self):
...         return json.dumps(dataclasses.asdict(self))
... 
>>> @dataclass
... class EmergeObject(EmergeData):
...     pass
... 
>>> @dataclass
... class EmergeFile(EmergeObject):
...     """Holds references to file"""
...     name: str = ""
...     path: str = "/"
...     perms: str = "rwxrwxrwx"
...     type: str = "file"
...     data: str = ""
...     uuid: str = ""
...     node: str = ""
...     def str(self):
...         return self.to_json(self)
...     def to_json(self):
...         import json
...         from types import ModuleType
...         import persistent.list
...         ser = {}
...         for i in [
...             v
...             for v in dir(self)
...             if not callable(getattr(self, v))
...             and v[0] != "_"
...             and isinstance(getattr(self, v), ModuleType) is False
...         ]:
...             if isinstance(getattr(self, i), EmergeObject):
...                 ser[i] = getattr(self, i)
...             if isinstance(getattr(self, i), persistent.list.PersistentList): 
...                 ser[i] = [str(o) for o in getattr(self, i)]
...             else:
...                 ser[i] = getattr(self, i)
...         return json.dumps(ser)
... 
>>> @dataclass
... class Address(EmergeFile):
...     street: str = ''
...     city: str = ''
...     state: str = ''
... 
>>> @dataclass
... class Customer(EmergeFile):
...     address: Address = None
...     name: str = ''
... 
>>> customer = Customer(
...     id="customer1",
...     name="johndoe",
...     path="/customers",
...     address=Address(id="address1", street="Park Place", city="Quincy", state="MA")
... )
>>> 
>>> customer
Customer(id='customer1', data='', date='Jan 19 2023 10:42:48', name='johndoe', path='/customers', perms='rwxrwxrwx', type='file', uuid='', node='', address=Address(id='address1', data='', date='Jan 19 2023 10:42:48', name='', path='/', perms='rwxrwxrwx', type='file', uuid='', node='', street='Park Place', city='Quincy', state='MA'))
>>> customer.address
Address(id='address1', data='', date='Jan 19 2023 10:42:48', name='', path='/', perms='rwxrwxrwx', type='file', uuid='', node='', street='Park Place', city='Quincy', state='MA')
>>> 
>>> 
>>> import pickle
>>> pickle.loads(pickle.dumps(customer))
Customer(id='customer1', data='', date='Jan 19 2023 10:42:48', name='johndoe', path='/customers', perms='rwxrwxrwx', type='file', uuid='', node='', address=Address(id='address1', data='', date='Jan 19 2023 10:42:48', name='', path='/', perms='rwxrwxrwx', type='file', uuid='', node='', street='Park Place', city='Quincy', state='MA'))
>>> import dill
>>> dill.loads(dill.dumps(customer))
Customer(id='customer1', data='', date='Jan 19 2023 10:42:48', name='johndoe', path='/customers', perms='rwxrwxrwx', type='file', uuid='', node='', address=Address(id='address1', data='', date='Jan 19 2023 10:42:48', name='', path='/', perms='rwxrwxrwx', type='file', uuid='', node='', street='Park Place', city='Quincy', state='MA'))

...and if we turn on pickle tracing, then we see that dill pickles all of the "sub-objects":

>>> dill.detect.trace(True)
>>> dill.loads(dill.dumps(customer))
┬ T2: <class '__main__.Customer'>
├┬ F2: <function _create_type at 0x10b64d560>
│└ # F2 [27 B]
├┬ T1: <class 'type'>
│├┬ F2: <function _load_type at 0x10b64d440>
││└ # F2 [25 B]
│└ # T1 [42 B]
├┬ T2: <class '__main__.EmergeFile'>
│├┬ T2: <class '__main__.EmergeObject'>
││├┬ T2: <class '__main__.EmergeData'>
│││├┬ T1: <class 'object'>
││││└ # T1 [21 B]
│││├┬ D2: <dict object at 0x010ba8bbe0>
││││├┬ D2: <dict object at 0x010b39da50>
│││││├┬ T1: <class 'str'>
││││││└ # T1 [18 B]
│││││└ # D2 [70 B]
││││├┬ F1: <function EmergeData.__str__ at 0x10b3bad40>
│││││├┬ F2: <function _create_function at 0x10b64d4d0>
││││││└ # F2 [31 B]
│││││├┬ Co: <code object __str__ at 0x10b2660c0, file "<stdin>", line 6>
││││││├┬ F2: <function _create_code at 0x10b64d5f0>
│││││││└ # F2 [27 B]
││││││└ # Co [166 B]
│││││├┬ D1: <dict object at 0x010afc3eb0>
││││││└ # D1 [22 B]
│││││├┬ D2: <dict object at 0x010ba725f0>
││││││└ # D2 [3 B]
│││││├┬ D2: <dict object at 0x010ba8bc80>
││││││├┬ D2: <dict object at 0x010ba72550>
│││││││└ # D2 [3 B]
││││││└ # D2 [54 B]
│││││└ # F1 [291 B]
││││├┬ T4: <class 'dataclasses._DataclassParams'>
│││││└ # T4 [32 B]
││││├┬ D2: <dict object at 0x010ba6a1e0>
│││││└ # D2 [85 B]
││││├┬ D2: <dict object at 0x010b0601e0>
│││││├┬ T4: <class 'dataclasses.Field'>
││││││└ # T4 [21 B]
│││││├┬ D2: <dict object at 0x010ba87550>
││││││├┬ DcM: <dataclasses._MISSING_TYPE object at 0x10b265c90>
│││││││└ # DcM [21 B]
││││││├┬ DcM: <dataclasses._MISSING_TYPE object at 0x10b265c90>
│││││││└ # DcM [21 B]
││││││├┬ Mp: <mappingproxy object at 0x010b265f10>
│││││││├┬ T1: <class 'mappingproxy'>
││││││││└ # T1 [31 B]
│││││││├┬ D2: <dict object at 0x010ba880f0>
││││││││└ # D2 [3 B]
│││││││└ # Mp [40 B]
││││││├┬ DcFB: _FIELD
│││││││└ # DcFB [20 B]
││││││└ # D2 [226 B]
│││││├┬ D2: <dict object at 0x010ba88140>
││││││├┬ DcM: <dataclasses._MISSING_TYPE object at 0x10b265c90>
│││││││└ # DcM [21 B]
││││││├┬ DcM: <dataclasses._MISSING_TYPE object at 0x10b265c90>
│││││││└ # DcM [21 B]
││││││├┬ DcFB: _FIELD
│││││││└ # DcFB [20 B]
││││││└ # D2 [97 B]
│││││├┬ D2: <dict object at 0x010ba871e0>
││││││├┬ DcM: <dataclasses._MISSING_TYPE object at 0x10b265c90>
│││││││└ # DcM [21 B]
││││││├┬ DcFB: _FIELD
│││││││└ # DcFB [20 B]
││││││└ # D2 [78 B]
│││││└ # D2 [464 B]
││││├┬ F1: <function __create_fn__.<locals>.__init__ at 0x10b3a6ef0>
│││││├┬ Co: <code object __init__ at 0x10b3be030, file "<string>", line 2>
││││││└ # Co [101 B]
│││││├┬ D1: <dict object at 0x010afc3eb0>
││││││└ # D1 [22 B]
│││││├┬ D2: <dict object at 0x010ba75460>
││││││└ # D2 [3 B]
│││││├┬ D2: <dict object at 0x010ba83e10>
││││││├┬ D2: <dict object at 0x010b404370>
│││││││└ # D2 [31 B]
││││││└ # D2 [78 B]
│││││└ # F1 [225 B]
││││├┬ F1: <function __create_fn__.<locals>.__repr__ at 0x10b3beb90>
│││││├┬ Co: <code object wrapper at 0x10b2521e0, file "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/dataclasses.py", line 350>
││││││└ # Co [368 B]
│││││├┬ D2: <dict object at 0x010ba6a460>
││││││└ # D2 [21 B]
│││││├┬ Ce2: <cell at 0x10b387b10: set object at 0x10b350aa0>
││││││├┬ F2: <function _create_cell at 0x10b64ddd0>
│││││││└ # F2 [27 B]
││││││└ # Ce2 [34 B]
│││││├┬ Ce2: <cell at 0x10b407110: function object at 0x10b396710>
││││││└ # Ce2 [9 B]
│││││├┬ D2: <dict object at 0x010b395050>
││││││├┬ F1: <function __create_fn__.<locals>.__repr__ at 0x10b396710>
│││││││├┬ Co: <code object __repr__ at 0x10b3a6270, file "<string>", line 2>
││││││││└ # Co [160 B]
│││││││├┬ D1: <dict object at 0x010afc3eb0>
││││││││└ # D1 [22 B]
│││││││├┬ D2: <dict object at 0x010b395370>
││││││││└ # D2 [3 B]
│││││││├┬ D2: <dict object at 0x010ba7ef00>
││││││││├┬ D2: <dict object at 0x010b395410>
│││││││││└ # D2 [3 B]
││││││││└ # D2 [50 B]
│││││││└ # F1 [252 B]
││││││└ # D2 [274 B]
│││││├┬ D2: <dict object at 0x010ba85050>
││││││└ # D2 [13 B]
│││││├┬ T4: <class '_frozen_importlib_external.SourceFileLoader'>
││││││└ # T4 [47 B]
│││││├┬ D2: <dict object at 0x010b2515a0>
││││││└ # D2 [115 B]
│││││├┬ T4: <class '_frozen_importlib.ModuleSpec'>
││││││└ # T4 [32 B]
│││││├┬ D2: <dict object at 0x010b060640>
││││││└ # D2 [268 B]
│││││├┬ D4: <dict object at 0x010af42b90>
││││││└ # D4 [19 B]
│││││├┬ M2: <module 're' from '/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/re.py'>
││││││├┬ F2: <function _import_module at 0x10b654680>
│││││││└ # F2 [29 B]
││││││└ # M2 [37 B]
│││││├┬ M2: <module 'sys' (built-in)>
││││││└ # M2 [18 B]
│││││├┬ M2: <module 'copy' from '/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/copy.py'>
││││││└ # M2 [10 B]
│││││├┬ M2: <module 'types' from '/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/types.py'>
││││││└ # M2 [10 B]
│││││├┬ M2: <module 'inspect' from '/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/inspect.py'>
││││││└ # M2 [10 B]
│││││├┬ M2: <module 'keyword' from '/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/keyword.py'>
││││││└ # M2 [10 B]
│││││├┬ M2: <module 'builtins' (built-in)>
││││││└ # M2 [23 B]
│││││├┬ M2: <module 'functools' from '/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/functools.py'>
││││││└ # M2 [10 B]
│││││├┬ M2: <module '_thread' (built-in)>
││││││└ # M2 [22 B]
│││││├┬ T4: <class 'dataclasses.FrozenInstanceError'>
││││││└ # T4 [35 B]
│││││├┬ T4: <class 'dataclasses._HAS_DEFAULT_FACTORY_CLASS'>
││││││└ # T4 [42 B]
│││││├┬ DcHDF: <factory>
││││││└ # DcHDF [34 B]
│││││├┬ T4: <class 'dataclasses._MISSING_TYPE'>
││││││└ # T4 [29 B]
│││││├┬ DcM: <dataclasses._MISSING_TYPE object at 0x10b265c90>
││││││└ # DcM [21 B]
│││││├┬ T4: <class 'dataclasses._FIELD_BASE'>
││││││└ # T4 [27 B]
│││││├┬ DcFB: _FIELD
││││││└ # DcFB [20 B]
│││││├┬ DcFB: _FIELD_CLASSVAR
││││││└ # DcFB [29 B]
│││││├┬ DcFB: _FIELD_INITVAR
││││││└ # DcFB [28 B]
│││││├┬ F2: <function _compile at 0x10b211170>
││││││└ # F2 [15 B]
│││││├┬ T4: <class 'dataclasses._InitVarMeta'>
││││││└ # T4 [28 B]
│││││├┬ T4: <class 'dataclasses.InitVar'>
││││││└ # T4 [23 B]
│││││├┬ F2: <function field at 0x10b2580e0>
││││││└ # F2 [21 B]
│││││├┬ F2: <function _tuple_str at 0x10b32fb90>
││││││└ # F2 [26 B]
│││││├┬ F2: <function _recursive_repr at 0x10b32f9e0>
││││││└ # F2 [31 B]
│││││├┬ F2: <function _create_fn at 0x10b370320>
││││││└ # F2 [29 B]
│││││├┬ F2: <function _field_assign at 0x10b3703b0>
││││││└ # F2 [32 B]
│││││├┬ F2: <function _field_init at 0x10b370440>
││││││└ # F2 [30 B]
│││││├┬ F2: <function _init_param at 0x10b3704d0>
││││││└ # F2 [30 B]
│││││├┬ F2: <function _init_fn at 0x10b370560>
││││││└ # F2 [27 B]
│││││├┬ F2: <function _repr_fn at 0x10b3705f0>
││││││└ # F2 [27 B]
│││││├┬ F2: <function _frozen_get_del_attr at 0x10b370680>
││││││└ # F2 [39 B]
│││││├┬ F2: <function _cmp_fn at 0x10b370710>
││││││└ # F2 [26 B]
│││││├┬ F2: <function _hash_fn at 0x10b3707a0>
││││││└ # F2 [27 B]
│││││├┬ F2: <function _is_classvar at 0x10b370830>
││││││└ # F2 [31 B]
│││││├┬ F2: <function _is_initvar at 0x10b3708c0>
││││││└ # F2 [30 B]
│││││├┬ F2: <function _is_type at 0x10b370950>
││││││└ # F2 [27 B]
│││││├┬ F2: <function _get_field at 0x10b3709e0>
││││││└ # F2 [29 B]
│││││├┬ F2: <function _set_new_attribute at 0x10b370a70>
││││││└ # F2 [37 B]
│││││├┬ F2: <function _hash_set_none at 0x10b370b00>
││││││└ # F2 [33 B]
│││││├┬ F2: <function _hash_add at 0x10b370b90>
││││││└ # F2 [28 B]
│││││├┬ F2: <function _hash_exception at 0x10b370c20>
││││││└ # F2 [34 B]
│││││├┬ D2: <dict object at 0x010b369dc0>
││││││└ # D2 [240 B]
│││││├┬ F2: <function _process_class at 0x10b370cb0>
││││││└ # F2 [33 B]
│││││├┬ F2: <function dataclass at 0x10b370d40>
││││││└ # F2 [28 B]
│││││├┬ F2: <function fields at 0x10b370dd0>
││││││└ # F2 [25 B]
│││││├┬ F2: <function _is_dataclass_instance at 0x10b370e60>
││││││└ # F2 [41 B]
│││││├┬ F2: <function is_dataclass at 0x10b370ef0>
││││││└ # F2 [31 B]
│││││├┬ F2: <function asdict at 0x10b370f80>
││││││└ # F2 [25 B]
│││││├┬ F2: <function _asdict_inner at 0x10b372050>
││││││└ # F2 [32 B]
│││││├┬ F2: <function astuple at 0x10b3720e0>
││││││└ # F2 [26 B]
│││││├┬ F2: <function _astuple_inner at 0x10b372170>
││││││└ # F2 [33 B]
│││││├┬ F2: <function make_dataclass at 0x10b372200>
││││││└ # F2 [33 B]
│││││├┬ F2: <function replace at 0x10b372290>
││││││└ # F2 [26 B]
│││││└ # F1 [4 MiB]
││││├┬ F1: <function __create_fn__.<locals>.__eq__ at 0x10b3a4c20>
│││││├┬ Co: <code object __eq__ at 0x10b396660, file "<string>", line 2>
││││││└ # Co [173 B]
│││││├┬ D1: <dict object at 0x010afc3eb0>
││││││└ # D1 [22 B]
│││││├┬ D2: <dict object at 0x010ba788c0>
││││││└ # D2 [6 B]
│││││├┬ D2: <dict object at 0x010ba810f0>
││││││├┬ D2: <dict object at 0x010ba78820>
│││││││└ # D2 [6 B]
││││││└ # D2 [57 B]
│││││└ # F1 [287 B]
││││└ # D2 [5 MiB]
│││└ # T2 [5 MiB]
││├┬ D2: <dict object at 0x010ba8bd20>
│││├┬ D2: <dict object at 0x010ba8bc30>
││││└ # D2 [26 B]
│││├┬ D2: <dict object at 0x010b237910>
││││└ # D2 [20 B]
│││├┬ F1: <function __create_fn__.<locals>.__init__ at 0x10b3a48c0>
││││├┬ Co: <code object __init__ at 0x10b396540, file "<string>", line 2>
│││││└ # Co [109 B]
││││├┬ D1: <dict object at 0x010afc3eb0>
│││││└ # D1 [22 B]
││││├┬ D2: <dict object at 0x010ba7ec80>
│││││└ # D2 [6 B]
││││├┬ D2: <dict object at 0x010ba6a820>
│││││├┬ D2: <dict object at 0x010b404410>
││││││└ # D2 [23 B]
│││││└ # D2 [76 B]
││││└ # F1 [246 B]
│││├┬ F1: <function __create_fn__.<locals>.__repr__ at 0x10b3a4560>
││││├┬ D2: <dict object at 0x010ba7ecd0>
│││││└ # D2 [11 B]
││││├┬ Ce2: <cell at 0x10b407090: set object at 0x10b350e60>
│││││└ # Ce2 [15 B]
││││├┬ Ce2: <cell at 0x10b407210: function object at 0x10b3a4680>
│││││└ # Ce2 [15 B]
││││├┬ D2: <dict object at 0x010b4048c0>
│││││├┬ F1: <function __create_fn__.<locals>.__repr__ at 0x10b3a4680>
││││││├┬ Co: <code object __repr__ at 0x10b3a4780, file "<string>", line 2>
│││││││└ # Co [170 B]
││││││├┬ D1: <dict object at 0x010afc3eb0>
│││││││└ # D1 [22 B]
││││││├┬ D2: <dict object at 0x010b392410>
│││││││└ # D2 [6 B]
││││││├┬ D2: <dict object at 0x010ba75230>
│││││││├┬ D2: <dict object at 0x010b4044b0>
││││││││└ # D2 [6 B]
│││││││└ # D2 [59 B]
││││││└ # F1 [283 B]
│││││└ # D2 [292 B]
││││├┬ D2: <dict object at 0x010ba75640>
│││││└ # D2 [22 B]
││││├┬ D4: <dict object at 0x010af42b90>
│││││└ # D4 [19 B]
││││├┬ DcHDF: <factory>
│││││└ # DcHDF [34 B]
││││├┬ DcM: <dataclasses._MISSING_TYPE object at 0x10b265c90>
│││││└ # DcM [21 B]
││││├┬ DcFB: _FIELD
│││││└ # DcFB [20 B]
││││├┬ DcFB: _FIELD_CLASSVAR
│││││└ # DcFB [29 B]
││││├┬ DcFB: _FIELD_INITVAR
│││││└ # DcFB [28 B]
││││└ # F1 [956 B]
│││├┬ F1: <function __create_fn__.<locals>.__eq__ at 0x10b3a4320>
││││├┬ Co: <code object __eq__ at 0x10b3a4a50, file "<string>", line 2>
│││││└ # Co [144 B]
││││├┬ D1: <dict object at 0x010afc3eb0>
│││││└ # D1 [22 B]
││││├┬ D2: <dict object at 0x010ba81320>
│││││└ # D2 [6 B]
││││├┬ D2: <dict object at 0x010ba755f0>
│││││├┬ D2: <dict object at 0x010ba812d0>
││││││└ # D2 [6 B]
│││││└ # D2 [57 B]
││││└ # F1 [258 B]
│││└ # D2 [1 MiB]
││└ # T2 [7 MiB]
│├┬ D2: <dict object at 0x010ba8b870>
││├┬ D2: <dict object at 0x010b404a00>
│││└ # D2 [73 B]
││├┬ F1: <function EmergeFile.str at 0x10b4033b0>
│││├┬ Co: <code object str at 0x10b3ba780, file "<stdin>", line 11>
││││└ # Co [97 B]
│││├┬ D1: <dict object at 0x010afc3eb0>
││││└ # D1 [22 B]
│││├┬ D2: <dict object at 0x010ba83c30>
││││└ # D2 [6 B]
│││├┬ D2: <dict object at 0x010b431d70>
││││├┬ D2: <dict object at 0x010ba83be0>
│││││└ # D2 [6 B]
││││└ # D2 [42 B]
│││└ # F1 [193 B]
││├┬ F1: <function EmergeFile.to_json at 0x10b403440>
│││├┬ Co: <code object to_json at 0x10b3a34b0, file "<stdin>", line 13>
││││├┬ Co: <code object <listcomp> at 0x10b3ba0c0, file "<stdin>", line 19>
│││││└ # Co [257 B]
││││├┬ Co: <code object <listcomp> at 0x10af94420, file "<stdin>", line 28>
│││││└ # Co [103 B]
││││└ # Co [859 B]
│││├┬ D1: <dict object at 0x010afc3eb0>
││││└ # D1 [22 B]
│││├┬ D2: <dict object at 0x010ba83fa0>
││││└ # D2 [6 B]
│││├┬ D2: <dict object at 0x010ba6a370>
││││├┬ D2: <dict object at 0x010ba83e60>
│││││└ # D2 [6 B]
││││└ # D2 [46 B]
│││└ # F1 [962 B]
││├┬ D2: <dict object at 0x010b412870>
│││└ # D2 [26 B]
││├┬ D2: <dict object at 0x010b404960>
│││├┬ D2: <dict object at 0x010ba906e0>
││││├┬ DcM: <dataclasses._MISSING_TYPE object at 0x10b265c90>
│││││└ # DcM [21 B]
││││├┬ DcFB: _FIELD
│││││└ # DcFB [20 B]
││││└ # D2 [81 B]
│││├┬ D2: <dict object at 0x010ba72640>
││││├┬ DcM: <dataclasses._MISSING_TYPE object at 0x10b265c90>
│││││└ # DcM [21 B]
││││├┬ DcFB: _FIELD
│││││└ # DcFB [20 B]
││││└ # D2 [81 B]
│││├┬ D2: <dict object at 0x010ba92c80>
││││├┬ DcM: <dataclasses._MISSING_TYPE object at 0x10b265c90>
│││││└ # DcM [21 B]
││││├┬ DcFB: _FIELD
│││││└ # DcFB [20 B]
││││└ # D2 [84 B]
│││├┬ D2: <dict object at 0x010ba92b40>
││││├┬ DcM: <dataclasses._MISSING_TYPE object at 0x10b265c90>
│││││└ # DcM [21 B]
││││├┬ DcFB: _FIELD
│││││└ # DcFB [20 B]
││││└ # D2 [87 B]
│││├┬ D2: <dict object at 0x010ba92d70>
││││├┬ DcM: <dataclasses._MISSING_TYPE object at 0x10b265c90>
│││││└ # DcM [21 B]
││││├┬ DcFB: _FIELD
│││││└ # DcFB [20 B]
││││└ # D2 [84 B]
│││├┬ D2: <dict object at 0x010ba92dc0>
││││├┬ DcM: <dataclasses._MISSING_TYPE object at 0x10b265c90>
│││││└ # DcM [21 B]
││││├┬ DcFB: _FIELD
│││││└ # DcFB [20 B]
││││└ # D2 [84 B]
│││├┬ D2: <dict object at 0x010ba92d20>
││││├┬ DcM: <dataclasses._MISSING_TYPE object at 0x10b265c90>
│││││└ # DcM [21 B]
││││├┬ DcFB: _FIELD
│││││└ # DcFB [20 B]
││││└ # D2 [84 B]
│││└ # D2 [743 B]
││├┬ F1: <function __create_fn__.<locals>.__init__ at 0x10b403680>
│││├┬ Co: <code object __init__ at 0x10b403540, file "<string>", line 2>
││││└ # Co [200 B]
│││├┬ D1: <dict object at 0x010afc3eb0>
││││└ # D1 [22 B]
│││├┬ D2: <dict object at 0x010ba87eb0>
││││└ # D2 [6 B]
│││├┬ D2: <dict object at 0x010ba92e10>
││││├┬ D2: <dict object at 0x010b404460>
│││││└ # D2 [56 B]
││││└ # D2 [109 B]
│││└ # F1 [394 B]
││├┬ F1: <function __create_fn__.<locals>.__repr__ at 0x10b403950>
│││├┬ D2: <dict object at 0x010ba92e60>
││││└ # D2 [11 B]
│││├┬ Ce2: <cell at 0x10b407dd0: set object at 0x10b350b90>
││││└ # Ce2 [15 B]
│││├┬ Ce2: <cell at 0x10b407d10: function object at 0x10b4038c0>
││││└ # Ce2 [15 B]
│││├┬ D2: <dict object at 0x010b4090f0>
││││├┬ F1: <function __create_fn__.<locals>.__repr__ at 0x10b4038c0>
│││││├┬ Co: <code object __repr__ at 0x10b4035d0, file "<string>", line 2>
││││││└ # Co [342 B]
│││││├┬ D1: <dict object at 0x010afc3eb0>
││││││└ # D1 [22 B]
│││││├┬ D2: <dict object at 0x010b4047d0>
││││││└ # D2 [6 B]
│││││├┬ D2: <dict object at 0x010ba92f50>
││││││├┬ D2: <dict object at 0x010b404d20>
│││││││└ # D2 [6 B]
││││││└ # D2 [59 B]
│││││└ # F1 [455 B]
││││└ # D2 [464 B]
│││├┬ D2: <dict object at 0x010ba92eb0>
││││└ # D2 [22 B]
│││├┬ D4: <dict object at 0x010af42b90>
││││└ # D4 [19 B]
│││├┬ DcHDF: <factory>
││││└ # DcHDF [34 B]
│││├┬ DcM: <dataclasses._MISSING_TYPE object at 0x10b265c90>
││││└ # DcM [21 B]
│││├┬ DcFB: _FIELD
││││└ # DcFB [20 B]
│││├┬ DcFB: _FIELD_CLASSVAR
││││└ # DcFB [29 B]
│││├┬ DcFB: _FIELD_INITVAR
││││└ # DcFB [28 B]
│││└ # F1 [1 MiB]
││├┬ F1: <function __create_fn__.<locals>.__eq__ at 0x10b403c20>
│││├┬ Co: <code object __eq__ at 0x10b403b70, file "<string>", line 2>
││││└ # Co [213 B]
│││├┬ D1: <dict object at 0x010afc3eb0>
││││└ # D1 [22 B]
│││├┬ D2: <dict object at 0x010ba88960>
││││└ # D2 [6 B]
│││├┬ D2: <dict object at 0x010ba92f00>
││││├┬ D2: <dict object at 0x010ba888c0>
│││││└ # D2 [6 B]
││││└ # D2 [57 B]
│││└ # F1 [327 B]
││└ # D2 [3 MiB]
│└ # T2 [11 MiB]
├┬ D2: <dict object at 0x010ba8ba50>
│├┬ D2: <dict object at 0x010b404dc0>
││├┬ T2: <class '__main__.Address'>
│││├┬ D2: <dict object at 0x010ba8c140>
││││├┬ D2: <dict object at 0x010b404b40>
│││││└ # D2 [59 B]
││││├┬ D2: <dict object at 0x010ba711e0>
│││││└ # D2 [26 B]
││││├┬ D2: <dict object at 0x010b404e60>
│││││├┬ D2: <dict object at 0x010ba71230>
││││││├┬ DcM: <dataclasses._MISSING_TYPE object at 0x10b265c90>
│││││││└ # DcM [21 B]
││││││├┬ DcFB: _FIELD
│││││││└ # DcFB [20 B]
││││││└ # D2 [84 B]
│││││├┬ D2: <dict object at 0x010ba71190>
││││││├┬ DcM: <dataclasses._MISSING_TYPE object at 0x10b265c90>
│││││││└ # DcM [21 B]
││││││├┬ DcFB: _FIELD
│││││││└ # DcFB [20 B]
││││││└ # D2 [84 B]
│││││├┬ D2: <dict object at 0x010ba71370>
││││││├┬ DcM: <dataclasses._MISSING_TYPE object at 0x10b265c90>
│││││││└ # DcM [21 B]
││││││├┬ DcFB: _FIELD
│││││││└ # DcFB [20 B]
││││││└ # D2 [84 B]
│││││└ # D2 [392 B]
││││├┬ F1: <function __create_fn__.<locals>.__init__ at 0x10b403dd0>
│││││├┬ Co: <code object __init__ at 0x10b4039c0, file "<string>", line 2>
││││││└ # Co [254 B]
│││││├┬ D1: <dict object at 0x010afc3eb0>
││││││└ # D1 [22 B]
│││││├┬ D2: <dict object at 0x010ba8be60>
││││││└ # D2 [6 B]
│││││├┬ D2: <dict object at 0x010ba92a50>
││││││├┬ D2: <dict object at 0x010b409190>
│││││││└ # D2 [77 B]
││││││└ # D2 [130 B]
│││││└ # F1 [475 B]
││││├┬ F1: <function __create_fn__.<locals>.__repr__ at 0x10b403f80>
│││││├┬ D2: <dict object at 0x010ba71640>
││││││└ # D2 [11 B]
│││││├┬ Ce2: <cell at 0x10b40d090: set object at 0x10b415140>
││││││└ # Ce2 [15 B]
│││││├┬ Ce2: <cell at 0x10b40d1d0: function object at 0x10b403ef0>
││││││└ # Ce2 [15 B]
│││││├┬ D2: <dict object at 0x010b409690>
││││││├┬ F1: <function __create_fn__.<locals>.__repr__ at 0x10b403ef0>
│││││││├┬ Co: <code object __repr__ at 0x10b403c90, file "<string>", line 2>
││││││││└ # Co [435 B]
│││││││├┬ D1: <dict object at 0x010afc3eb0>
││││││││└ # D1 [22 B]
│││││││├┬ D2: <dict object at 0x010b409c80>
││││││││└ # D2 [6 B]
│││││││├┬ D2: <dict object at 0x010ba71780>
││││││││├┬ D2: <dict object at 0x010b409410>
│││││││││└ # D2 [6 B]
││││││││└ # D2 [59 B]
│││││││└ # F1 [548 B]
││││││└ # D2 [557 B]
│││││├┬ D2: <dict object at 0x010ba716e0>
││││││└ # D2 [22 B]
│││││├┬ D4: <dict object at 0x010af42b90>
││││││└ # D4 [19 B]
│││││├┬ DcHDF: <factory>
││││││└ # DcHDF [34 B]
│││││├┬ DcM: <dataclasses._MISSING_TYPE object at 0x10b265c90>
││││││└ # DcM [21 B]
│││││├┬ DcFB: _FIELD
││││││└ # DcFB [20 B]
│││││├┬ DcFB: _FIELD_CLASSVAR
││││││└ # DcFB [29 B]
│││││├┬ DcFB: _FIELD_INITVAR
││││││└ # DcFB [28 B]
│││││└ # F1 [1 MiB]
││││├┬ F1: <function __create_fn__.<locals>.__eq__ at 0x10b4160e0>
│││││├┬ Co: <code object __eq__ at 0x10b403e40, file "<string>", line 2>
││││││└ # Co [252 B]
│││││├┬ D1: <dict object at 0x010afc3eb0>
││││││└ # D1 [22 B]
│││││├┬ D2: <dict object at 0x010ba8c730>
││││││└ # D2 [6 B]
│││││├┬ D2: <dict object at 0x010ba71460>
││││││├┬ D2: <dict object at 0x010ba8c690>
│││││││└ # D2 [6 B]
││││││└ # D2 [57 B]
│││││└ # F1 [366 B]
││││└ # D2 [2 MiB]
│││└ # T2 [2 MiB]
││└ # D2 [2 MiB]
│├┬ D2: <dict object at 0x010ba92fa0>
││└ # D2 [26 B]
│├┬ D2: <dict object at 0x010b409140>
││├┬ D2: <dict object at 0x010ba71870>
│││├┬ DcM: <dataclasses._MISSING_TYPE object at 0x10b265c90>
││││└ # DcM [21 B]
│││├┬ DcFB: _FIELD
││││└ # DcFB [20 B]
│││└ # D2 [81 B]
││├┬ D2: <dict object at 0x010ba71820>
│││├┬ DcM: <dataclasses._MISSING_TYPE object at 0x10b265c90>
││││└ # DcM [21 B]
│││├┬ DcFB: _FIELD
││││└ # DcFB [20 B]
│││└ # D2 [86 B]
││└ # D2 [275 B]
│├┬ F1: <function __create_fn__.<locals>.__init__ at 0x10b4163b0>
││├┬ Co: <code object __init__ at 0x10b416270, file "<string>", line 2>
│││└ # Co [218 B]
││├┬ D1: <dict object at 0x010afc3eb0>
│││└ # D1 [22 B]
││├┬ D2: <dict object at 0x010ba8baa0>
│││└ # D2 [6 B]
││├┬ D2: <dict object at 0x010ba718c0>
│││├┬ D2: <dict object at 0x010b409500>
││││└ # D2 [66 B]
│││└ # D2 [119 B]
││└ # F1 [423 B]
│├┬ F1: <function __create_fn__.<locals>.__repr__ at 0x10b416560>
││├┬ D2: <dict object at 0x010ba71910>
│││└ # D2 [11 B]
││├┬ Ce2: <cell at 0x10b40d850: set object at 0x10b415050>
│││└ # Ce2 [15 B]
││├┬ Ce2: <cell at 0x10b40d710: function object at 0x10b4164d0>
│││└ # Ce2 [15 B]
││├┬ D2: <dict object at 0x010b40b140>
│││├┬ F1: <function __create_fn__.<locals>.__repr__ at 0x10b4164d0>
││││├┬ Co: <code object __repr__ at 0x10b416420, file "<string>", line 2>
│││││└ # Co [375 B]
││││├┬ D1: <dict object at 0x010afc3eb0>
│││││└ # D1 [22 B]
││││├┬ D2: <dict object at 0x010b4094b0>
│││││└ # D2 [6 B]
││││├┬ D2: <dict object at 0x010ba71af0>
│││││├┬ D2: <dict object at 0x010b40b190>
││││││└ # D2 [6 B]
│││││└ # D2 [59 B]
││││└ # F1 [488 B]
│││└ # D2 [497 B]
││├┬ D2: <dict object at 0x010ba71960>
│││└ # D2 [22 B]
││├┬ D4: <dict object at 0x010af42b90>
│││└ # D4 [19 B]
││├┬ DcHDF: <factory>
│││└ # DcHDF [34 B]
││├┬ DcM: <dataclasses._MISSING_TYPE object at 0x10b265c90>
│││└ # DcM [21 B]
││├┬ DcFB: _FIELD
│││└ # DcFB [20 B]
││├┬ DcFB: _FIELD_CLASSVAR
│││└ # DcFB [29 B]
││├┬ DcFB: _FIELD_INITVAR
│││└ # DcFB [28 B]
││└ # F1 [1 MiB]
│├┬ F1: <function __create_fn__.<locals>.__eq__ at 0x10b416710>
││├┬ Co: <code object __eq__ at 0x10b4165d0, file "<string>", line 2>
│││└ # Co [226 B]
││├┬ D1: <dict object at 0x010afc3eb0>
│││└ # D1 [22 B]
││├┬ D2: <dict object at 0x010ba90820>
│││└ # D2 [6 B]
││├┬ D2: <dict object at 0x010ba71b40>
│││├┬ D2: <dict object at 0x010ba907d0>
││││└ # D2 [6 B]
│││└ # D2 [57 B]
││└ # F1 [340 B]
│└ # D2 [5 MiB]
├┬ M2: <module 'dill._dill' from '/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py'>
│└ # M2 [17 B]
├┬ T1: <class 'set'>
│└ # T1 [27 B]
└ # T2 [17 MiB]
┬ D2: <dict object at 0x010b40b1e0>
├┬ D2: <dict object at 0x010b409c30>
│└ # D2 [141 B]
└ # D2 [268 B]
Customer(id='customer1', data='', date='Jan 19 2023 10:42:48', name='johndoe', path='/customers', perms='rwxrwxrwx', type='file', uuid='', node='', address=Address(id='address1', data='', date='Jan 19 2023 10:42:48', name='', path='/', perms='rwxrwxrwx', type='file', uuid='', node='', street='Park Place', city='Quincy', state='MA'))
>>>
>>> dill.__version__
'0.3.7.dev0'

Note I'm using the latest dill with Python-3.7 on a MacOS... I'm not sure what version of dill, which python, and what OS you are using.

I noticed that you are using the persistent module, which sounds like it's intended to do some state preservation. If you are seeing issues with dill and persistent, it might be that persistent is doing something that is implemented specifically for pickle... you might want to reach out to the persistent developers and ask them about it.

Answer 7 · 2023-01-19T16:03:36.000Z

Here's my output. Address doesn't show up with dill. Let me get version #'s of my env.

{"id": "address1", "data": "", "date": "Jan 19 2023 09:31:27", "name": "", "path": "/", "perms": "rwxrwxrwx", "type": "file", "uuid": "", "node": "", "street": "Park Place", "city": "Quincy", "state": "MA"}
WITH DILL:
{"id": "customer1", "data": "", "date": "Jan 19 2023 09:31:27", "name": "johndoe", "path": "/customers", "perms": "rwxrwxrwx", "type": "file", "uuid": "", "node": "", "address": {}}
WITH PICKLE:
{"id": "customer1", "data": "", "date": "Jan 19 2023 09:31:27", "name": "johndoe", "path": "/customers", "perms": "rwxrwxrwx", "type": "file", "uuid": "", "node": "", "address": {"id": "address1", "data": "", "date": "Jan 19 2023 09:31:27", "name": "", "path": "/", "perms": "rwxrwxrwx", "type": "file", "uuid": "", "node": "", "street": "Park Place", "city": "Quincy", "state": "MA"}}

Answer 8 · 2023-01-19T16:04:31.000Z

Python 3.10.6
dill==0.3.5.1
OS=Pop_OS! (Ubuntu) 22.04

Answer 9 · 2023-01-19T16:14:54.000Z

You can see in my trace that dill is picking up the Address:

├┬ D2: <dict object at 0x010ba8ba50>
│├┬ D2: <dict object at 0x010b404dc0>
││├┬ T2: <class '__main__.Address'>

You might try to update your dill and see if that does anything. You are at least one release behind the latest dill. You might also try using dill.settings['recurse'] = True, and turning on the trace to see if there's something we can see that's indicates what's going on for you.

Answer 10 · 2023-01-19T16:24:32.000Z

I will try updating it. but you seem to be using a dev branch/version of dill and an old/obsolete version of python as well.

Answer 11 · 2023-01-19T16:25:07.000Z

I would like to see what happens when you run my example code same as me and share the output vs patching it into python >>> prompt

Answer 12 · 2023-01-19T16:26:22.000Z

Upgrading dill fixed it. So it was an issue with dill. (broken) 0.3.5.1, upgrade to 0.3.6 (works)

Answer 13 · 2023-01-19T17:25:26.000Z

I generally test against all supported versions (3.7 - 3.12), I just cut and pasted the 3.7 output.
Yeah, I do test with the dev version of dill first, specifically to see if a upgrade may be the best solution. :)

Thanks for reporting.