threading.Thread fails to pickle in python > 3.13.0a5
Opened this issue · 4 comments
mmckerns commented
Starting in python 3.13.0a5, a threading.Thread instance is not serializable. This is due to the thread attribute pointing to an unserializable _thread._ThreadHandle instance, where in earlier versions the _thread instance pointed to None. Deleting the _ThreadHandle enables the Thread to be serializable again. Presumably, there's a good way to obtain an appropriate thread handle from the thread or using the threading module.
Python 3.13.0a5 (main, Mar 16 2024, 18:36:37) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import threading
>>> t = threading.Thread()
>>> t.start()
>>> import dill
>>> dill.dumps(t)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
dill.dumps(t)
~~~~~~~~~~^^^
File "/Users/mmckerns/lib/python3.13/site-packages/dill/_dill.py", line 280, in dumps
dump(obj, file, protocol, byref, fmode, recurse, **kwds)#, strictio)
~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mmckerns/lib/python3.13/site-packages/dill/_dill.py", line 252, in dump
Pickler(file, protocol, **_kwds).dump(obj)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^
File "/Users/mmckerns/lib/python3.13/site-packages/dill/_dill.py", line 420, in dump
StockPickler.dump(self, obj)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/pickle.py", line 483, in dump
self.save(obj)
~~~~~~~~~^^^^^
File "/Users/mmckerns/lib/python3.13/site-packages/dill/_dill.py", line 414, in save
StockPickler.save(self, obj, save_persistent_id)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/pickle.py", line 599, in save
self.save_reduce(obj=obj, *rv)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/pickle.py", line 713, in save_reduce
save(state)
~~~~^^^^^^^
File "/Users/mmckerns/lib/python3.13/site-packages/dill/_dill.py", line 414, in save
StockPickler.save(self, obj, save_persistent_id)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/pickle.py", line 556, in save
f(self, obj) # Call unbound method with explicit self
~^^^^^^^^^^^
File "/Users/mmckerns/lib/python3.13/site-packages/dill/_dill.py", line 1217, in save_module_dict
StockPickler.save_dict(pickler, obj)
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/pickle.py", line 968, in save_dict
self._batch_setitems(obj.items())
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/pickle.py", line 992, in _batch_setitems
save(v)
~~~~^^^
File "/Users/mmckerns/lib/python3.13/site-packages/dill/_dill.py", line 414, in save
StockPickler.save(self, obj, save_persistent_id)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/pickle.py", line 574, in save
rv = reduce(self.proto)
~~~~~~^^^^^^^^^^^^
TypeError: cannot pickle '_thread._ThreadHandle' object
>>> t.__dict__
{'_name': 'Thread-1', '_daemonic': False, '_ident': 123145514024960, '_native_id': 9483263, '_tstate_lock': <unlocked _thread.lock object at 0x1107a61c0>, '_handle': <_thread._ThreadHandle object: ident=123145514024960>, '_started': <threading.Event at 0x10fdfdf10: set>, '_is_stopped': False, '_initialized': True, '_stderr': <_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>, '_invoke_excepthook': <function _make_invoke_excepthook.<locals>.invoke_excepthook at 0x110772ca0>}
>>> t._handle = None
>>> pt = dill.dumps(t)
>>> _t = dill.loads(pt)
>>>
mmckerns commented
while it's not directly related, might be an opportunity to deal with objects in #334
musicinmybrain commented
The current failure on Python 3.13.0b1 looks like this:
$ gh repo clone uqfoundation/dill
$ cd dill
$ python3.13 -V
Python 3.13.0b1
$ tox -e py313
.pkg: _optional_hooks> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_sdist> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_wheel> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: prepare_metadata_for_build_wheel> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: build_sdist> python /usr/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta
py313: install_package> python -I -m pip install --force-reinstall --no-deps /home/ben/src/forks/dill/.tox/.tmp/package/2/dill-0.3.9.dev0.tar.gz
py313: commands[0]> .tox/py313/bin/python -m pip install .
Processing /home/ben/src/forks/dill
Installing build dependencies ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... done
Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: dill
Building wheel for dill (pyproject.toml) ... done
Created wheel for dill: filename=dill-0.3.9.dev0-py3-none-any.whl size=116162 sha256=86a3739b5a701f9aefd7840a31f695eefa0e8408a55028bfe497423a5d87d061
Stored in directory: /tmp/pip-ephem-wheel-cache-sukup0uc/wheels/4b/69/4e/74fd16e96eb0e05af8c484f673696525e999353b5fec3ad50d
Successfully built dill
Installing collected packages: dill
Attempting uninstall: dill
Found existing installation: dill 0.3.9.dev0
Uninstalling dill-0.3.9.dev0:
Successfully uninstalled dill-0.3.9.dev0
Successfully installed dill-0.3.9.dev0
py313: commands[1]> .tox/py313/bin/python dill/tests/__main__.py
.<function <lambda> at 0x7fecfbe8be20>
<function <lambda> at 0x7f937fe8be20>
<function <lambda> at 0x7fed0408be20>
<function <lambda> at 0x7fbd8328be20>
<function <lambda> at 0x7f2a7ea8be20>
Traceback (most recent call last):
File "/home/ben/src/forks/dill/dill/tests/test_detect.py", line 154, in <module>
test_bad_things()
~~~~~~~~~~~~~~~^^
File "/home/ben/src/forks/dill/dill/tests/test_detect.py", line 32, in test_bad_things
assert len(s) is len(a) # TypeError (and possibly PicklingError)
^^^^^^^^^^^^^^^^
AssertionError
...F.......................
py313: exit 1 (1.79 seconds) /home/ben/src/forks/dill> .tox/py313/bin/python dill/tests/__main__.py pid=182079
py313: FAIL code 1 (6.48=setup[2.72]+cmd[1.97,1.79] seconds)
evaluation failed :( (6.53 seconds)
musicinmybrain commented
s is:
{('TypeError', "cannot pickle 'frame' object"), ('TypeError', "cannot pickle 'FrameLocalsProxy' object")}
and a is:
{'TypeError': "cannot pickle 'FrameLocalsProxy' object"}
which appears related in some way to PEP 667.
mmckerns commented
If I remember correctly -- from more than a decade ago -- the primary reason that a frame object couldn't be pickled is that the GIL is imbedded in the frame object and could only be written to in C. In python 3.13, there have been a lot of recent changes to support removal of the GIL. I expect it's related to that.