Attempting to `dill` a function defined in a `doctest` run with `pytest` causes a `TypeError: cannot pickle 'EncodedFile' object`.
qthequartermasterman opened this issue · 0 comments
This is cross posted from pytest-dev/pytest#12448, because the pytest devs suggest that this is a problem with dill.
Description of the Problem
When I attempt to dill a function defined inside of a doctest, pytest raises an error, complaining that TypeError: cannot pickle 'EncodedFile' object. Best I can tell, this is the EncodedFile object in pytest.
This error only occurs when I run those doctests via pytest. They run just fine via vanilla doctest. The same code also works just fine in a script and as a regular pytest test function.
I fear that this is related to pytest-dev/pytest#10845, which doesn't seem to have a solution.
Minimal Reproducible Example (Serialization)
class MyClass:
"""Running a file containing this class with `python -m pytest this_file.py --doctest-modules` will fail with `TypeError: cannot pickle 'EncodedFile' object`.
Examples:
>>> def template_function():
... return "Hello, World!"
>>> import dill
>>> dill.dumps(template_function)
"""Using pytest to run those doctests will cause the TypeError: cannot pickle 'EncodedFile' object.
Using vanilla doctest works fine. (Ignore the fact that the test failed. The expected output did not match. Note that it did successfully dump the function using dill.)
Failed example:
dill.dumps(template_function)
Expected nothing
Got:
b'\x80\x04\x95L\x03\x00\x00\x00\x00\x00\x00\x8c\ndill.dill\x94\x8c\x10_create_function\x94\x93\x94(h\x00\x8c\x0c_create_code\x94\x93\x94(C\x02\x02\x01\x94K\x00K\x00K\x00K\x00K\x01K\x03C\x06\x97\x00d\x01S\x00\x94N\x8c\rHello, World!\x94\x86\x94))\x8c <doctest test_client.MyClass[0]>\x94\x8c\x11template_function\x94h\nK\x01C\x07\x80\x00\xd8\x0b\x1a\x88?\x94C\x00\x94))t\x94R\x94}\x94\x8c\x08__name_\x94\x8c\x0btest_client\x94sh\nNNt\x94R\x94}\x94}\x94\x8c\x0f__annotations__\x94}\x94s\x86\x94bh\x0f(h\x10h\x11\x8c\x07__doc__\x94N\x8c\x0b__package__\x94\x8c\x00\x94\x8c\n__loader__\x94\x8c\x1a_frozen_importlib_external\x94\x8c\x10SourceFileLoader\x94\x93\x94)\x81\x94}\x94(\x8c\x04name\x94h\x11\x8c\x04path\x94\x8c@/Users/username/test_client.py\x94ub\x8c\x08__spec__\x94\x8c\x11_frozen_importlib\x94\x8c\nModuleSpec\x94\x93\x94)\x81\x94}\x94(h"h\x11\x8c\x06loader\x94h \x8c\x06origin\x94h$\x8c\x0cloader_state\x94N\x8c\x1asubmodule_search_locations\x94N\x8c\x19_uninitialized_submodules\x94]\x94\x8c\r_set_fileattr\x94\x88\x8c\x07_cached\x94\x8cY/Users/username/pycache/test_client.cpython-311.pyc\x94\x8c\r_initializing\x94\x89ub\x8c\x08__file__\x94h$\x8c\n__cached__\x94h3\x8c\x0c__builtins__\x94cbuiltins\n__dict__\n\x8c\x07MyClass\x94h\x11h8\x93\x94h\nh\x13\x8c\x04dill\x94h\x00\x8c\x0e_import_module\x94\x93\x94h:\x85\x94R\x94u0.'
1 items had failures:
1 of 3 in test_client.MyClass
Using a regular pytest function to perform the test also works fine. This is only when attempting to dill a function defined in a doctest run with pytest.
Minimal Reproducible Example (Deserialization after disabling pytest's logging capture)
If I extend the example to include dill.loads, and then disable capture with pytest -s, I get a recursion error. The serialization is successful (because disabling capture means that EncodedFile isn't injected), but dill is trying to load something recursively, I think? Again, this code works as expected with vanilla doctest and in a script.
class MyClass:
"""Running a file containing this class with `python -m pytest -s this_file.py --doctest-modules` will fail with `RecursionError('maximum recursion depth exceeded').
Examples:
>>> def template_function():
... return "Hello, World!"
>>> import dill
>>> string = dill.dumps(template_function)
>>> dill.loads(string)()
'Hello, World!'
"""010 >>> string = dill.dumps(template_function)
011 >>> dill.loads(string)()
UNEXPECTED EXCEPTION: RecursionError('maximum recursion depth exceeded')
Traceback (most recent call last):
File "/opt/homebrew/Caskroom/miniforge/base/envs/my-env/lib/python3.11/doctest.py", line 1350, in __run
exec(compile(example.source, filename, "single",
File "<doctest test_client.MyClass[3]>", line 1, in <module>
File "/opt/homebrew/Caskroom/miniforge/base/envs/my-env/lib/python3.11/site-packages/dill/_dill.py", line 286, in loads
return load(file, ignore, **kwds)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniforge/base/envs/my-env/lib/python3.11/site-packages/dill/_dill.py", line 272, in load
return Unpickler(file, ignore=ignore, **kwds).load()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniforge/base/envs/my-env/lib/python3.11/site-packages/dill/_dill.py", line 419, in load
obj = StockUnpickler.load(self)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniforge/base/envs/my-env/lib/python3.11/site-packages/pluggy/_manager.py", line 74, in __getattr__
return getattr(self._dist, attr, default)
^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniforge/base/envs/my-env/lib/python3.11/site-packages/pluggy/_manager.py", line 74, in __getattr__
return getattr(self._dist, attr, default)
^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniforge/base/envs/my-env/lib/python3.11/site-packages/pluggy/_manager.py", line 74, in __getattr__
return getattr(self._dist, attr, default)
^^^^^^^^^^
[Previous line repeated 955 more times]
RecursionError: maximum recursion depth exceeded
Environment
This was run on Pytest 8.2.2, Python 3.11 on an M1 Macbook Air (macOS Sonoma 14.4.1). The same errors were also seen on Python 3.9 on the same Macbook and on Python 3.11 on an amd64 Ubuntu Machine (Ubuntu 22.04 LTS). All experiments were run with dill==0.3.8.
Pip list
Package Version
---------- -------
coverage 7.5.3
dill 0.3.8
iniconfig 2.0.0
packaging 24.1
pip 24.0
pluggy 1.5.0
pytest 8.2.2
pytest-cov 5.0.0
setuptools 70.0.0
uv 0.2.10
wheel 0.43.0