Attempting to `dill` a function defined in a `doctest` run with `pytest` causes a `TypeError: cannot pickle 'EncodedFile' object`.
qthequartermasterman opened this issue · 6 comments
- a detailed description of the bug or problem you are having
- output of
pip listfrom the virtual environment you are using - pytest and operating system versions
- minimal example if possible
Description of the Problem
When I attempt to dill a function defined inside of a doctest, pytest raises an error, complaining that TypeError: cannot pickle 'EncodedFile' object. Best I can tell, this is the EncodedFile object in pytest.
This error only occurs when I run those doctests via pytest. They run just fine via vanilla doctest. The same code also works just fine in a script and as a regular pytest test function.
I fear that this is related to #10845, which doesn't seem to have a solution.
Minimal Reproducible Example
class MyClass:
"""Running a file containing this class with `python -m pytest this_file.py --doctest-modules` will fail with `TypeError: cannot pickle 'EncodedFile' object`.
Examples:
>>> def template_function():
... return "Hello, World!"
>>> import dill
>>> dill.dumps(template_function)
"""Using pytest to run those doctests will cause the TypeError: cannot pickle 'EncodedFile' object.
Using vanilla doctest works fine. (Ignore the fact that the test failed. The expected output did not match. Note that it did successfully dump the function using dill.)
Failed example:
dill.dumps(template_function)
Expected nothing
Got:
b'\x80\x04\x95L\x03\x00\x00\x00\x00\x00\x00\x8c\ndill.dill\x94\x8c\x10_create_function\x94\x93\x94(h\x00\x8c\x0c_create_code\x94\x93\x94(C\x02\x02\x01\x94K\x00K\x00K\x00K\x00K\x01K\x03C\x06\x97\x00d\x01S\x00\x94N\x8c\rHello, World!\x94\x86\x94))\x8c <doctest test_client.MyClass[0]>\x94\x8c\x11template_function\x94h\nK\x01C\x07\x80\x00\xd8\x0b\x1a\x88?\x94C\x00\x94))t\x94R\x94}\x94\x8c\x08__name_\x94\x8c\x0btest_client\x94sh\nNNt\x94R\x94}\x94}\x94\x8c\x0f__annotations__\x94}\x94s\x86\x94bh\x0f(h\x10h\x11\x8c\x07__doc__\x94N\x8c\x0b__package__\x94\x8c\x00\x94\x8c\n__loader__\x94\x8c\x1a_frozen_importlib_external\x94\x8c\x10SourceFileLoader\x94\x93\x94)\x81\x94}\x94(\x8c\x04name\x94h\x11\x8c\x04path\x94\x8c@/Users/username/test_client.py\x94ub\x8c\x08__spec__\x94\x8c\x11_frozen_importlib\x94\x8c\nModuleSpec\x94\x93\x94)\x81\x94}\x94(h"h\x11\x8c\x06loader\x94h \x8c\x06origin\x94h$\x8c\x0cloader_state\x94N\x8c\x1asubmodule_search_locations\x94N\x8c\x19_uninitialized_submodules\x94]\x94\x8c\r_set_fileattr\x94\x88\x8c\x07_cached\x94\x8cY/Users/username/pycache/test_client.cpython-311.pyc\x94\x8c\r_initializing\x94\x89ub\x8c\x08__file__\x94h$\x8c\n__cached__\x94h3\x8c\x0c__builtins__\x94cbuiltins\n__dict__\n\x8c\x07MyClass\x94h\x11h8\x93\x94h\nh\x13\x8c\x04dill\x94h\x00\x8c\x0e_import_module\x94\x93\x94h:\x85\x94R\x94u0.'
1 items had failures:
1 of 3 in test_client.MyClass
Using a regular pytest function to perform the test also works fine. This is only when attempting to dill a function defined in a doctest run with pytest.
Pytest and Operating System versions
This was run on Pytest 8.2.2, Python 3.11 on an M1 Macbook Air (macOS Sonoma 14.4.1). The same errors were also seen on Python 3.9 on the same Macbook and on Python 3.11 on an amd64 Ubuntu Machine (Ubuntu 22.04 LTS).
Pip list
Package Version
---------- -------
coverage 7.5.3
dill 0.3.8
iniconfig 2.0.0
packaging 24.1
pip 24.0
pluggy 1.5.0
pytest 8.2.2
pytest-cov 5.0.0
setuptools 70.0.0
uv 0.2.10
wheel 0.43.0
This is also probably relevant. If I extend the example to include dill.loads, and then disable capture with pytest -s, I get a recursion error. Again, this code works as expected with vanilla doctest and in a script.
class MyClass:
"""Running a file containing this class with `python -m pytest this_file.py --doctest-modules` will fail with `TypeError: cannot pickle 'EncodedFile' object`.
Examples:
>>> def template_function():
... return "Hello, World!"
>>> import dill
>>> string = dill.dumps(template_function)
>>> dill.loads(string)()
'Hello, World!'
"""010 >>> string = dill.dumps(template_function)
011 >>> dill.loads(string)()
UNEXPECTED EXCEPTION: RecursionError('maximum recursion depth exceeded')
Traceback (most recent call last):
File "/opt/homebrew/Caskroom/miniforge/base/envs/my-env/lib/python3.11/doctest.py", line 1350, in __run
exec(compile(example.source, filename, "single",
File "<doctest test_client.MyClass[3]>", line 1, in <module>
File "/opt/homebrew/Caskroom/miniforge/base/envs/my-env/lib/python3.11/site-packages/dill/_dill.py", line 286, in loads
return load(file, ignore, **kwds)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniforge/base/envs/my-env/lib/python3.11/site-packages/dill/_dill.py", line 272, in load
return Unpickler(file, ignore=ignore, **kwds).load()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniforge/base/envs/my-env/lib/python3.11/site-packages/dill/_dill.py", line 419, in load
obj = StockUnpickler.load(self)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniforge/base/envs/my-env/lib/python3.11/site-packages/pluggy/_manager.py", line 74, in __getattr__
return getattr(self._dist, attr, default)
^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniforge/base/envs/my-env/lib/python3.11/site-packages/pluggy/_manager.py", line 74, in __getattr__
return getattr(self._dist, attr, default)
^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniforge/base/envs/my-env/lib/python3.11/site-packages/pluggy/_manager.py", line 74, in __getattr__
return getattr(self._dist, attr, default)
^^^^^^^^^^
[Previous line repeated 955 more times]
RecursionError: maximum recursion depth exceeded
Does it work when IO capture is disabled? pytest -s
Does it work when IO capture is disabled?
pytest -s
I get a different error during unpickling that also only occurs with pytest. You can see my second post in this issue for those details.
Based on the traceback dill pickles random objects
We might be able to give better errors but we cant fix dill doing things that are fragile
Facing the same issue. Something's off with module parser: running pytest tests/subdir passes
(.venv) user:~/proj/proj-agent $ pytest tests/agent/ PASSED
(.venv) user:~/proj/proj-agent [j1] [!1] $ pytest tests/agent/
==================================================================== test session starts ====================================================================
platform darwin -- Python 3.11.0, pytest-8.2.1, pluggy-1.5.0
rootdir: /Users/user/proj/proj-agent
configfile: pytest.ini
plugins: anyio-4.3.0, Faker-25.6.0, asyncio-0.23.7, mock-3.14.0, xdist-3.6.1
asyncio: mode=Mode.STRICT
collected 7 items
tests/agent/tools/test_registry.py ... [ 42%]
tests/agent/tools/test_tool.py .... [100%]
===================================================================== 7 passed in 0.27s =====================================================================
whereas running bare pytest without a path specified - thus project root - will have dill tests fail:
(.venv) user:~/proj/proj-agent [j1] $ pytest -k test_tool FAIL
(.venv) user:~/proj/proj-agent [j1] $ pytest -k test_tool
==================================================================== test session starts ====================================================================
platform darwin -- Python 3.11.0, pytest-8.2.1, pluggy-1.5.0
rootdir: /Users/user/proj/proj-agent
configfile: pytest.ini
plugins: anyio-4.3.0, Faker-25.6.0, asyncio-0.23.7, mock-3.14.0, xdist-3.6.1
asyncio: mode=Mode.STRICT
collected 128 items / 124 deselected / 4 selected
tests/agent/tools/test_tool.py F.FF [100%]
========================================================================= FAILURES ==========================================================================
_____________________________________________________________________ test_tool_wrapper _____________________________________________________________________
fn = <function fn.<locals>._fn at 0x187c70c20>
def test_tool_wrapper(fn):
> otool = Tool.wrap(fn)
tests/agent/tools/test_tool.py:19:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
proj/agent/tools/tool.py:64: in wrap
payload=dill.dumps(obj),
.venv/lib/python3.11/site-packages/dill/_dill.py:280: in dumps
dump(obj, file, protocol, byref, fmode, recurse, **kwds)#, strictio)
.venv/lib/python3.11/site-packages/dill/_dill.py:252: in dump
Pickler(file, protocol, **_kwds).dump(obj)
.venv/lib/python3.11/site-packages/dill/_dill.py:420: in dump
StockPickler.dump(self, obj)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:487: in dump
self.save(obj)
.venv/lib/python3.11/site-packages/dill/_dill.py:414: in save
StockPickler.save(self, obj, save_persistent_id)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:560: in save
f(self, obj) # Call unbound method with explicit self
.venv/lib/python3.11/site-packages/dill/_dill.py:1985: in save_function
_save_with_postproc(pickler, (_create_function, (
.venv/lib/python3.11/site-packages/dill/_dill.py:1112: in _save_with_postproc
pickler._batch_setitems(iter(source.items()))
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:998: in _batch_setitems
save(v)
.venv/lib/python3.11/site-packages/dill/_dill.py:414: in save
StockPickler.save(self, obj, save_persistent_id)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:603: in save
self.save_reduce(obj=obj, *rv)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:717: in save_reduce
save(state)
.venv/lib/python3.11/site-packages/dill/_dill.py:414: in save
StockPickler.save(self, obj, save_persistent_id)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:560: in save
f(self, obj) # Call unbound method with explicit self
.venv/lib/python3.11/site-packages/dill/_dill.py:1217: in save_module_dict
StockPickler.save_dict(pickler, obj)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:972: in save_dict
self._batch_setitems(obj.items())
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:998: in _batch_setitems
save(v)
.venv/lib/python3.11/site-packages/dill/_dill.py:414: in save
StockPickler.save(self, obj, save_persistent_id)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:603: in save
self.save_reduce(obj=obj, *rv)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:717: in save_reduce
save(state)
.venv/lib/python3.11/site-packages/dill/_dill.py:414: in save
StockPickler.save(self, obj, save_persistent_id)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:560: in save
f(self, obj) # Call unbound method with explicit self
.venv/lib/python3.11/site-packages/dill/_dill.py:1217: in save_module_dict
StockPickler.save_dict(pickler, obj)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:972: in save_dict
self._batch_setitems(obj.items())
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:998: in _batch_setitems
save(v)
.venv/lib/python3.11/site-packages/dill/_dill.py:414: in save
StockPickler.save(self, obj, save_persistent_id)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:603: in save
self.save_reduce(obj=obj, *rv)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:717: in save_reduce
save(state)
.venv/lib/python3.11/site-packages/dill/_dill.py:414: in save
StockPickler.save(self, obj, save_persistent_id)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:560: in save
f(self, obj) # Call unbound method with explicit self
.venv/lib/python3.11/site-packages/dill/_dill.py:1217: in save_module_dict
StockPickler.save_dict(pickler, obj)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:972: in save_dict
self._batch_setitems(obj.items())
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:998: in _batch_setitems
save(v)
.venv/lib/python3.11/site-packages/dill/_dill.py:414: in save
StockPickler.save(self, obj, save_persistent_id)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:560: in save
f(self, obj) # Call unbound method with explicit self
.venv/lib/python3.11/site-packages/dill/_dill.py:1217: in save_module_dict
StockPickler.save_dict(pickler, obj)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:972: in save_dict
self._batch_setitems(obj.items())
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:998: in _batch_setitems
save(v)
.venv/lib/python3.11/site-packages/dill/_dill.py:414: in save
StockPickler.save(self, obj, save_persistent_id)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:603: in save
self.save_reduce(obj=obj, *rv)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:717: in save_reduce
save(state)
.venv/lib/python3.11/site-packages/dill/_dill.py:414: in save
StockPickler.save(self, obj, save_persistent_id)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:560: in save
f(self, obj) # Call unbound method with explicit self
.venv/lib/python3.11/site-packages/dill/_dill.py:1217: in save_module_dict
StockPickler.save_dict(pickler, obj)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:972: in save_dict
self._batch_setitems(obj.items())
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:998: in _batch_setitems
save(v)
.venv/lib/python3.11/site-packages/dill/_dill.py:414: in save
StockPickler.save(self, obj, save_persistent_id)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:603: in save
self.save_reduce(obj=obj, *rv)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:717: in save_reduce
save(state)
.venv/lib/python3.11/site-packages/dill/_dill.py:414: in save
StockPickler.save(self, obj, save_persistent_id)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:560: in save
f(self, obj) # Call unbound method with explicit self
.venv/lib/python3.11/site-packages/dill/_dill.py:1217: in save_module_dict
StockPickler.save_dict(pickler, obj)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:972: in save_dict
self._batch_setitems(obj.items())
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:998: in _batch_setitems
save(v)
.venv/lib/python3.11/site-packages/dill/_dill.py:414: in save
StockPickler.save(self, obj, save_persistent_id)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:603: in save
self.save_reduce(obj=obj, *rv)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:717: in save_reduce
save(state)
.venv/lib/python3.11/site-packages/dill/_dill.py:414: in save
StockPickler.save(self, obj, save_persistent_id)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:560: in save
f(self, obj) # Call unbound method with explicit self
.venv/lib/python3.11/site-packages/dill/_dill.py:1217: in save_module_dict
StockPickler.save_dict(pickler, obj)
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:972: in save_dict
self._batch_setitems(obj.items())
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:998: in _batch_setitems
save(v)
.venv/lib/python3.11/site-packages/dill/_dill.py:414: in save
StockPickler.save(self, obj, save_persistent_id)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <dill._dill.Pickler object at 0x187c27e90>, obj = <_io.TextIOWrapper name="<_io.FileIO name=6 mode='rb+' closefd=True>" mode='r+' encoding='utf-8'>
save_persistent_id = True
def save(self, obj, save_persistent_id=True):
self.framer.commit_frame()
# Check for persistent id (defined by a subclass)
pid = self.persistent_id(obj)
if pid is not None and save_persistent_id:
self.save_pers(pid)
return
# Check the memo
x = self.memo.get(id(obj))
if x is not None:
self.write(self.get(x[0]))
return
rv = NotImplemented
reduce = getattr(self, "reducer_override", None)
if reduce is not None:
rv = reduce(obj)
if rv is NotImplemented:
# Check the type dispatch table
t = type(obj)
f = self.dispatch.get(t)
if f is not None:
f(self, obj) # Call unbound method with explicit self
return
# Check private dispatch table if any, or else
# copyreg.dispatch_table
reduce = getattr(self, 'dispatch_table', dispatch_table).get(t)
if reduce is not None:
rv = reduce(obj)
else:
# Check for a class with a custom metaclass; treat as regular
# class
if issubclass(t, type):
self.save_global(obj)
return
# Check for a __reduce_ex__ method, fall back to __reduce__
reduce = getattr(obj, "__reduce_ex__", None)
if reduce is not None:
> rv = reduce(self.proto)
E TypeError: cannot pickle 'EncodedFile' object
../../.pyenv/versions/3.11.0/lib/python3.11/pickle.py:578: TypeError
The pickled subject being a function referenced from within a fixture's inner scope
@pytest.fixture
def fn():
def _fn(*args):
import functools
if not args:
return 0
return functools.reduce(lambda x, y: x + y, args)
return _fn
Other observations
- Adding
__init__.pytotests/and subdirs make tests fail with the same error no mater what path is specified - ...
You need to disable assertion rewrite for at least the module
And then hope
It's generally unsafe to serialize inner functions in any way
Dill is playing with fire there and it breaks whenever someone throws in objects that aren't serializable