pytest-cov fails with pytest-xdist and dynamic_context = "test_function"

Question

pytest-cov fails with pytest-xdist and dynamic_context = "test_function"

masaccio opened this issue a year ago · 2 comments

Summary

I am trying to use pytest-cov fails with pytest-xdist and want to get coverage for each trace function to ultimately see which lines and arcs are covered by which test function (which I'll then use to create a minimal set of tests that generate n% coverage eliminating redundant tests).

I can use a static context with --cov-context=test together with pytest-xdist but when I try dynamic_context = "test_function" I get an internal error.

Reproducer

This works:

git clone git@github.com:masaccio/numbers-parser.git
poetry install
poetry run pytest -n logical tests/test_version.py

The relevant sections of my pyproject.toml are:

[tool.coverage.run]
branch = true
# dynamic_context = "test_function"
omit = ["src/numbers_parser/generated/*.py"]

[tool.coverage.html]
directory = "coverage_html_report"
show_contexts = true

[tool.pytest.ini_options]
addopts = "--cov=src/numbers_parser --cov-report=term-missing:skip-covered --cov-context=test"

Edit pyproject.toml to remove the comment for dynamic_context. I also removed --cov-context=test but that doesn't make any difference in that the internal error still happens.

[tool.coverage.run]
branch = true
dynamic_context = "test_function"
omit = ["src/numbers_parser/generated/*.py"]

Then re-run poetry run pytest -n logical tests/test_version.py and I observe an internal error:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/coverage/sqldata.py", line 1173, in _execute
INTERNALERROR>     return self.con.execute(sql, parameters)    # type: ignore[arg-type]
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> sqlite3.OperationalError: no such table: file
INTERNALERROR> 
INTERNALERROR> During handling of the above exception, another exception occurred:
INTERNALERROR> 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/coverage/sqldata.py", line 1178, in _execute
INTERNALERROR>     return self.con.execute(sql, parameters)    # type: ignore[arg-type]
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> sqlite3.OperationalError: no such table: file
INTERNALERROR> 
INTERNALERROR> The above exception was the direct cause of the following exception:
INTERNALERROR> 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/_pytest/main.py", line 270, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/_pytest/main.py", line 324, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/pluggy/_hooks.py", line 433, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/pluggy/_manager.py", line 112, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/pluggy/_callers.py", line 133, in _multicall
INTERNALERROR>     teardown[0].send(outcome)
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/pytest_cov/plugin.py", line 298, in pytest_runtestloop
INTERNALERROR>     self.cov_controller.finish()
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/pytest_cov/engine.py", line 44, in ensure_topdir_wrapper
INTERNALERROR>     return meth(self, *args, **kwargs)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/pytest_cov/engine.py", line 348, in finish
INTERNALERROR>     self.cov.save()
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/coverage/control.py", line 757, in save
INTERNALERROR>     data = self.get_data()
INTERNALERROR>            ^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/coverage/control.py", line 838, in get_data
INTERNALERROR>     self._post_save_work()
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/coverage/control.py", line 869, in _post_save_work
INTERNALERROR>     self._data.touch_files(paths, plugin_name)
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/coverage/sqldata.py", line 615, in touch_files
INTERNALERROR>     self._file_id(filename, add=True)
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/coverage/sqldata.py", line 417, in _file_id
INTERNALERROR>     self._file_map[filename] = con.execute_for_rowid(
INTERNALERROR>                                ^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/coverage/sqldata.py", line 1219, in execute_for_rowid
INTERNALERROR>     with self.execute(sql, parameters) as cur:
INTERNALERROR>   File "/opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/contextlib.py", line 137, in __enter__
INTERNALERROR>     return next(self.gen)
INTERNALERROR>            ^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/coverage/sqldata.py", line 1207, in execute
INTERNALERROR>     cur = self._execute(sql, parameters)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/jon/Library/Caches/pypoetry/virtualenvs/numbers-parser-xoMXdnO9-py3.11/lib/python3.11/site-packages/coverage/sqldata.py", line 1195, in _execute
INTERNALERROR>     raise DataError(f"Couldn't use data file {self.filename!r}: {msg}") from exc
INTERNALERROR> coverage.exceptions.DataError: Couldn't use data file '/Users/jon/Downloads/numbers-parser/.coverage.Jons-MacBook-Air.93985.543340': no such table: file

My config includes some adopts for pytest but commenting those out and using poetry run pytest -n logical --cov=src/numbers_parser tests/test_version.py alone is sufficient to crash. Playing around with arguments I can see ordering doesn't help but the number of threads does. At least on my test machine:

poetry run pytest --cov=src/numbers_parser -n 1 tests/test_version.py succeeds
poetry run pytest --cov=src/numbers_parser -n 2 tests/test_version.py succeeds
poetry run pytest --cov=src/numbers_parser -n 3 tests/test_version.py fails

Versions

% poetry run pip freeze | grep pytest
pytest==7.4.0
pytest-check==1.3.0
pytest-console-scripts==1.4.1
pytest-cov==4.1.0
pytest-profiling==1.7.0
pytest-xdist==3.3.1
% poetry run python --version
Python 3.11.4

Answer 1 · 2024-09-18T00:44:40.000Z

I have managed to reproduce the problem but I don't understand what's the point of using dynamic_context=test_function when pytest-cov's --cov-context option already handles that for you.

The problem is caused by should_start_context_test_function triggering a switch in the middle of some xdist internals (xdist.scheduler.load.tests_finished).

Please justify your usecase.

I think the only reasonable thing to do here is make pytest-cov raise an error when it detects that you are using dynamic_context=test_function and xdist, and a warning when you are using just dynamic_context=test_function.

Answer 2 · 2024-09-18T01:55:26.000Z

To give some context, this is the implementation of should_start_context_test_function (the context switcher set if you have dynamic_context=test_function):

def should_start_context_test_function(frame: FrameType) -> str | None:
    """Is this frame calling a test_* function?"""
    co_name = frame.f_code.co_name
    if co_name.startswith("test") or co_name == "runTest":
        return qualname_from_frame(frame)
    return None

You will notice that xdist.scheduler.load.tests_finished will unfortunately match, and trigger a switch at a very inopportune time:

79527.ff6f: Erasing data file '/tmp/pytest-of-ionel/pytest-75/test_dynamic_context0/.coverage.dev.79527.XYVLYICx.master1'
79527.ff6f:   File "<frozen runpy>", line 198, in _run_module_as_main
79527.ff6f:   File "<frozen runpy>", line 88, in _run_code
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib/python3.12/site-packages/pytest/__main__.py", line 7, in <module>
79527.ff6f:     raise SystemExit(pytest.console_main())
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib/python3.12/site-packages/_pytest/config/__init__.py", line 197, in console_main
79527.ff6f:     code = main()
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib/python3.12/site-packages/_pytest/config/__init__.py", line 174, in main
79527.ff6f:     ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
79527.ff6f:     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
79527.ff6f:     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
79527.ff6f:     res = hook_impl.function(*args)
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib/python3.12/site-packages/_pytest/main.py", line 332, in pytest_cmdline_main
79527.ff6f:     return wrap_session(config, _main)
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib/python3.12/site-packages/_pytest/main.py", line 285, in wrap_session
79527.ff6f:     session.exitstatus = doit(config, session) or 0
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib/python3.12/site-packages/_pytest/main.py", line 339, in _main
79527.ff6f:     config.hook.pytest_runtestloop(session=session)
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
79527.ff6f:     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
79527.ff6f:     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
79527.ff6f:     res = hook_impl.function(*args)
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib/python3.12/site-packages/xdist/dsession.py", line 123, in pytest_runtestloop
79527.ff6f:     self.loop_once()
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib/python3.12/site-packages/xdist/dsession.py", line 149, in loop_once
79527.ff6f:     if self.sched.tests_finished:
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib/python3.12/site-packages/xdist/scheduler/load.py", line 84, in tests_finished
79527.ff6f:     @property
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib64/python3.12/site-packages/coverage/collector.py", line 404, in switch_context
79527.ff6f:     self.flush_data()
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib64/python3.12/site-packages/coverage/collector.py", line 486, in flush_data
79527.ff6f:     self.covdata.add_arcs(self.mapped_file_dict(arc_data))
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib64/python3.12/site-packages/coverage/sqldata.py", line 123, in _wrapped
79527.ff6f:     return method(self, *args, **kwargs)
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib64/python3.12/site-packages/coverage/sqldata.py", line 531, in add_arcs
79527.ff6f:     self._start_using()
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib64/python3.12/site-packages/coverage/sqldata.py", line 860, in _start_using
79527.ff6f:     self.erase()
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib64/python3.12/site-packages/coverage/sqldata.py", line 831, in erase
79527.ff6f:     self._debug.write(f"Erasing data file {self._filename!r}")
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib64/python3.12/site-packages/coverage/debug.py", line 107, in write
79527.ff6f:     dump_stack_frames(out=self.output, skip=1)
79527.ff6f:   File "/home/ionel/open-source/pytest-cov/.tox/py312-pytest81-xdist350-coverage76/lib64/python3.12/site-packages/coverage/debug.py", line 268, in dump_stack_frames
79527.ff6f:     ''.join(traceback.format_stack()),