no-data-collected under unix when using filterwarnings in pytest.ini

[mariuswallraff]

Summary

First of all sorry for this incomplete bug report (no reproducer). Feel free to close it, but I figured it might still help somebody.

The problem is that the coverage data collection fails if the filterwarnings option in pytest.ini is used to ignore a custom warning defined in our project, but only under Linux (Debian and Alpine) and Mac; it works fine under Windows with the nominally same conda environment and plugin versions, and it works fine as soon as the option is commented out.

I sadly cannot tell you if/which additional conditions play into this.

Expected vs actual result

expected:
TOTAL 17353 5185 70%

actual:
TOTAL 17353 17353 0%
with this showing at the end of running the tests (or once for every worker if using pytest-xdist):

/opt/conda/envs/analytics_complete/lib/python3.10/site-packages/coverage/control.py:883: CoverageWarning: No data was collected. (no-data-collected)
  self._warn("No data was collected.", slug="no-data-collected")

Reproducer

Our pytest call:

    pytest
      --cov=XXX
      --cov-config=.coveragerc
      --cov-report term-missing
      --junitxml=report.xml
      --color=no
      --code-highlight=no
      --numprocesses=4
      tests/XXX

where XXX is a subfolder of the working directory, and the top level of our package.

Our pytest.ini:

[pytest]
addopts = -v -s --durations=20 --durations-min=1.0 --numprocesses=auto --strict-markers --color=yes --code-highlight=yes

filterwarnings =
    ignore::XXX.PrototypeWarning
markers =
    slow: marks tests as slow
testpaths = tests
xfail_strict = true

(some arguments are doubled/overridden because one is running in GitLab's CI/CD pipeline, while the ini is also used for local execution, but I played around with those options and they are not at fault)

Our .coveragerc:

[run]
omit =
    XXX/pipelines/*

[report]
exclude_also =
    # don't complain about overload stubs (which are never executed)
    @overload

    # don't complain about assertions that are not generally worth writing tests for
    raise AssertionError
    raise NotImplementedError

Not a real reproducer, sorry, but I tried various different combinations of options (including disabling xdist), and the only thing that helped was commenting out filterwarnings.

Versions

============================= test session starts ==============================
platform linux -- Python 3.10.13, pytest-7.4.3, pluggy-1.3.0
rootdir: /builds/xxx/code/analytics
configfile: pytest.ini
plugins: nbval-0.10.0, xdist-3.5.0, cov-4.1.0, anyio-4.1.0
created: 2/2 workers

These versions are identical on the different systems (i.e., between the unix ones where this fails with filterwarnings active, and Windows, where it works).

The problem is very likely also true for versions that are about a year old, because that was when we first encountered the problem (but thought it was an incompatibility with xdist).

Config

tox.ini is not used, setup.cfg does not contain testing-related information, for the rest see above

Code

Sorry, can't share our code, and didn't create a minimal example.

[ionelmc]

So you're saying using filterwarnings makes your coverage drop to 0%? Can't really help without a reproducer.

[flying-sheep]

Here’s a reproducer: https://github.com/flying-sheep/pytest-cov-filterwarnings

As its readme says, the cause is that pytest imports warning classes referenced in filterwarnings before pytest-cov is initialized.

[ivirshup]

FWIW, I believe this is connected to #437

[flying-sheep]

Ah yeah, seems like this issue is a duplicate of #437 then.

As @ionelmc says in #635 (comment): This is a chicken-and-egg problem. Also pytest documents its plugin order: it loads builtin plugins first, e.g. the warnings plugin, which has a hook that runs immediately: pytest_load_initial_conftests.

So I guess the only way to prevent that is to pass -p no:warnings.

Now the question is, if we do that, can we re-enable it at runtime via pytest_plugins = ["warnings"]?