Is there a way to mark `xfail` tests, also with matching output?
stdedos opened this issue · 0 comments
stdedos commented
I would like to be able:
- Test is marked as
xfail
if-f it does not matchout: |
, but matchesxfail: |
(imaginary attribute), but continues - Test is marked as
xpass
if-f it matchesout: |
(preferably strict, since it will lead to actually noticing in CI/CD - Otherwise it is marked as
fail
This is clearly wrong:
https://github.com/stdedos/matplotlib-stubs-hoel/actions/runs/6221692521/job/16884150094
And I'd like to:
- Track its progress - if it somehow "changes". I cannot do that right now, since
expected_fail
means "negative test" (instead of pytest'sxfail
) - Revert "the useless todo". If it happens to match "exactly", then this will be triggered. But otherwise, the slightest deviation will continue masking this
This is (hopefully) illustrated by this example:
$ cat test_test.py
import pytest
EXPECTED = 2
XFAIL = 3
@pytest.mark.parametrize(
"result",
[
1,
pytest.param(EXPECTED, marks=pytest.mark.xfail(strict=True)),
pytest.param(XFAIL, marks=pytest.mark.xfail(strict=True)),
],
)
def test_increment(result):
assert result == EXPECTED
$ pytest test_test.py
========================== test session starts ==========================
platform linux -- Python 3.8.10, pytest-7.4.1, pluggy-1.3.0
configfile: pyproject.toml
plugins: mypy-plugins-3.0.0, dash-2.13.0, pudb-0.7.0, mypy-testing-0.1.1
collected 3 items
test_test.py FFx [100%]
=============================== FAILURES ================================
___________________________ test_increment[1] ___________________________
result = 1
@pytest.mark.parametrize(
"result",
[
1,
pytest.param(EXPECTED, marks=pytest.mark.xfail(strict=True)),
pytest.param(XFAIL, marks=pytest.mark.xfail(strict=True)),
],
)
def test_increment(result):
> assert result == EXPECTED
E assert 1 == 2
test_test.py:16: AssertionError
___________________________ test_increment[2] ___________________________
[XPASS(strict)]
======================== short test summary info ========================
FAILED test_test.py::test_increment[1] - assert 1 == 2
FAILED test_test.py::test_increment[2]
===================== 2 failed, 1 xfailed in 0.10s ======================