tpvasconcelos/ridgeplot

Fix `test_regressions` tests failing in GitHub Actions

Opened this issue · 1 comments

I had to mark the test_regressions tests with XFAIL when running the checks in CI because they are currently failing when being run in GitHub Actions. See here and here.

This is probably due to small differences in output between environments (local vs. remote macOS, Windows, and Linux).

The test, for reference:

@pytest.mark.xfail(
# TODO: Fix this (i.e. re-enable these tests)!
reason=(
"Currently breaking in CI, probably due to small "
"differences in output between environments."
),
condition=os.getenv("CI") is not None,
)

Regardless of the solution, one nice feature to implement here is the ability to upload the plot artefacts that resulted in the failed tests and expose these artefacts to the relevant PR. This discussion might be useful for this:

For some reason, the tests/e2e/test_examples.py::test_examples_width_height_set[lincoln_weather-load_lincoln_weather] hangs forever and never finishes.

Some examples here: https://github.com/tpvasconcelos/ridgeplot/actions/runs/7481625128/job/20363565240

This ran succeeded but it took over 14minutes to run the e2e test suite: https://github.com/tpvasconcelos/ridgeplot/actions/runs/7481625128/job/20363565240

For this reason, I have also changed the xfail mark to a skipif (8faa9df)