Performance regression in 0.18.1 – 10x slowdown in pytest collection
Opened this issue · 7 comments
We're using pytest-asyncio in a repo with 20000+ tests. With 0.18.0 I get a pytest collection time of about 25 seconds, running pytest . --collect-only. Updating to 0.18.1 increases the collection time to almost 300 seconds.
0.18.1 only contains one logical change, so that's the cause, but I don't (yet) have a clue about a resolution.
We are by the way using the strict asyncio mode.
Could it be an option to not allow async fixtures to be used in sync tests for strict mode?
Thanks for pointing out this issue. I don't think pytest-asyncio has enough awareness of its performance, especially in large code bases.
This could be a duplicate of #720.
There has been recent work in pytest to raise a warning if a sync test requests an async fixture (see pytest-dev/pytest#10839). Specifically, pytest-dev/pytest#10839 (comment) points out that pytest-asyncio doesn't do anything if a sync test requests an async fixture. A more specific warning was added to pytest-asyncio in #979 (unreleased at the time of writing). I don't think pytest-asyncio should immediately raise an error just yet. We should at least release the update that emits the warning and see what issues are reported by the users. Then we can completely deprecate it and raise an error.
That said, I think the first step would be to add some kind of performance test to the pytest-asyncio tests. We could use this as a baseline for optimizing the collection time.
The pre-release pytest-asyncio==1.0.0a1 creates significantly fewer fixtures, which should improve the collection performance. @davidparsson Please let us know whether this improves or solves the situation.
@davidparsson pytest-asyncio v1.0.0 is out and should address this issue.
I'm closing this due to a lack of feedback.
I'm happy to reopen the issue when the performance improvements in v1.0 didn't address the problem sufficiently.
@seifertm, thanks for the attempt. However, the collection performance is still greatly degraded since 0.18.0, from ~32 s to almost 4 minutes. I suggest reopening this.
0.18.0:
$ pytest directory/ --collect-only
Test session starts (platform: darwin, Python 3.12.10, pytest 8.4.1, pytest-sugar 1.0.0)
PyQt6 6.8.1 -- Qt runtime 6.8.2 -- Qt compiled 6.8.2
rootdir: <redacted>
configfile: pyproject.toml
plugins: xprocess-0.23.0, cov-6.0.0, sugar-1.0.0, timeout-2.1.0, anyio-4.8.0, qt-4.3.1, typeguard-2.13.3, mock-3.3.1, xdist-3.3.1, base-url-2.1.0, memray-1.3.0, split-0.10.0, alembic-verify-0.1.4, asyncio-1.0.0, playwright-0.5.2
asyncio: mode=Mode.STRICT, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 21862 items / 302 deselected / 21560 selected .
.
.
pytest directory/ --collect-only 27.21s user 4.31s system 98% cpu 31.940 total
1.0.0:
$ pytest directory/ --collect-only
Test session starts (platform: darwin, Python 3.12.10, pytest 8.4.1, pytest-sugar 1.0.0)
PyQt6 6.8.1 -- Qt runtime 6.8.2 -- Qt compiled 6.8.2
rootdir: <redacted>
configfile: pyproject.toml
plugins: xprocess-0.23.0, cov-6.0.0, sugar-1.0.0, timeout-2.1.0, anyio-4.8.0, qt-4.3.1, typeguard-2.13.3, mock-3.3.1, xdist-3.3.1, base-url-2.1.0, memray-1.3.0, split-0.10.0, alembic-verify-0.1.4, asyncio-1.0.0, playwright-0.5.2
asyncio: mode=Mode.STRICT, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 21862 items / 302 deselected / 21560 selected
.
.
.
pytest directory/ --collect-only 224.86s user 5.04s system 99% cpu 3:50.85 total
Sorry for my late reply here. It seems GitHub has messed up my notifications.
Thanks for the feedback on this! Reopening…
Pytest-asyncio still doesn't have any performance benchmark to track this. For the lack of a large codebase, I suggest to create a micro benchmark for collection performance and see if the issue is visible there. This will give us a baseline for future performance improvements and allow us to detect performance regressions before they get released.