tarpas/pytest-testmon

Testmon db still corrupted with --collect-only

adam-mrozik opened this issue · 5 comments

adam-mrozik commented

Hey,

(Had to edit the issue). Turns out my collect-only was actually corrupting the file. When I did run first time with --collect-only --testmon, 4 files where collected. Subsequent run without --collect-only collected 0 files.

emavgl commented

Thank you for reporting! I am experiencing similar issues, resulting in all the tests to be collected to run, I am now removing the --collect-only to check if that is the cause of that issue.

pokopt commented

Hi @adam-mrozik, I'm trying to reproduce your problem. I have three files with three test in each file. Running
pytest --collect--only --testmon
for the first time lists all tests (3 modules / 3 function tests each). Running
pytest --testmon
as next step will execute all 9 tests with no problem.
Does your workflow differ?
Thanks in advance for any updates.

adam-mrozik commented

Hmm, lately I have actually been noticing, that I have similar issue to @emavgl . I am still trying to debug if there isn't some internal workflow issue on my side.

My workflow currently consists of generating testmon file on CI after each run, and after merge replacing master testmon file.
This, however breaks in two cases:

  • When someone reverts/merges without CICD completing (then I am having an out-of-date testmon file)
  • When PR is merged if it is out of date with master.

I am looking if any of these do break the testmon behaviour. Right now though I am experiencing a lot of random things, some of them like @emavgl reported. Will let you now when I am able to reproduce, cause I am aware my report is semi-useful right now

root@c74a0f828c43:/app/accounts# pytest tests --testmon
=========================================================================================================== test session starts ===========================================================================================================
platform linux -- Python 3.8.16, pytest-7.2.0, pluggy-1.0.0
testmon: changed files: 0, skipping collection of 101 files, environment: default
rootdir: /app/accounts
plugins: forked-1.4.0, hypothesis-6.46.7, testmon-1.4.2, flaky-3.7.0, cov-3.0.0, timeout-2.1.0, case-1.5.3 xdist-2.5.0
collected 415 items / 193 deselected / 222 selected

One thing that is weird for me is testmon recognizing there are 0 changes, but still decides to run some tests. Do you know what might be the reason?

emavgl commented

Unfortunately I have also hard time to reproduce the issue, since I have experienced this on a big project. If can help I add some details:

  • have integrated testmon in Github Actions. I save the testmon database using Github action cache.

  • I run it in conjunction with xdist plugin with

 python -m pytest -n 4 tests/integration --dist loadfile --testmon
  • my pytests have fixtures and conftest files

I was running testmon with collect only with the purpose of listing the pytest tests to be executed

python -m pytest tests/integration --testmon-nocollect --collect-only -q

I am still not exactly sure that collect only is the issue. I have recently removed it and I will keep monitoring it in the next days. So far I did two runs with no problems.

tarpas commented

There has been quite a lot of changes recently in testmon and no news regarding this. Please reach out in case you have problems with testmon with a new ticket.