CI invocations for non-captcha tests?
zstumgoren opened this issue · 2 comments
Related to #149
@palewire raised the below questions about test invocations in a CI and non-CI context in #147
How would we handle the command-line options to create cases where:
A) Only CAPTCHA tests should be run
B) Only login-only based "slow" webtests should be run
C) All webtests, regardless of login method, should be run
For A, pyest --runslow -m captcha
should work. This would not be run on CI, but is intended for manual runs by core devs and contributors (ideally prior to merging to master
).
For B and C, one option would be to add yet another pyest marker such as noncaptcha
to all webtests without a captcha.
We could then selectively target tests as below:
# Run all slow tests except for CAPTCHA-based
# For use on CI
pytest --runslow -m noncaptcha
# For local dev
# Run all tests
pytest --runslow
# Run only CAPTCHA
pytest --runslow -m captcha
The above assumes that we devise a solution for passing in login credentials via environment variables, per #149.
This sounds like a reasonable set of next steps to me. I'll mark the no-captcha tests and reconcile the test commands in the Makefile with what you've outlined above. We can deal with the credentials stuff for CI after that.
This is done and can be closed.