API Question: Configure pytest without running it
Opened this issue · 4 comments
Hi pytest team,
I am developing the ipytest package, which allows you to run pytest inside the notebook. Recently a question regarding the pytest API came up: is there some API to configure pytest without running it? I browsed the API docs but couldn't find an easy way and just wanted to make sure I did not overlook something. This issue is not breaking and hence resolving it is more a nice to have.
The motivation: a user reported the issue, that pytest.mark.parametrize gives misleading error messages. The root cause for this behavior (see detailed description here) seems to be that the mark generator behaves differently when pytest is configured. For ipytest, the tests are all defined before pytest is run (and configured). Therefore the error message of the mark generator is a little bit tricky to parse.
there currently is no external api to configure pytest (and tbh, with how the pytest startup is currently lined up its very error prone to orchestrate
i would like to change that, but its not as high as other details on my priority list since its going to be tricky and take a while
if you would like to take a stab at making it more externally usable im happy to help/mentor however
In principle, yes I would be willing to work on this functionality. From my side there are two comments:
- This contribution would be a side-project for me, so there is a limit to the amount of time I can invest. Just bringing it up, since you hinted at complexity of the issue. If it's unrealistic to expect substantial progress within a couple of weekends, I'm not sure it make sense. But that's probably easiest for you to judge.
- I would like to make sure, the end result would solve my original problem, so we should probably align a little bit on the details of the changes you had mind. From my side, I would like some API to run the different lifecycle steps independently, something along the the lines of
pytest.init(); pytest.run(); pytest.terminate(), names need more work ;).
The Pytester https://docs.pytest.org/en/latest/reference/reference.html#pytest.Pytester class already implements some APIs to create pytest sessions. Maybe some of the logic which is not specific to unit testing could be moved to the normal Session class (or a new subclass) to provide a programmatic API? (And the same logic can be used from the hooks.)
Another option might be to make the hooks the official API, and rework them to make it easier to use.
E.g. some hooks like pytest_cmdline_main are not easy to use programmatically because they have full control of the execution. Maybe the hooks could "yield" back control to the caller when they call another hook, instead of doing direct calls. Similarly to asynchronous code. Or the hooks can simply be split further.
hooks are already an official api
pytesters config object creation barely is good enough for testsuites, im hesitant to open it for more general consumption