[Test Impr.] VM Standardization [S]
ArneTR opened this issue · 4 comments
We want to minimize the cases where tests pass/fail on local the inverse in our VM pipeline.
One of the main culprits of this is a different set of metric providers that are used locally vs in the VM.
The simplest [S] solution do this would be that instead of using a test-config that is based on the user's local config file, we have a standardized test-config that is the same as the one used in the VM.
A smarter [M] solution would be to keep the config copying functionality, but add a mode to the tests that you can run with pytest XX --vm-environment which uses VM-standardized test-config file.
- After discussion, this is added complexity. Makes more sense to just have a standardized config so we know for sure what people ran with the tests
Bumping this task up as I ran into the issue again that the tests were behaving unexpectedly due to a local temporary change to my config file. I want the tests to run on a defined standard configuration.
If we spot multiple confusing configurations we just need more tests.
Again another case that just happened to me: Since I had a live endpoint URL in my config.yml the actual API was requested.
This copying is definitely not the way to go.
Please link the PR so that this issue closes automatically