Reduce test_on_push workflow run-time
Closed this issue · 2 comments
Currently we are at ~ 9 minutes to run the test_on_push workflow. I think we should keep this around 5-10 minutes, so this should probably be reduced unless others have different thoughts @martinjrobins @NicolaCourtier?
A few thoughts on this:
- Reduce the number of optimisers that are performance tested (i.e. asserted on the identified parameter)
- Reduce the maximum number of iterations for each optimiser during performance testing (this may result in more failures)
- Split the test_on_push workflow so that performance tests are only run for a PR.
I agree that we should reduce the time! On my laptop, reducing the max_iterations from 250 to 100 takes the run time for test_parameterisations.py from 14 minutes down to 6 minutes, and the tests still pass. I think, in future, we could also test the optimisers in isolation from the battery models in order to speed things up further. I would prefer this to reducing the number of optimisers that are tested.
I've created a linked branch with three updates:
- moved the setting of the default model, parameters and spm_cost to fixtures in test_parameterisations
- reduced the max_iterations to 100 in test_parameterisations : test_spm_optimisers
- parameterised test_examples based on the file path so that progress through each example can be seen
I'll leave PR open for further input.