Isolating executions of multiple tests across same model when using TestSuite.judge()
appukuttan-shailesh opened this issue · 1 comments
I was taking a look at TestSuite, whereby users are able to specify a number of models and tests to be evaluated.
I have found on some occasions that the order in which these tests were executed affected the outcome. On closer inspection, this discrepancy was arising out of changes that a particular test was making on the model, which affected the "initial state" of the model for the other tests.
When running tests individually, using Test.judge(), I generally prefer to run each as a separate subprocess/fork. Do you think a similar (optional) feature would be useful for TestSuite where each judge() would be isolated in its own subproces, thereby not affecting the execution of other model-test pairs. Not sure if any pitfalls exist in doing so, or if this is already available.
That's a great idea. I will look into it.
TestSuite currently has hooks that can be executed after the completion of each test, which you could use to for example reset the value of some model attribute after a test.