Warning for variables missing in regression test
Opened this issue · 0 comments
timothy-nunn commented
At present, the regression tests make the following assertions:
- Assert that an MFile exists
- Assert that the MFile contains some text
- Assert that the
ifail
value indicates success - Assert there are no differences
- Assert that the new MFile contains all of the variables present in the reference MFile
- Assert that the reference MFile contains all of the variables present in the new MFile
Via these 6 assertions, a test failure indicates if an MFile has changed due to the changes on a branch. However, the final 3 assertions contain information that could be useful for a converged MFile. However, for example, if assertion 4 failed, the reviewer has no way of knowing if conditions 5 or 6 were met.
I propose that we warn before assertion 4 about assertions 5 and 6 (ie warn the reviewer if the MFiles contain different variables). That way, the reviewer gets the information from assertions 5 and 6 even if the test fails before there.
Note #3227 will stop these warnings being swamped by debug logs.