fail-fast "PASSES M out of N tests" output is insufficient
Closed this issue · 1 comments
apjanke commented
When you run plain test
or test2
, fail-fast behavior is the default.
But the non-verbose output for test2
in this case isn't sufficient; it doesn't make it clear that the test run was aborted.
>> test2 isosurface
PASSES 20 out of 21 tests
>>
That sounds like there were 21 tests total in the file, and only one of them failed. But there are actually at least 30 tests.
Add some additional output there: a line saying "test run aborted", and probably the detailed output for the failed test block.
apjanke commented
Better:
>> test2 isosurface
----- Failure details: block 25 -----
***** testify.internal.BistBlock: error (index=25)
is_test=1 is_xtest=0
error_id=''
pattern=/Y must match the size of V/
Code:
y = -14:6:11;
fvc = isosurface (x, y, z, val, iso);
-> success=0, msg=!!!!! Incorrect error raised: expected error message matching /Y must match the size of V/, but got 'isosurface: X must match the size of V'
Aborted test run due to test failure.
PASSES 20 out of 21 tests
>>
>> test2 isosurface -no-fail-fast
----- Failure details: block 25 -----
***** testify.internal.BistBlock: error (index=25)
is_test=1 is_xtest=0
error_id=''
pattern=/Y must match the size of V/
Code:
y = -14:6:11;
fvc = isosurface (x, y, z, val, iso);
-> success=0, msg=!!!!! Incorrect error raised: expected error message matching /Y must match the size of V/, but got 'isosurface: X must match the size of V'
----- Failure details: block 26 -----
***** testify.internal.BistBlock: error (index=26)
is_test=1 is_xtest=0
error_id=''
pattern=/Z must match the size of V/
Code:
z = linspace (16, 18, 5);
fvc = isosurface (x, y, z, val, iso);
-> success=0, msg=!!!!! Incorrect error raised: expected error message matching /Z must match the size of V/, but got 'isosurface: X must match the size of V'
----- Failure details: block 33 -----
***** testify.internal.BistBlock: warning (index=33)
is_test=1 is_xtest=0
error_id=''
pattern=/colors will be calculated, but no output argument to receive it./
Code:
[f, v] = isosurface (val, iso, yy);
-> success=0, msg=!!!!! error raised: isosurface: COL must match the size of V
PASSES 26 out of 30 tests (1 known bug)
>>