phate/jlm

CI does not report error when unit tests fail

Closed this issue · 1 comments

I just noticed it in PR #387 that the CI build job (which also runs the unit tests) does not fail even though a unit test fails. The only thing that failed was the valgrind-check job (that is how I noticed it).

The rule is as following:

$(RUNTESTS): run-% : %
	@echo -n "Running test $^ ... " ; \
	if $^ >$^.log 2>&1 ; then \
		echo "success" ; \
	else \
		echo "failed. See $^.log:" ; cat $^.log ; \
	fi

I have not really investigated, but I believe that the issue arises because the only thing that really fails is the test invocation, but not the actual make invocation that runs this test.

FYI: @caleridas @sjalander @haved

You're absolutely right, this is the second time I have pushed changes to GitHub only to discover that some tests were failing locally