muter-mutation-testing/muter

Muter incorrectly fails when successful test contains "fatal error" text

ZevEisenberg opened this issue · 0 comments

TestSuiteOutcome.swift:57 checks to see if the entire test log contains the text "fatal error". But sometimes, this is valid in a test log. Instead, we should look at the process.terminationReason at MutationTestingIODelegate.swift:28. Related to #185.

One way to reproduce this is to have a test suite that uses assertionFailure("foo") in a test. The failure may not halt the process, but will still print a message.

We're still looking into why an assertionFailure would put Fatal Error in the log but not halt the tests; will post here if we find out.

A nice way to test this could be to mock Process so we can return arbitrary error codes without having to build a whole sample project to reproduce this issue.