Problems not identified in string-comparison errors on PHPUnit 5.7

[sminnee]

For my sins, I'm using an older version of PHPUnit. I've noticed that problems aren't always picked up.

Repo - working case

This output marks a problem correctly:

PHPUnit 5.7.27 by Sebastian Bergmann and contributors.

...F..........                                                    14 / 14 (100%)

Time: 652 ms, Memory: 8.00MB

There was 1 failure:

1) SilverStripe\Config\Tests\Transformer\YamlTransformerTest::testNoNameStatement
Failed asserting that false is true.

/Users/samminnee/Projects/silverstripe-config/tests/Transformer/YamlTransformerTest.php:183

FAILURES!
Tests: 14, Assertions: 23, Failures: 1.

Generating code coverage report in Clover XML format ... done
The terminal process "/bin/bash '-c', ''/Users/samminnee/Projects/silverstripe-config/vendor/bin/phpunit' --colors=always '/Users/samminnee/Projects/silverstripe-config/tests/Transformer/YamlTransformerTest.php''" terminated with exit code: 1.

Terminal will be reused by tasks, press any key to close it.

Outcome

The failure is linked as a problem

Repo - broken case

But this output does not:

> Executing task: '/Users/samminnee/Projects/silverstripe-config/vendor/bin/phpunit' --colors=always '/Users/samminnee/Projects/silverstripe-config/tests/Transformer/YamlTransformerTest.php' <

PHPUnit 5.7.27 by Sebastian Bergmann and contributors.

...F..........                                                    14 / 14 (100%)

Time: 734 ms, Memory: 8.00MB

There was 1 failure:

1) SilverStripe\Config\Tests\Transformer\YamlTransformerTest::testNoNameStatement
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'blahy'
+'blah'

/Users/samminnee/Projects/silverstripe-config/tests/Transformer/YamlTransformerTest.php:184

FAILURES!
Tests: 14, Assertions: 23, Failures: 1.

Generating code coverage report in Clover XML format ... done
The terminal process "/bin/bash '-c', ''/Users/samminnee/Projects/silverstripe-config/vendor/bin/phpunit' --colors=always '/Users/samminnee/Projects/silverstripe-config/tests/Transformer/YamlTransformerTest.php''" terminated with exit code: 1.

Terminal will be reused by tasks, press any key to close it.

Outcome

No problems are linked

Expected outcome

The failure is linked as a problem

[elonmallin]

Thanks for the bug report, unfortunately I'm not super actively maintaining this so I can't support older versions. At least if it requires different problem matchers for different versions it sounds quite tricky to fix.

Perhaps the current problem matcher could be patched though. I'd happily accept a PR that works for both cases :)

[elonmallin]

I'll close this as I'll aim to only support later versions of phpunit and v5 is end of life https://phpunit.de/supported-versions.html