failOnWarning="true" but tests with warnings marked as passed in PHPStorm

[doctorby]

Q	A
PHPUnit version	8.5.16
PHP version	7.4.20
Installation Method	Composer

Summary

Was introduced here #4623

Running tests from PHPSTORM, before PHPUnit 8.5.15, having failOnWarning="true", test with warnings was marked as failed, now it's marked as passed.

Current behavior

Tests with warnings marked as passed in PHPSTORM

How to reproduce

Create test.
Make mistake in dataprovider method or in setUp e.g. try to configure a method that does not exist in mocked class
Run test
Test marked as passed

Expected behavior

Having failOnWarning="true" -> tests with a warning should be marked as failed

I understand there were some reasons for #4623

But there must be some other solution.
Why is it important? With just one mistake in the seUp() method (e.g. ->method('not_existed_method') ), all tests inside the class will not be executed, but will be marked as "passed" - obviously this is not what is expected.

If I remove method from class, I expect that tests, where such method was configured, will fail

[pawel-slowik]

Hi, I also have run into this issue.

I have tried to create a workaround for it by implementing a PHPUnit extension, but this turned out to be unfeasible because the TeamCity logger is not really designed to be reused in an extension. Also the class is marked @internal so it wouldn't be a good idea anyway.

The issue is somewhat mitigated by the fact that often PHPStorm will highlight / hint at the mistake that is the actual cause for the warnings. But that's not always the case, and also while merging you may not even have the relevant file open in the IDE, so you are likely to miss the hint.

[pawel-slowik]

We can't really have warnings in TeamCity since individual tests can only be treated as failed or passed:
https://youtrack.jetbrains.com/issue/WI-57958/PHPStorm-reports-phpunit-warnings-as-errors-which-clutters-the-output#focus=Comments-27-4765518.0-0
Maxim Kolmakov Commented 16 Mar 2021 10:06

There is no specific TeamCity message that will indicate that there is a warning inside a test. The only messages are testFailed, testIgnored, and no message (meaning, test passed).

The testsuite as a whole cannot be marked as failed when all its tests are marked passed because there's no property on the testSuiteFinished message that could indicate this: https://www.jetbrains.com/help/teamcity/service-messages.html#Reporting+Tests

Therefore the only way to make PHPStorm fail a test that produces a warning is to add a testFailed message for it. This change should be pretty straightforward, I could work on a PR for it. But as we can see in #4591, some users don't want that.

So how do we resolve this? Here are some ideas:

1. Output a testFailed message when failOnWarning is set to true, do not output it otherwise. This seems to be the simplest way to go, however the failOnWarning parameter is clearly intended to only Signal failure using shell exit code. I have doubts if it's really good idea to just repurpose it to mean something else for the TeamCity logger.

2. Add a new configuration parameter for the TeamCity logger?

3. Try to establish some reasonable level of a test issue that should emit a testFailed message. We currently have a rather detailed system of events, so we can tell apart a Warning from a PhpunitWarning / PhpDeprecation /
   PhpunitDeprecation etc. https://github.com/sebastianbergmann/phpunit/tree/11.3/src/Event/Events/Test/Issue
   Since the #4591 issue is mainly about PHPUnit deprecations: Run a test with at() matcher or assertRegExp() or other soft-deprecated features, maybe we could just skip those and emit testFailed for everything else?

What do you think?

[sebastianbergmann]

The only thing controlled by failOnWarning="true" is whether a warning causes the test runner to exit with a shell exit code signaling an error or not. It is outside the control of this project whether PhpStorm acts on PHPUnit's shell exit code or not.

Please talk to Jetbrains about PHPUnit's TeamCity logger and/or PhpStorm's PHPUnit support should this still be an issue.

[uuf6429]

To everyone one-upping this issue, here's the corresponding JetBrains ticket: https://youtrack.jetbrains.com/issue/WI-78513/PHPUnit-wrong-exit-code-handling - if you want to make a difference, voice your concern there.

[pawel-slowik]

Hi, I usually try to avoid commenting on closed issues, but in this case there still seems to be some interest in another solution. Because of my earlier comments I've been getting some private messages / emails asking for advice on this, so I feel it would be a good idea to share some further insights.

First, the JetBrains folks won't be able to tackle this by themselves. PhpStorm relies on TeamCity protocol messages instead of PHPUnit's exit status to display test failures. This is not a bug, but a feature that allows it to show the locations of issues in the test sources. For this to work with regard to warnings, the IDE must receive the messages that were removed in #4623. There's no way around it, except for completely reengineering testing support in PhpStorm, which is unlikely to happen since they are using the same protocol for multiple IDEs and frameworks.

On the plus side, in recent PHPUnit versions mocking and stubbing failures are handled as errors, not as warnings. A testFailed message is emitted and the failure is visible as an error in PhpStorm's UI. I have verified this for PHPUnit 10.x / 11.x / 12.x and PhpStorm 2025.1.3. So at least this case can be considered solved.

Finally, if you're still interested in getting the warnings to show in PhpStorm, I have created a PHPUnit extension for this purpose: pawel-slowik/phpunit-phpstorm-issue-logging. It works for my projects, maybe it'll work for yours too 😄