Tests set as incomplete in an @after method are marked as errors

[timvandijck]

Q	A
PHPUnit version	10.1.1
PHP version	8.1
Installation Method	Composer

Summary

Using the markTestIncomplete() in a method annotated by @after results in the tests being marked as errors in stead of incomplete.

Current behavior

In PHPUnit 9(.6.7) my example code produces following output (only pasting the last 2 lines to keep it relevant):

OK, but incomplete, skipped, or risky tests!
Tests: 2, Assertions: 2, Incomplete: 1.

In PHPUnit 10 the same code produces following output:

ERRORS!
Tests: 2, Assertions: 2, Errors: 1.

When using markTestIncomplete() inside a test instead of an after method the test is correctly marked as incomplete.

How to reproduce

<?php

namespace Tests\Unit;

use PHPUnit\Framework\TestCase;

class ExampleTest extends TestCase
{
    public function test_that_true_is_true(): void
    {
        $this->assertTrue(true);
    }

    /** @after */
    public function after(): void
    {
        $this->markTestIncomplete('test is incomplete');
    }
}

Expected behavior

When tests set as incomplete in an after method they should show as incomplete in the output and not as an error.

[sebastianbergmann]

I can reproduce this.

Test.php

<?php declare(strict_types=1);
use PHPUnit\Framework\Attributes\After;
use PHPUnit\Framework\TestCase;

final class Test extends TestCase
{
    public function testOne(): void
    {
        $this->assertTrue(true);
    }

    #[After]
    public function after(): void
    {
        $this->markTestIncomplete('test is incomplete');
    }
}

phpunit --no-output --log-events-text php://stdout Test.php
PHPUnit Started (PHPUnit 10.1.1-1-g59a73550c0 using PHP 8.2.5 (cli) on Linux)
Test Runner Configured
Test Suite Loaded (1 test)
Event Facade Sealed
Test Runner Started
Test Suite Sorted
Test Runner Execution Started (1 test)
Test Suite Started (Test, 1 test)
Test Preparation Started (Test::testOne)
Test Prepared (Test::testOne)
Assertion Succeeded (Constraint: is true, Value: true)
Test Passed (Test::testOne)
Test Errored (Test::testOne)
test is incomplete
Test Finished (Test::testOne)
Test Suite Finished (Test, 1 test)
Test Runner Execution Finished
Test Runner Finished
PHPUnit Finished (Shell Exit Code: 2)

Also note that the events for Test::after are missing from the event log:

After Test Method Called (Test::after)
After Test Method Finished:
- Test::after

[sebastianbergmann]

@timvandijck May I ask what your use case is for marking a test case as incomplete in a hook method that is called after the test method itself has finished? This does not make sense to me, to be honest.

[timvandijck]

@sebastianbergmann we have a package to use snapshot testing in PHPUnit.
The package exposes a trait you can use in you testclass. One of the methods inside the trait is one that uses the @after annotation to look if any snapshots had mismatched and if needed mark the test as incomplete with a formatted message.

Here's a reference: https://github.com/spatie/phpunit-snapshot-assertions/blob/main/src/MatchesSnapshots.php#L31

[sebastianbergmann]

The "after test method" is not, and was never, intended for a use case like that. By the time such methods are called, no decision on the outcome of a test can be made anymore. That this worked for you in the past, with PHPUnit 9, for instance, was accidental.

The markTestIncompleteIfSnapshotsHaveChanged() method in your trait should use the #[PostCondition] attribute instead of #[After] (or the @postCondition annotation instead of the @after annotation if you want/have to use annotations).

[timvandijck]

@sebastianbergmann thank you for the clarification and quick response. We'll try and do it this way.

[sebastianbergmann]

@timvandijck Can this be closed? Thanks!

[sebastianbergmann]

No feedback, closing.