Tests are not correctly marked as incomplete when creating/updating snapshots
Closed this issue · 3 comments
As the title says, tests are not correctly marked as Incomplete
when creating/updating snapshots. Instead PHPUnit displays them as Errors
.
PHP 8.2.4
laravel/framework 10.7.1
phpunit/phpunit 10.1.0
spatie/phpunit-snapshot-assertions 5.0.1
Manually calling $this->markTestIncomplete('Test incomplete')
within a test works perfectly fine. But this packages calls this method from within the markTestIncompleteIfSnapshotsHaveChanged()
method which is using an @after
annotation. Exceptions thrown during this test lifecycle appear to be treated differently.
Testcases
<?php
use App\Models\User;
use Illuminate\Foundation\Testing\TestCase;
use Spatie\Snapshots\MatchesSnapshots;
class SnapshotTest extends TestCase
{
use MatchesSnapshots;
public function test_snapshot_matches()
{
$user = User::query()->findOrFail(1);
$this->assertMatchesJsonSnapshot($user->toJson());
}
public function test_is_incomplete()
{
$this->markTestIncomplete('This test is incomplete.');
}
}
Command
vendor/bin/phpunit -d --update-snapshots
Expected Result
PHPUnit 10.1.0 by Sebastian Bergmann and contributors.
II 2 / 2 (100%)
Time: 00:00.143, Memory: 34.00 MB
OK, but there are issues!
Tests: 2, Assertions: 3, Incomplete: 2.
Actual Result
PHPUnit 10.1.0 by Sebastian Bergmann and contributors.
EI 2 / 2 (100%)
Time: 00:00.154, Memory: 38.50 MB
ERRORS!
Tests: 2, Assertions: 3, Errors: 1, Incomplete: 1.
@tklie thank you for reporting this problem.
I believe this is an issue with PHPUnit as the same issue occurs when marking tests as incomplete in an after method even without our package.
I created an issue with them: sebastianbergmann/phpunit#5337
Their response was quick, see sebastianbergmann/phpunit#5337 (comment)
The "after test method" is not, and was never, intended for a use case like that. By the time such methods are called, no decision on the outcome of a test can be made anymore. That this worked for you in the past, with PHPUnit 9, for instance, was accidental.
The markTestIncompleteIfSnapshotsHaveChanged() method in your trait should use the #[PostCondition] attribute instead of #[After] (or the
@postCondition
annotation instead of the@after
annotation if you want/have to use annotations).
Since I was just working on a project using this package, I tried it and confirm changing @postCondition
worked.