paratestphp/paratest

unexpected test execution with overlapping test suites

yurii-stickee-2023 opened this issue · 1 comments

yurii-stickee-2023 commented
Q A
ParaTest version v7.4.3
PHPUnit version 10.5.13
PHP version 8.3.11

Summary

In short, if we have 2 test suites that have same tests inside, PHPUnit will execute test suites sequentially, thus calling the same test case multiple times.

    <testsuites>
        <testsuite name="default">
            <directory>tests</directory>
        </testsuite>
        <testsuite name="duplicate">   <-------- THIS FORCES PHPUNIT TO RUN SAME TESTS AGAIN
            <directory>tests</directory>
        </testsuite>
    </testsuites>

Current behavior

./vendor/bin/phpunit

.............                                                   202 / 202 (100%)

Time: 00:00.122, Memory: 8.00 MB

./vendor/bin/paratest

...............................................................  63 / 202 ( 31%)
......................................
Time: 00:00.295, Memory: 10.00 MB

As you can see, it shows 63 tests completed of 202 expected.
It is even more crazy than expected of unexpected, because it should show at least 101?

How to reproduce: command, code and error stack trace

You can checkout repository with the sample https://github.com/yurii-stickee-2023/yurii-paratest-bug

Expected behavior

It should execute 202 tests? Maybe PHPUnit has option how to exclude duplicates, but probably not, as it executes code as test suites.

Maybe throw an exception that it is not supported?

regards

p.s.
You can change test from my repository to see that it actually runs 101 tests while showing 63, something like

    /**
     * @dataProvider providesData
     */
    public function testDummy1($a)
    {
        echo $a.'.';
        $this->assertTrue(true);
    }
Slamdunk commented

Your sample gives the same output between ParaTest and PHPUnit, if upgraded to PHPUnit 11:

$ vendor/bin/phpunit
PHPUnit 11.3.5 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.3.11
Configuration: /repos/yurii-paratest-bug/phpunit.xml

...............................................................  63 / 101 ( 62%)
......................................                          101 / 101 (100%)

Time: 00:00.012, Memory: 8.00 MB

There was 1 PHPUnit test runner warning:

1) Cannot add file /repos/yurii-paratest-bug/tests/DummyTest.php to test suite "duplicate" as it was already added to test suite "default"

WARNINGS!
Tests: 101, Assertions: 101, Warnings: 1, PHPUnit Deprecations: 2.



$ vendor/bin/paratest
ParaTest v7.5.4 upon PHPUnit 11.3.5 by Sebastian Bergmann and contributors.

Processes:     12
Runtime:       PHP 8.3.11
Configuration: /repos/yurii-paratest-bug/phpunit.xml

...............................................................  63 / 101 ( 62%)
......................................                          101 / 101 (100%)

Time: 00:00.167, Memory: 8.00 MB

There was 1 PHPUnit test runner warning:

1) Cannot add file /repos/yurii-paratest-bug/tests/DummyTest.php to test suite "duplicate" as it was already added to test suite "default"

WARNINGS!
Tests: 101, Assertions: 101, Warnings: 1, PHPUnit Deprecations: 2.

Please upgrade your dependencies as bugs are fixed only in the latest version of ParaTest.

  • 👍1