Tests are failing on CI

Versions:

graphql-laravel Version: 8.x / master
Laravel Version: all
PHP Version: all supported one

Description:

Around 30% of the tests started to fail, seems like the SQL snapshots are either wrongly recorded or changes etc.

Steps To Reproduce:

Just run the tests (also locally)
See https://github.com/rebing/graphql-laravel/actions/workflows/tests.yml => https://github.com/rebing/graphql-laravel/actions/runs/4592449345

Not having a clue yet. I found https://github.com/rebing/graphql-laravel/actions/runs/4592908080 that Laravel 6 still works, but all other versions fail (that's the 8.x branch; L6 has been removed from master, that's why master also fails completely).

Since L6 or L8 hardly get updated, I checked the 8.x branch in Laravel, but there were no recent changes which would attribute to any of this (https://github.com/laravel/framework/tree/8.x ), also I wouldn't expect this project the only one being impacted.

Must be one of the dependencies, so I tried to be smart and compare the composer installation output from the working vs. non-working. But alas, the versions vary greatly simply because of L6 vs. L8 alone. When I removed all the packages from both sides with the same version, or when a package did not exist in the other one, I was left with:

Laravel 6 (working):

  - Installing orchestra/testbench (v4.0.1): Extracting archive
  - Installing orchestra/testbench-core (v4.18.0): Extracting archive
  - Installing phpunit/php-code-coverage (7.0.15): Extracting archive
  - Installing phpunit/php-file-iterator (2.0.5): Extracting archive
  - Installing phpunit/php-text-template (1.2.1): Extracting archive
  - Installing phpunit/php-timer (2.1.3): Extracting archive
  - Installing phpunit/phpunit (8.5.33): Extracting archive
  - Installing sebastian/code-unit-reverse-lookup (1.0.2): Extracting archive
  - Installing sebastian/comparator (3.0.5): Extracting archive
  - Installing sebastian/diff (3.0.3): Extracting archive
  - Installing sebastian/environment (4.2.4): Extracting archive
  - Installing sebastian/exporter (3.1.5): Extracting archive
  - Installing sebastian/global-state (3.0.2): Extracting archive
  - Installing sebastian/object-enumerator (3.0.4): Extracting archive
  - Installing sebastian/object-reflector (1.1.2): Extracting archive
  - Installing sebastian/recursion-context (3.0.1): Extracting archive

Laravel 8 (not working):

  - Installing orchestra/testbench (v6.27.1): Extracting archive
  - Installing orchestra/testbench-core (v6.31.1): Extracting archive
  - Installing phpunit/php-code-coverage (9.2.26): Extracting archive
  - Installing phpunit/php-file-iterator (3.0.6): Extracting archive
  - Installing phpunit/php-text-template (2.0.4): Extracting archive
  - Installing phpunit/php-timer (5.0.3): Extracting archive
  - Installing phpunit/phpunit (9.6.6): Extracting archive
  - Installing sebastian/code-unit-reverse-lookup (2.0.3): Extracting archive
  - Installing sebastian/comparator (4.0.8): Extracting archive
  - Installing sebastian/diff (4.0.4): Extracting archive
  - Installing sebastian/environment (5.1.5): Extracting archive
  - Installing sebastian/exporter (4.0.5): Extracting archive
  - Installing sebastian/global-state (5.0.5): Extracting archive
  - Installing sebastian/object-enumerator (4.0.4): Extracting archive
  - Installing sebastian/object-reflector (2.0.4): Extracting archive
  - Installing sebastian/recursion-context (4.0.5): Extracting archive

My eyes were on orchestra/testbench , as it's one of the most integral parts between the tests and laravels setup. This was in fact a quick win, because I found the regression between two releases:

first, remove orchestra/testbench-core from composer.json to ease installing specific versions
remove any composer.lock you might have

GOOD: composer require --dev orchestra/testbench:8.0.11 -W

./composer.json has been updated
Running composer update orchestra/testbench --with-all-dependencies
Loading composer repositories with package information
Updating dependencies
Lock file operations: 0 installs, 2 updates, 0 removals
  - Downgrading orchestra/testbench (v8.1.0 => v8.0.11)
  - Downgrading orchestra/testbench-core (v8.2.0 => v8.1.2)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 0 installs, 2 updates, 0 removals
  - Downgrading orchestra/testbench-core (v8.2.0 => v8.1.2): Extracting archive
  - Downgrading orchestra/testbench (v8.1.0 => v8.0.11): Extracting archive
Generating autoload files
93 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
No security vulnerability advisories found

+ $ vendor/bin/phpunit --filter testWithoutSelectFields
PHPUnit 9.6.6 by Sebastian Bergmann and contributors.

Warning:       Your XML configuration validates against a deprecated schema.
Suggestion:    Migrate your XML configuration using "--migrate-configuration"!

.                                                                   1 / 1 (100%)

Time: 00:00.157, Memory: 40.50 MB

OK (1 test, 3 assertions)

BAD: composer require --dev orchestra/testbench:8.1.0 -W

./composer.json has been updated
Running composer update orchestra/testbench --with-all-dependencies
Loading composer repositories with package information
Updating dependencies
Lock file operations: 0 installs, 2 updates, 0 removals
  - Upgrading orchestra/testbench (v8.0.11 => v8.1.0)
  - Upgrading orchestra/testbench-core (v8.1.2 => v8.2.0)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 0 installs, 2 updates, 0 removals
  - Upgrading orchestra/testbench-core (v8.1.2 => v8.2.0): Extracting archive
  - Upgrading orchestra/testbench (v8.0.11 => v8.1.0): Extracting archive
Generating autoload files
93 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
No security vulnerability advisories found

+ $ vendor/bin/phpunit --filter testWithoutSelectFields
PHPUnit 9.6.6 by Sebastian Bergmann and contributors.

Warning:       Your XML configuration validates against a deprecated schema.
Suggestion:    Migrate your XML configuration using "--migrate-configuration"!

F                                                                   1 / 1 (100%)

Time: 00:00.175, Memory: 40.50 MB

There was 1 failure:

1) Rebing\GraphQL\Tests\Database\SelectFieldsTest::testWithoutSelectFields
SQL queries mismatch
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-'select * from "posts" where "posts"."id" = ? limit 1;'
+'select * from "posts" where "posts"."id" = ? limit 1;
+select * from "posts" where "posts"."id" = ? limit 1;'

/Users/neo/src/graphql-laravel/tests/Support/Traits/SqlAssertionTrait.php:124
/Users/neo/src/graphql-laravel/tests/Database/SelectFieldsTest.php:85
/Users/neo/src/graphql-laravel/vendor/orchestra/testbench-core/src/TestCase.php:96

FAILURES!
Tests: 1, Assertions: 1, Failures: 1.

I noticed this already during debugging that the event was fired twice or so.

Checked https://github.com/orchestral/testbench/releases/tag/v8.1.0 and there we have it:

Add supports for setup<Concern> and teardown<Concern> with imported traits.

We do this in

graphql-laravel/tests/TestCaseDatabase.php

Lines 26 to 28 in 8b156d5

    
           if (isset($uses[SqlAssertionTrait::class])) { 
        
               $this->setupSqlAssertionTrait(); 
        
           }

and no effectively due to this change, the trait was set up twice.

It's likely this commit from testbench-core orchestral/testbench-core@08064ff

Actually, it's this commmit which introduced the feature: orchestral/testbench-core@b824c5e (I already looked at a refactored version which required excluding hard-coded traits, probably due to the double-setup reason we've here).

8.x is fixed, need to merge it into master but having unrelated merge conflicts -> will check later

	if (isset($uses[SqlAssertionTrait::class])) {
	$this->setupSqlAssertionTrait();
	}