test.sequential for tests that rely on the sequence

[turadg]

The problem

In our organization the vast majority of uses of test.serial are just to enforce that the tests don't run in parallel. They can still be run independently with the -m to select.

In some of our tests, test.serial is used for tests that depend on a sequence of execution. For example, a scenario test in which there are phases (e.g. A, B, C). Each test is both testing a phase and setting up the next phase. We could turn it into one giant test, but they are logically independent and a failure in A can allow B to still be tested and pass. We could have multiple tests like A[setupA,testA] and B[setupA,setupB,testB] but these setups are very slow and we want to do each only once.

Proposal

Maintain test.serial as "not parallel". Consider randomizing the order so that it's not relied upon to be executed in sequence.

Add a test.sequential for tests that actually do depend on the sequence. Selecting one of these with "-m" executes the preceding tests.