nextest-rs/nextest

Feature Request: Conditional skipping test based on setup scripts

andylokandy opened this issue · 1 comments

andylokandy commented

Motivation

In large-scale projects or projects heavily integrated with third-party services, it is often necessary to skip certain tests when specific conditions are not met. For instance, if a required database is not available, running integration tests dependent on this database should be skipped to save time and avoid irrelevant test failures.

Existing solutions

  • Rust issue #68007: A long-standing issue suggests runtime test skipping but hasn't progressed. Need fundamental change on libtest.

  • Community crate test-with: This uses a proc macro to add #[ignore] at compile time. While useful, it lacks the flexibility.

Proposal

Enhance nextest to support conditional test skipping based on the output of a setup script. The configuration may look like:

[[profile.default.scripts]]
filter = "test(~integration)"
run-if = 'check-db'
# alternatively, we can implement an option `skip-if-fail`
# setup = 'check-db'
# skip-if-fail = true

[script.check-db]
command = 'cargo run -p check-db'

When the script outputs 'false', the tests in the specified group (filtered by "test(~integrate)") should be marked as SKIP.

Finished `test` profile [unoptimized + debuginfo] target(s) in 0.13s
Starting 5 tests across 1 binaries (run ID: 7d7253f2-e050-46cd-98ef-0815cff44a4c, nextest profile: default)
    PASS [   0.002s] ut::foo
    PASS [   0.003s] ut::bar
    PASS [   0.003s] ut::baz
    SKIP [   0.003s] integration::foo
    SKIP [   0.003s] integration::bar
------------
    Summary [   0.022s] 5 tests run: 3 passed, 2 skipped

On CI, we can run cargo nextest run --run-ignored all to force them to run.

sunshowers commented

Thanks for the report! This seems like a good idea. The way I'd probably do this is: currently, setup scripts are either pass or fail. We'd want to add a third "skip" state for them.

As for how to indicate this: the most obvious answer is a special exit code, but that can be a bit brittle. So maybe we should have people write something like "skip" out to a file, along with maybe a message that can be displayed.

To set expectations: We might need this at some point at work, but otherwise I don't imagine I'll get to this soon.

  • 👍2