nextest-rs/nextest

Run --no-default-features and --all-features suites

Opened this issue · 4 comments

tailhook commented

Hi! Nextest looks pretty great!

One common thing I have in many crates is running tests with different feature sets. Would be nice to be able to merge those tests into the single set of tests instead of running sequentially via separate commands, if it's at all possible.

sunshowers commented

Hi there, thanks for the feature request!

I think this gets complicated really soon:

  • just the standard none/default/all features or the full power set of features
  • exclude certain feature combinations
  • mutually exclusive features
  • etc

I think this should probably be handled outside of nextest rather than inside it. For example, in CI you could use a matrix build strategy, and for more advanced behavior you could write your own tool that computes the power set of features. With matrix builds in particular, it's likely that doing so in several jobs is likely to be significantly faster than doing everything in the same job.

I'm really hesitant to add this to nextest itself, I think the feature doesn't really carry its weight in terms of maintenance burden (especially because it can't be reused across cargo build, cargo clippy etc).

Let me know what you think!

sunshowers commented

BTW if you want to build something like this yourself, check out https://crates.io/crates/guppy, another thing I wrote (which nextest uses, in fact!)

tailhook commented

Couple of thoughts:

  1. Running those as separate jobs in CI is good, but it sometimes possible to forget to add a feature to the list of tests (if nextest will work both locally and on CI and tests are sharded rather than sliced by feature it might be better)
  2. Running locally is more of a problem since you have to use a wrapper command, like cargo-make, and then you will not see the aggregated table of results
  3. For the feature sets, you're right, some config should be used to define exact combinations of features that needs to be tested. So if there is no other purpose of the configuration file for nextest it's kinda extra burden. But if nextest have some config anyway (say for #27) same approach may be used, I think.
  4. Generally, in Rust ecosystem it's quite common to forget testing different feature sets (in the last few months, I've seen two libraries that had a bug in --no-default-features), so I think it's valueable for the community.
sunshowers commented

Thanks! I totally agree it's valuable for the community, I just don't think nextest itself should implement it internally. If someone else (like yourself!) implements this as a library/binary pair for use in the Cargo ecosystem generally, I'd be happy to see if nextest can integrate it in the future.

I think the point about sharding is really interesting! One of the issues is that you end up having to do lots of builds because of the way Cargo's feature unification works, so figuring out an optimal strategy is quite complicated.

  • 👍1