Implement framework for comparing guppy's results to cargo

[sunshowers]

guppy is a reimplementation of parts of cargo. It would be cool to compare guppy's results with cargo's continuously against real-world repositories.

Here's one way to make that happen:

• Add a new binary crate called (say) guppy-comparer which depends on both guppy and cargo.
• In this crate, implement a runner which:
  • takes a Rust workspaces (using the one at cwd is ok to begin with, but may also take path/git/vcs sources or even be based on crater)
  • uses the cargo API to load up the repo
  • uses guppy to load up the cargo metadata
  • knows how to interchange guppy and cargo data structures
  • generates N (say 10k by default) random queries using proptest
  • runs those queries on guppy and cargo
  • compares results, and fails if any query has non-matching results.

Examples of things to test:

• Once we have implementations of both the classic and new feature resolvers, it should be possible to compare guppy's and cargo's respective resolvers. The random queries would consist of which feature IDs to select and which resolver to use.

[metajack]

Is there an API for queries in cargo? Or are you thinking of somehow adapting the graph to cargo's native structs?

[sunshowers]

There are a few APIs for queries, e.g. https://docs.rs/cargo/0.43.1/cargo/ops/fn.resolve_ws_with_opts.html. It won't cover everything that guppy does but it'll still be useful probably!

[sunshowers]

@bmwill asked me about why this would be useful when cargo metadata already has resolved data. The reason is that cargo metadata has resolve results for the full workspace; the resolution for a subset of workspace packages might be a bit more restricted. It'll be good to compare results for subgraph resolutions.

[sunshowers]

Done with cargo-compare.