brson/stdx

Create stdx-check tool

brson opened this issue · 9 comments

brson commented

One of the main roles I have in mind for stdx is as tool to measure the compatibility of the larger ecosystem. The stdx-check tool determine whether any given crate was 'compatible' with stdx, serving two purposes:

  • By running stdx-check across every crate on crates.io (using cargobomb probably), we could say what percent is compatible, which would inform the development of stdx, and also give users confidence that stdx is actually leading them down the right path.
  • Users could run stdx-check on their own crates, to validate their stdx compatibility

At the most basic level I imagine it doing the following:

  • Rewriting ./Cargo.toml to add the a stdx dependency (from git for now perhaps), outputting to Cargo-stdx.toml
  • Running cargo test on the new manifest to see if it passes
  • Inspecting the generated lockfile to see if any of the stdx crates have dupes.

Other nice follow on work:

  • Support multiple stdx revisions, probably by include_string their tomls
  • Verify a crate directly off of crates.io, which would require frobbing their Cargo.tomls.
  • Write a tool that runs stdx-check across a list of crates.io crates and generates a report with the results, indicating stdx ecosystem compatibility.
davidpeklak commented

Hi, when I try to run cargo test --manifest-path Cargo-stdx.toml, cargo complains, saying "the manifest-path must be a path to a Cargo.toml file". (It seems it does not accept files with a different name.) Do I have to output to a "Cargo.toml" file, but in another directory?

brson commented

@davidpeklak Seems like it'll have to go into a different directory then, yeah.

brson commented

Though that makes things much more complex, since any relative paths will be broken. It might be easier to either:

  • Duplicate the entire directory structure and rewrite the resulting Cargo.toml
  • Back up Cargo.toml, rewrite Cargo.toml, restore the original Cargo.toml
Arnavion commented

Is there a reason cargo only accepts Cargo.toml even when the manifest path is explicitly provided? Perhaps that could be the thing to change instead.

Ref https://github.com/rust-lang/cargo/pull/1955/files#diff-bfa065cd67c70cba1eccd5787281245fR47

davidpeklak commented

I offer this cargo custom command as a first solution to issue #35, hoping to fulfil what is described as 'the most basic level' above. Note that it does not work with the current version of stdx, because of issues with some dependencies of stdx.
However, I got it working with a stripped-down version of stdx

cargo stdx-check --stdxgit https://github.com/davidpeklak/stdx.git

on the walkdir crate.
If this is the right direction, I am happy to spend more time on this.

brson commented

@davidpeklak oh wow thank you! Yes, that is exactly the kind of thing I was hoping for. The README description sounds like it is pretty much doing exactly what I wanted.

What's going on with 'chrono'? Why can't cargo find it? That seems pretty weird...

Ok, so what do you think the next steps are?

We could start building out the tooling for running stdx-check on ecosystem crates. A really simple way to start this would to just create a static list of crates and have the stdx CI run stdx-check over all of them every commit, report the percentage compatibility, maybe push them to a GitHub pages. We could start with perhaps this list, maybe add a number of higher-level crates as well. Would need to unbreak the CI, but I'm sure that's not hard to just disable whatever is busted. I like this idea a lot.

You might consider merging stdx-check into this repo, though I understand if you would like to keep it independent.

Please do email me if I fail to follow up in a timely way.

davidpeklak commented

@brson thanks for the feedback!
Regarding chrono: v0.3.1 has been yanked. I have raised issue #46 to upgrade the dependency to v0.4.0 and created PR #47 to fix it.

Regarding the next steps, I'll try to follow your suggestions, can I use this thread if I have questions or is IRC a better way?

brson commented

@davidpeklak please ask on thread here. I will try to circle back regularly, but if you are on IRC please do ping me with the link.

davidpeklak commented

@brson I would like to merge cargo-stdx-check into this repo, but I am not sure how. I tried making stdx a workspace and add cargo-stdx-check as a crate, but it complains:

warning: compiling this new crate may not work due to invalid workspace configuration

current package believes it's in a workspace when it's not:
current:   stdx/cargo-stdx-check/Cargo.toml
workspace: stdx/Cargo.toml

this may be fixable by ensuring that this crate is depended on by the workspace root: stdx/Cargo.toml

It does not make sense to me to add it as a dependency though. Do you have any suggestions?