Allow crates to be published with cyclic dev-dependencies
Marwes opened this issue · 43 comments
In gluon I have the project split up into multiple crates with the intent that users can include only the crates they need if they desire a smaller binary footprint. For convenience I provide a main gluon crate which provides an easy to use interface when one uses all the crates.
While this split works fine for mostly everything I want to use the easy interface when writing documentation tests so I added the gluon crate as a dev-dependency in gluon_vm and this compiles without issue since a little bit back. It does however break when trying to publish gluon_vm as the gluon dev-dependency has not been published yet!
Currently I managed to work around it by specifying a version range but it would be nice if there was a way to publish a group of crates as a whole or to ignore the version check if a dev-dependency does not exist on crates.io (yet).
gluon = { path = "..", version = "<0.6.0, >=0.4.2" } # GLUON
Yeah in general I think we should basically ignore dev-dependencies when we publish. We shouldn't attempt to resolve them, verify they're already published, etc.
Maybe dev-dependencies don't have to be shown in the crate's page too.
I bumped into this issue as well. From what I can tell the workaround above gets us by this for now. 👍
+1 to this-- it now affects the futures crate.
This affects a lot of crates, what do we need to do to fix this? Isn't there a check somewhere that checks for the dependencies before publishing, where we can just filter dev-dependencies away and call it a day? [0]
I mean, this would even affect core and std, where std could depend on core to build, but core would depend on std for tests. The check is spurious anyways, because this sort of dependencies are not circular: build core -> build std -> build core tests is ok. Tests and "dev-only" artifacts are their own crates (core+tests is a different crate than core, and so are every test/foo.rs in core, and core/examples, etc.).
I'll admit that the core/std analogy is imperfect because these crates are too magical, but when something wouldn't be correct for the std library, chances are it won't be correct for users either.
FWIW I just tried to prepare coresimd/stdsimd for a quick crates.io release, and ran into this, where coresimd builds standalone and stdsimd depends on coresimd, but coresimd depends on stdsimd for testing.
The only solution I came up with is the same that gluon uses which is insane: publish coresimd prunning dev-dependencies, publish stdsimd, re-publish coresimd with a minor version update to use the newly publish stdsimd as dev-dependencies.
Which is a lot of effort, for.. what exactly? Is it possible for users to execute cargo test in their whole dependency graph?
[0] cc @matklad, are you willing to mentor this?
It might be enough to just skip dev dependencies here:
cargo/src/cargo/ops/registry.rs
Line 110 in af3f1cd
and also in packaging: https://github.com/rust-lang/cargo/blob/0e7a46e3276f56822b6ef48e341c522be27db17e/src/cargo/ops/cargo_package.rs
The only solution I came up with is the same that gluon uses which is insane: publish coresimd prunning dev-dependencies, publish stdsimd, re-publish coresimd with a minor version update to use the newly publish stdsimd as dev-dependencies.
Hm, I think a simpler work-around is possible? Just comment-out dev-dependencies when publishing. This is what we do in Cargo, which has a path = dev-dep.
For mentoring instructions, I believe you've basically nailed it :) The only thing I have to add is that the test should go to this file, and it probably makes sense to start with a test and work through all errors.
For the tests, I think we should cover at least two scenarios:
- publishing a path dev-dep should not fail (slight modification of this test)
- publishing with a "cycle" via a dev-dep should not fail (the original motivation for the issue).
Hm I'm actually not sure that the fix would go in that location nor that we could easily write a test for this. AFAIK the check for "does this dependency exist" is done on crates.io, not locally in Cargo. In that sense we'd need to test with crates.io, not with just local Cargo test suite business.
I also think the "fix" for this is to basically stop informing crates.io about dev-dependencies. Although we did that initially I'm not really sure why we need to do that, as they're never used and are otherwise just bloating the index. The fix for that would go around here by filtering out dev-dependencies. Cargo would then also need to be updated to not verify that dev-dependencies have versions (allowing path dependencies without a version).
This would be a relatively large change though (and somewhat of a policy shift) so it would likely need some more scrutiny and discussion before happening.
I also think the "fix" for this is to basically stop informing crates.io about dev-dependencies.
Hm, probably a crazy idea, but, given that we already modify Cargo.toml on publishing to rewrite path dependencies (source), could we, at the same step, just drop dev-deps altogether as well?
Yes I think such a fix would work, but it alone wouldn't be 100% sufficient because crates.io doesn't actually parse the manifest for dependency information (it only uses what was sent in JSON)
I guess the proposed change would also now allow git dependencies in dev-dependencies?
I just bumped into this issue as well. Is there a workaround?
You can manually strip dev-dependencies from your Cargo.toml before publishing.
Fair enough, thank you!
I’ll just do that for the first crate, then push the second, then make a patch version to the first one to restore everything.
For some reason, cargo requires you to commit all your changes before publishing, but since crates.io doesn't require the code that's published to match the code in the linked repository, what I do is branch the crate, remove dev-dependencies everywhere, publish everything, and then delete the branch.
You don’t need that: cargo publish --allow-dirty.
I just blogged about what I did here. Thanks again for your help!
Bump. The manual comment out dev deps seems like a real kludge. If that's the solution, cargo publish should just do it.
trying to automate publishing to crates.io as part of CI/ops workflows
what's a good tool to safely comment a few dozen interdependent dev-dependencies across repos and then reinstate them from circleci after the cargo publish completes?
what's a good tool to safely comment a few dozen interdependent dev-dependencies across repos and then reinstate them from circleci after the cargo publish completes?
FWIW I use a shell script to just rename the Cargo2.toml into Cargo.toml and publish with allow-dirty. I suppose a more robust solution could use the toml crate to parse the Cargo.toml file, remove all dev-dependencies keys, and write a different Cargo.toml file without them that gets used instead. Shouldn't be too hard, but unless you have a lot of files that you need to alter probably not worth the effort.
When I change my Cargo.toml, I just copy it over to Cargo2.toml, and cargo publish will complain of all the dev-dependencies anyways so I can't really forget to remove them there.
futures/ci/remove-dev-dependencies uses toml_edit to remove dev-dependencies from a workspace in-place. It's used during CI for testing rather than publishing though, I'm not sure if you'd want to change anything about it for publish, but it's a pretty trivial tool.
Hm, probably a crazy idea, but, given that we already modify Cargo.toml on publishing to rewrite path dependencies (source), could we, at the same step, just drop dev-deps altogether as well?
How would you run tests that rely on dev-dependencies if cargo just trims all of them on publish?
How would you run tests that rely on dev-dependencies if cargo just trims all of them on publish?
That won't work, because the dev-dependencies required won't be available under that solution. We need a better solution.
i thought dev dependencies aren't used by anything published, and tests aren't run against what is published?
Some users want the tests of the crates in crates.io to work, and that is kind of reasonable, e.g., if you are building a crate with many dependencies on a not widely tested platform, you might want to run the tests of all your dependencies to discover if there is a bug in any of those crates on that target.
Another application is crater, which downloads crates from crates.io and is able to run its tests.
The hack of removing the dev-dependencies to make cargo publish work means that if you download a crate from crates.io and run its tests, they won't compile because dependencies are missing.
One major user of running tests on published packages is Crater, it's actually in the developers best interests to make sure their tests run from the package so that their code is tested when potentially-breaking changes are proposed in rustc. (Although if it's in a public git repo it may be tested by Crater from there as well)
i don't know what the solution is then, because its very common to need "circular dependencies" when doing integration testing across related crates because you want to spin up several fixtures etc. for context in the test
i don't know what the solution is then
AFAICT, the first solution proposed in this thread satisfies all constraints without downsides.
An alternative approach that might also solve the problem would be to allow crates.io to stage releases in a group, allowing the capacity to upload, without necessitating publishing, and then, when sufficient uploads are in place to solve dependency resolution, flip a switch to have them all become "published" simultaneously.
That's the real problem here right?
That circular dependencies require an atomic release of the relevant assets.
Dev dependencies are kind of a weird concept. There isn't a single set of dependencies used for all development tasks. It depends on the environment, e.g. CI won't be running build in watch mode.
Dev dependencies are kind of a weird concept. There isn't a single set of dependencies used for all development tasks. It depends on the environment, e.g. CI won't be running build in watch mode.
Indeed, this is why I also want to champion a more explicit "test" dependency grade, because the broad spectrum of "development" is far too nebulous.
https://internals.rust-lang.org/t/explicit-test-dependency-grade/11107
Dev dependencies are kind of a weird concept. There isn't a single set of dependencies used for all development tasks. It depends on the environment, e.g. CI won't be running build in watch mode.
i feel the same way about how much of development tooling is not compatible with stable rust 😅
I was looking at this issue as a possible first contribution due to the E-mentor label. But it seems to not be agreed what to do, even among experienced contributors.
I understood that Alex's proposal was to remove the dev-dependencies from the JSON that is used in communicating with crates.io, but leave the manifest (Cargo.toml) untouched. Thus when you downloaded a crate (plus a crate's dependencies) the dev-dependencies for tests would be compiled and available during running of tests.
@andrewdavidmackenzie that's about right. As mentioned, that would be a fairly significant policy change (and would, for example, remove the dev-dependencies display on crates.io, which would probably be good to have input from the crates.io team). Perhaps one compromise would be to only filter out dev-dependencies that are involved in a cycle, though I'm not sure how difficult that would be or what problems that might cause.
Since matklad is no longer on the team, and I don't think anyone particularly has bandwidth to mentor this right now, I'm going to remove the mentor tag. This probably needs some discussion with the community (maybe on IRLO), to see what impact it might have. It might also need buy-in from both the cargo and crates.io teams.
That depends on whether crates.io shows those dependencies from the JSON submitted via API or via the uploaded cargo.toml no?
Ok, I'll look for a simpler first issue. When I looked their were 3 easy and 3 mentored....now down to 5....and none of them look particularly easy to get stuck into as a noob.
An alternative approach that might also solve the problem would be to allow crates.io to stage releases in a group
🤔 cargo publish --all (would publish everything in current workspace that's a) not publish = false b) has higher version than already published). This seems to be already an existing idea #1169, but if that implemented, it would be nice if these played well together. I mean, one usually has these dev-dependency cycles inside a workspace.
Removing dev dependencies from crates when publishing means that the tests don't build on published versions. This is causing problems for us in AOSP, where we vendor in crates from crates.io (as that's the definitive source of released versions), but want to be able to run tests on them. Using Cargo.toml.orig for this isn't a solution either, as it may have path dependencies which won't make sense.
We have no way to guarantee the test will run from there publish artifacts. They could require all kinds of additional setup. (I've been known to write tests that require access to a command line SAT solver.) Any attempt to run tests from published artifacts has to be on a best effort basis.
futures/ci/remove-dev-dependenciesusestoml_editto remove dev-dependencies from a workspace in-place. It's used during CI for testing rather than publishing though, I'm not sure if you'd want to change anything about it for publish, but it's a pretty trivial tool.
It was removed in rust-lang/futures-rs@73ade1e1
It was replaced with https://github.com/taiki-e/cargo-hack#--no-dev-deps , which works for publishing.
Eventually, futures was able to avoid the need to edit the manifest before publishing by removing version from path dev-dependencies: rust-lang/futures-rs#2305
PR #7333 added auto-stripping of dev-dependencies that do not have a version specified. This was actually added for when a dev-dependency exists but isn't published. See rust-lang/crates-io-cargo-teams#46 & rust-lang/crates.io#1789. cargo add takes advantage of this by not adding the version field for path dev-dependencies. That gets us through cargo package verification and whatever crates.io does.
However, people using this, especially with cargo add making it the default, decreases the coverage we get from crater. The question is if there is anything more we can and should do.
One idea is to completely ignore dev-dependencies at each stage of the publish process. At minimum, we need to ensure the Cargo.lock is appropriate for uploading. Maybe we could skip that in some cases but now we need to find the right rules (or add configuration) to control this. I believe we already exclude transitive dev-dependencies when generating your Cargo.lock so I assume that should be fine if they don't exist. No idea if we can do similar for cargo install. So maybe something can be done but there are a lot of ifs and potential caveats to it. The question is if the pay off is enough for this at this point.
btw rust-lang/rfcs#3452 could allow nesting dev dependencies as another way of breaking cycles.
Most of the time I see this discussed, it is in the context where a path dependency exists to a package that might not be published yet.
There are cases, like async-stream and tokio-test (see tokio-rs/async-stream#102) where path dependencies are not involved because its cross-repo. This simplifies the problem because the cyclic dev-dependencies have to be published for any of this to work. As mentioned earlier in this thread and again in https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/Is.20dev-dependency.20information.20from.20the.20index.20used.3F, dev-dependencies don't matter for the index, so we could strip them from the Summary we upload. We could even keep them in the Cargo.toml for now to not interfere with crater.