Allow using source replacement with fallback to crates.io
Opened this issue ยท 12 comments
We're working on using source replacement in Firefox so that we can vendor our crate dependencies into mozilla-central. This works pretty well in practice, but we'd also like to enable developers to do "normal" Rust development without extra hassle. The current implementation includes adding a .cargo/config
in the Firefox objdir to point at our vendored crates, and also running cargo build --frozen
to ensure that we're not hitting the network. For our CI builds this is perfect. For local development it's not. We talked about adding a --enable-rust-developer-mode
switch to turn those off, but it would be nice if we could do that by default in non-automation builds. However, if we don't do source replacement by default that means that Firefox builds will start requiring a network connection, which we have previously not required. Additionally, it seems silly to fetch crates from crates.io when they're sitting there in the source checkout.
It would be great to have a way to use source replacement but allow fallback--we should be able to point cargo at our vendored crates, but if someone adds a new dependency cargo should be able to fetch it from crates.io as usual (assuming we're not building with --frozen
). This way the default build would simply use our vendored crates, but adding new crate dependencies would not require adding any special build options, and developers would just run the mach vendor
command we'll be adding to vendor in the new dependencies for landing.
This sounds like a good feature to have to me! My preference of how to do this would be a "multi source" along the lines of:
[source.foo]
# ...
[source.bar]
# ...
[source.baz]
any = ['foo', 'bar']
[source.crates-io]
replace-with = 'baz'
The definition of an "any" source would be to just check each source in order and return the first that has the package. Cargo would then also assume that all sources have the same checksum, but wouldn't verify it until you built a crate.
For simplicity, could we use the same configuration for "developer mode" and "CI mode", and just have the latter build with --frozen
?
I believe that would work, yeah, at least with what I'm thinking
Similarly, when using a top-level shared vendor directory and top-level source replacement config and a developer wants to add a new dependency, there appears to be no way to easily vendor in the new dependency without blanking out the source replacement configuration, running cargo vendor
, and then re-adding back the configuration. This is a pain, but perhaps I missed a better way?
How about something like cargo vendor sync
or cargo vendor download
that bypasses the vendor / source replacement config for just that command...which might cover this case too?
Of course, if someone is experimenting/developing they could add dependencies that ultimately aren't needed and bloat the repo...maybe a better way would be cargo vendor stage
(looks in vendor dir, falls back to crates.io, but DOESN'T write it to the global vendor and instead writes it to a local vendored spot).
The developer experience might look like:
- Add deps to
Cargo.toml
- Run
cargo vendor stage
. This downloads missing crates from crates.io and vendors somewhere temporary (or downloads to the global vendor directory with a file named.VENDOR-STAGED
so that downstream tools can do smart things). - Iterate, perhaps adding or removing deps.
- Run
cargo vendor sync
to download and save any deps in thevendor
dir (or remove.VENDOR-STAGED
in each crate from step #2). - Commit changes in version control
Not sure this is 100% the same as the original issue, feel free to tell me to file another issue.
We do have the same issue in the Firefox build, but we sort of cheat around it because the .cargo/config
file that specifies the source replacement is placed in the objdir, which is where we run cargo during the build, but mach vendor rust
runs cargo vendor
from the srcdir, so it doesn't use the source replacement.
@LegNeato yeah those sound like real issues! Want to open a or some bugs over at cargo-vendor?
@luser yeah, I've been reading bugzilla about your setup ๐ . I think we need to support not having a meta buildsystem / script layer above making the end-to-end vendoring workflow in the shared vendor case...even if most big companies will merely call out via their existing tools. I'll file some new issues!
This was discussed at the recent Mozilla all-hands (cc @froydnj @rillian) as an issue that'd help Gecko quite a bit. Instead of what I mentioned above though I think I'd probably advocate instead for something like:
[source.crates-io]
vendor-cache = 'path/to/vendor'
Or something along those. I don't think we should prematurely try to generalize this, and let's just stick to the workflow of "I've got vendored crates in my source tree which come from crates.io" and then Cargo can just "do the right thing"
That sounds fine. As I mentioned earlier, if the same config would "just work" for our CI and our developer builds that would be great, and we would simply make CI builds use --frozen
but not developer builds, so that forgetting to vendor something is still an error in CI, but cargo would be free to simply fetch the new crate for a local build.
Yeah I don't think it'd be too hard to set up something like that, I think we could make it work!
Hi, was this ever implemented? I can't find anything in the documentation: https://doc.rust-lang.org/cargo/reference/config.html
I'm trying to use cargo vendor
as an alternative to the currently hack-ish solutions to only build the dependencies of a project (for caching purposes), and I guess cargo vendor
could work?
But without a crates.io
fallback this doesn't sound like a great idea.
8 years on, and this is still an issue. I also need vendor'd crates for CI to work, but that breaks local development without a crates.io fallback.