rust-lang/cargo

Tracking issue for -Z features=host_dep

ehuss opened this issue · 18 comments

ehuss commented

Implementation: #7820
Documentation: https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#features

Summary
-Z features=host_dep causes the feature resolver to prevent features enabled on build dependencies or proc-macros from being enabled for normal dependencies. For example:

[dependencies]
log = "0.4"

[build-dependencies]
log = {version = "0.4", features=['std']}

When building the build script, the log crate will be built with the std feature. When building the library of your package, it will not enable the feature.

Unresolved issues

  • Update cargo metadata
  • Does this cause existing projects to fail?
    • Projects can add missing features as a backwards-compatible workaround. However, if it causes problems for too many projects, we may need to find some way to opt-in.
  • Does this significantly increase build times?
    • This can cause common dependencies to be built multiple times, when they were previously shared (when --target is not used). Projects can add features to re-unify if they want, but it may not be clear or obvious what to add and when. We may need something that identifies these duplicate dependencies, and give suggestions on how to unify them. Or, we may need to make this opt-in somehow so the user needs to make this an explicit decision.
  • Does the feature resolver need to support decoupling of shared proc-macro dependencies?
    • Currently this isn't possible because Cargo does not have the necessary information to know which dependencies are proc-macros during resolution time. Either we would need to add the proc-macro status to the index, or try to do feature resolution while dependencies are downloaded. The latter is very difficult due to the way downloading works. The former isn't too hard. Nobody has asked for this ability, so it is unclear how important it is. Presumably that is because proc-macros are relatively new, and tend to only use proc-macro-specific dependencies like syn/quote.
    • Implemented in #8028.
Amanieu commented

Does the feature resolver need to support decoupling of shared proc-macro dependencies?

Yes please. I've encountered an issue related to this already when a proc-macro enabled many features in the rand crate which caused my build to fail since my target does not have std. See rust-lang/hashbrown#124 for details.

  • 👍9
valarauca commented

Does the feature resolver need to support decoupling of shared proc-macro dependencies?

Seeing as #4866 was about proc-macro's it does seem to be in demand. Also could #4866 be re-opened as #7820 doesn't address it?

  • 😕1
tarcieri commented

UPDATE: never mind, this is a red herring. the proc macro itself was referencing std, because I forgot to disable its std feature 😅. With the std feature it works, but I can see how this could come up in dependency-laden proc macros that activate the std feature in dependencies shared by the proc macro.

I just ran into the proc macro problem trying to use displaydoc in a no_std project:

$ cargo build -Z features=build_dep --no-default-features --target thumbv7em-none-eabihf --release
error[E0433]: failed to resolve: use of undeclared type or module `std`
 --> rust/src/error.rs:7:30
  |
7 | #[derive(Copy, Clone, Debug, Display, Eq, PartialEq)]
  |                              ^^^^^^^ use of undeclared type or module `std`

It would definitely be super duper handy to be able to use proc macros that link against std in no_std targets, and also to ensure there's some open issue tracking it until it's fixed.

Seeing as #4866 was about proc-macro...

@valarauca it looks like some people discussed proc macros in the comments, but #4866 itself was about build-dependencies

Perhaps it's worth opening a new issue?

ia0 commented

I would also be interested in having -Z features=build_dep work with proc-macro. I would use it for the data-encoding-macro crate.

  • 👍1
ehuss commented

proc-macro support has been added in #8028, and should be in the latest nightly (2020-03-27). proc-macros and build dependencies are unified (since they are all "host" artifacts), but shouldn't affect anything else. Also, the option has been renamed to -Zfeatures=host_dep to reflect this change (the old build_dep option is an alias). I'd appreciate it if people could take it for a test drive and let me know how it works. This resulted in a major change in how crates are downloaded, so any testing is appreciated.

  • 🎉4
  • 🚀3
valarauca commented

I created this "project" as a slightly contrived example.

  1. real_project configures fake-dep (via fake-dev-dep) to panic. see: link
  2. real_project requires fake-dev-dep to panic . The real_project/build.rs file will exit(1) if fake-dep does not panic). see: link
  3. real_project by default expects fake-dep to not panic within its test link

Results:

  • On stable running cargo test within real_project results in test failure (test::crate_worked_successfully panicked at hello world)
  • On nightly-2020-03-18 running cargo test within real_project results in a test failure (test::crate_worked_successfully panicked at hello world)
  • On 2020-03-27 running cargo test -Zfeatures=host_dep the test passes!

I hope this is a valid tests. The idea is build-dependencies and dependencies should have different, and conflicting features.

The results seem positive. Thanks for the work on this!

tinoadams commented

For what it's worth, my sample project which cross compiles to thumbv7m-none-eabi used to complaint about memchr needing std as soon as I imported bindgen in the build-dependencies. With the new dependency resolution strategy I can now run bindgen in build.rs during build time. Hooray and thank you for the great enhancement.

https://github.com/tinoadams/bluepill_mynewt_rust_blink/blob/master/rust/app/Cargo.toml

  • 🎉1
sbourdeauducq commented

Is there a way to enable this via .cargo/config or Cargo.toml, or do I need to pass -Z features=host_dep manually to Cargo every time?

phil-opp commented

Is there a way to enable this via .cargo/config or Cargo.toml, or do I need to pass -Z features=host_dep manually to Cargo every time?

Probably a bit offtopic, but I opened #8127 to propose a general solution to this problem.

  • 👍2
  • ❤️1
cbiffle commented

The availability of resolver 2 (and specifically host_dep) has been critical for us being able to continue using Cargo in our embedded build (cross compiling for thumbv7em-none-eabihf). We use serde both in the build (to consume and produce metainformation), and in the firmware (to marshal IPC), and I was preparing to move us to a different build system because of Cargo blindly forcing std on the firmware. I am delighted to report that I have not had to do this.

This has forced us onto a pretty recent nightly, though, which I had been trying to avoid -- resolver is the only reasonable way to express this, rather than expecting all developers to remember to type -Z host_dep every time, and we are using a virtual workspace, so I had to fast forward all the way to nightly-2020-05-01 to get a combination of features that worked / a build with all the artifacts we needed. So I look forward to the fixed resolver behavior making it to stable!

  • 👍6
eddyb commented

@ehuss would it make sense to change the "Summary" in the issue description to say host_dep instead of build_dep?
Or maybe mention that it was renamed, as it was confusing at first.

  • 👍1
BrandonArp commented

This feature is working great for an embedded project I'm working on. Like others, I was hitting no_std and bindgen conflict. It's really great to see this feature! Thanks!

  • 👍3
blaind commented

Is there a way to enable this via .cargo/config or Cargo.toml, or do I need to pass -Z features=host_dep manually to Cargo every time?

Probably a bit offtopic, but I opened #8127 to propose a general solution to this problem.

Seems like this has been merged. Had to search a bit how to use it, so for anyone else who's using this - here's a way to enable the flag in .cargo/config

[unstable]
features = ["host_dep"]
  • 👍3
h4x3rotab commented

When stable?

  • 👍3
  • 👎3
  • 😕1
ctron commented

Works great. Would like to see this too.

jonasbb commented

Hi @ehuss,

three days ago (2021-01-05) I noticed some unstable CI build where I use -Zfeatures=dev_dep,host_dep. I could reproduce it locally only using host_dep, which is why I post here. I never noticed it without the feature.

Error Log 
error[E0053]: method `from_list` has an incompatible type for trait
Error:    --> serde_with_macros\src\lib.rs:431:14
    |
431 |     #[derive(FromMeta, Debug)]
    |              ^^^^^^^^ expected enum `syn::attr::NestedMeta`, found enum `NestedMeta`
    |
    = note: expected fn pointer `fn(&[syn::attr::NestedMeta]) -> std::result::Result<_, _>`
               found fn pointer `fn(&[NestedMeta]) -> std::result::Result<_, _>`
    = note: perhaps two different versions of crate `syn` are being used?
    = note: this error originates in a derive macro (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0053]: method `from_field` has an incompatible type for trait
Error:    --> serde_with_macros\src\lib.rs:465:14
    |
465 |     #[derive(FromField, Debug)]
    |              ^^^^^^^^^ expected struct `syn::data::Field`, found struct `syn::Field`
    |
    = note: expected fn pointer `fn(&syn::data::Field) -> std::result::Result<_, _>`
               found fn pointer `fn(&syn::Field) -> std::result::Result<_, _>`
    = note: perhaps two different versions of crate `syn` are being used?
    = note: this error originates in a derive macro (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0053]: method `from_field` has an incompatible type for trait
Error:    --> serde_with_macros\src\lib.rs:482:14
    |
482 |     #[derive(FromField, Debug)]
    |              ^^^^^^^^^ expected struct `syn::data::Field`, found struct `syn::Field`
    |
    = note: expected fn pointer `fn(&syn::data::Field) -> std::result::Result<_, _>`
               found fn pointer `fn(&syn::Field) -> std::result::Result<_, _>`
    = note: perhaps two different versions of crate `syn` are being used?
    = note: this error originates in a derive macro (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0053]: method `from_derive_input` has an incompatible type for trait
Error:   --> serde_with_macros\src\utils.rs:26:10
   |
26 | #[derive(FromDeriveInput, Debug)]
   |          ^^^^^^^^^^^^^^^ expected struct `syn::derive::DeriveInput`, found struct `DeriveInput`
   |
   = note: expected fn pointer `fn(&syn::derive::DeriveInput) -> std::result::Result<_, _>`
              found fn pointer `fn(&DeriveInput) -> std::result::Result<_, _>`
   = note: perhaps two different versions of crate `syn` are being used?
   = note: this error originates in a derive macro (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0308]: mismatched types
Error:    --> serde_with_macros\src\lib.rs:438:68
    |
438 |     let container_options = match SerdeContainerOptions::from_list(&args) {
    |                                                                    ^^^^^ expected slice, found struct `Vec`
    |
    = note: expected reference `&[syn::attr::NestedMeta]`
               found reference `&Vec<NestedMeta>`

error[E0308]: mismatched types
Error:    --> serde_with_macros\src\lib.rs:431:14
    |
431 |     #[derive(FromMeta, Debug)]
    |              ^^^^^^^^ expected enum `syn::attr::Meta`, found enum `syn::Meta`
    |
    = note: expected reference `&syn::attr::Meta`
               found reference `&syn::Meta`
    = note: perhaps two different versions of crate `syn` are being used?
    = note: this error originates in a derive macro (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0308]: mismatched types
Error:    --> serde_with_macros\src\lib.rs:499:55
    |
499 |     let serde_as_options = SerdeAsOptions::from_field(field)?;
    |                                                       ^^^^^ expected struct `syn::data::Field`, found struct `syn::Field`
    |
    = note:      expected reference `&syn::data::Field`
            found mutable reference `&mut syn::Field`
    = note: perhaps two different versions of crate `syn` are being used?

error[E0308]: mismatched types
Error:    --> serde_with_macros\src\lib.rs:500:59
    |
500 |     let serde_with_options = SerdeWithOptions::from_field(field)?;
    |                                                           ^^^^^ expected struct `syn::data::Field`, found struct `syn::Field`
    |
    = note:      expected reference `&syn::data::Field`
            found mutable reference `&mut syn::Field`
    = note: perhaps two different versions of crate `syn` are being used?

error[E0308]: mismatched types
Error:    --> serde_with_macros\src\lib.rs:465:14
    |
465 |     #[derive(FromField, Debug)]
    |              ^^^^^^^^^ expected struct `syn::attr::Attribute`, found struct `Attribute`
    |
    = note: expected reference `&syn::attr::Attribute`
               found reference `&Attribute`
    = note: perhaps two different versions of crate `syn` are being used?
    = note: this error originates in a derive macro (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0308]: mismatched types
Error:    --> serde_with_macros\src\lib.rs:465:14
    |
465 |     #[derive(FromField, Debug)]
    |              ^^^^^^^^^
    |              |
    |              expected enum `syn::attr::NestedMeta`, found enum `NestedMeta`
    |              this expression has type `syn::attr::NestedMeta`
    |
    = note: perhaps two different versions of crate `syn` are being used?
    = note: this error originates in a derive macro (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0308]: mismatched types
Error:    --> serde_with_macros\src\lib.rs:465:14
    |
465 |     #[derive(FromField, Debug)]
    |              ^^^^^^^^^ expected enum `syn::attr::Meta`, found enum `syn::Meta`
    |
    = note: expected reference `&syn::attr::Meta`
               found reference `&syn::Meta`
    = note: perhaps two different versions of crate `syn` are being used?
    = note: this error originates in a derive macro (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0308]: mismatched types
Error:    --> serde_with_macros\src\lib.rs:482:14
    |
482 |     #[derive(FromField, Debug)]
    |              ^^^^^^^^^ expected struct `syn::attr::Attribute`, found struct `Attribute`
    |
    = note: expected reference `&syn::attr::Attribute`
               found reference `&Attribute`
    = note: perhaps two different versions of crate `syn` are being used?
    = note: this error originates in a derive macro (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0308]: mismatched types
Error:    --> serde_with_macros\src\lib.rs:482:14
    |
482 |     #[derive(FromField, Debug)]
    |              ^^^^^^^^^
    |              |
    |              expected enum `syn::attr::NestedMeta`, found enum `NestedMeta`
    |              this expression has type `syn::attr::NestedMeta`
    |
    = note: perhaps two different versions of crate `syn` are being used?
    = note: this error originates in a derive macro (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0308]: mismatched types
Error:    --> serde_with_macros\src\lib.rs:482:14
    |
482 |     #[derive(FromField, Debug)]
    |              ^^^^^^^^^ expected enum `syn::attr::Meta`, found enum `syn::Meta`
    |
    = note: expected reference `&syn::attr::Meta`
               found reference `&syn::Meta`
    = note: perhaps two different versions of crate `syn` are being used?
    = note: this error originates in a derive macro (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0308]: mismatched types
Error:   --> serde_with_macros\src\utils.rs:26:10
   |
26 | #[derive(FromDeriveInput, Debug)]
   |          ^^^^^^^^^^^^^^^ expected struct `syn::attr::Attribute`, found struct `Attribute`
   |
   = note: expected reference `&syn::attr::Attribute`
              found reference `&Attribute`
   = note: perhaps two different versions of crate `syn` are being used?
   = note: this error originates in a derive macro (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0308]: mismatched types
Error:   --> serde_with_macros\src\utils.rs:26:10
   |
26 | #[derive(FromDeriveInput, Debug)]
   |          ^^^^^^^^^^^^^^^
   |          |
   |          expected enum `syn::attr::NestedMeta`, found enum `NestedMeta`
   |          this expression has type `syn::attr::NestedMeta`
   |
   = note: perhaps two different versions of crate `syn` are being used?
   = note: this error originates in a derive macro (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0308]: mismatched types
Error:   --> serde_with_macros\src\utils.rs:26:10
   |
26 | #[derive(FromDeriveInput, Debug)]
   |          ^^^^^^^^^^^^^^^ expected enum `syn::attr::Meta`, found enum `syn::Meta`
   |
   = note: expected reference `&syn::attr::Meta`
              found reference `&syn::Meta`
   = note: perhaps two different versions of crate `syn` are being used?
   = note: this error originates in a derive macro (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0277]: the trait bound `syn::Path: FromMeta` is not satisfied
Error:   --> serde_with_macros\src\utils.rs:26:10
   |
26 | #[derive(FromDeriveInput, Debug)]
   |          ^^^^^^^^^^^^^^^ the trait `FromMeta` is not implemented for `syn::Path`
   |
   = note: required because of the requirements on the impl of `FromMeta` for `Option<syn::Path>`
   = note: required by `from_meta`
   = note: this error originates in a derive macro (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0308]: mismatched types
Error:   --> serde_with_macros\src\utils.rs:36:60
   |
36 |         match <Self as FromDeriveInput>::from_derive_input(&input) {
   |                                                            ^^^^^^ expected struct `syn::derive::DeriveInput`, found `&DeriveInput`
   |
   = note: expected reference `&syn::derive::DeriveInput`
              found reference `&&DeriveInput`

error: aborting due to 19 previous errors
Error: aborting due to 19 previous errors
Some errors have detailed explanations: E0053, E0277, E0308.
For more information about an error, try `rustc --explain E0053`.
error: could not compile `serde_with_macros`
Error: could not compile `serde_with_macros`
To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
Warning: error: build failed
Error: The process 'C:\Rust\.cargo\bin\cargo.exe' failed with exit code 101

Instance 1: https://github.com/jonasbb/serde_with/runs/1669122665#step:5:139
Instance 2: https://github.com/jonasbb/serde_with/runs/1658483317#step:5:139

The error occurs on both Ubuntu and Windows, but is non-deterministic. I think the first PR where I noticed that is jonasbb/serde_with#238 but I re-run the failed tests and now I cannot find the old logs. Before that point in time I haven't seen the issue before.

ehuss commented

Thanks for the report @jonasbb! I have posted a fix at #9059.