Unable to find crate `proc_macro` on musl target
golddranks opened this issue · 26 comments
Using procedural derive macros on musl seems to work just fine for the most of the cases, but there is one pitfall; in some cases like this dtolnay/proc-macro-hack#6 build fails because not only the crate that implements the macro (crate marked with proc-macro = true), but also another, normal crate links to the proc_macro crate. "Normal" crates are unable to find proc_macro when built on musl target:
Compiling proc-macro-hack v0.3.0
error[E0463]: can't find crate for `proc_macro`
--> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro-hack-0.3.0/src/lib.rs:1:1
|
1 | extern crate proc_macro;
| ^^^^^^^^^^^^^^^^^^^^^^^^ can't find crate
error: aborting due to previous error
error: Could not compile `proc-macro-hack`.
Yes this is currently intentional. The proc_macro crate is only a dynamic library which the musl target currently doesn't support. I unfortunately don't think this will be easy to fix...
I understand why the proc macro crates need to be dynamic: they are loaded and run dynamically by the compiler. However what I don't understand is why the proc_macro crate isn't allowed to be linked both dynamically and statically? I'd imagine someone may want to use the types defined in proc_macro in some helper crate, outside the dynamically linked macro crate, as is the case here.
Is there some technical hurdle that makes it hard to do, or is it just not done because it's such a minor case?
The simplest test for this seems to be:
test.rs with just single line:
extern crate proc_macro;
build with (succeeds):
rustc test.rs --crate-type lib
and with (fails):
rustc test.rs --crate-type lib --target=x86_64-unknown-linux-musl
When RUST_LOG=info is enabled, the library probes can be seen.
Okay, I did some testing.
I git cloned the Rust source, edited away the crate-type = ["dylib"] annotation in libproc_macro, librustc_data_structures, librustc_errors, libsyntax and libsyntax_pos. (Transitive deps of libproc_macro)
After that, I built a rlib of libproc_macro with cargo build --target=x86_64-unknown-linux-musl. (This is on the newest nightly.)
After that, I copied the build artefacts from ~/rust/src/target/x86_64-unknown-linux-musl/debug/deps/ to ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib.
After that, I try to compile the things that failed before with musl target → it works.
So, it seems that there isn't any real technical hurdles. (or if there is, I haven't encountered them yet)
@alexcrichton, what do you think of distributing rlibs of aforementioned libraries with the musl target?
Btw. I did this work on a Linux host. Tried to do the same on macOS (with musl target of course), but on macOS the linking fails with ld: unknown option: --as-needed. The linker is passed a flag --as-needed which seems to be supported on GNU linker only.
Ah, okay, the --as-needed behaviour is explained here: #34282
Yes there are technical hurdles in terms of build system of doing that. This is also rarely what you want because there's no reason to have a full Rust parser linked into a binary (when it's not used).
I see, that's a convincing argument.
Hello,
sorry to dig up a closed issue, but I'm experiencing the same problem during an OpenWrt cross build (i686-unknown-linux-musl is the Rust target and VirtualBox is the OpenWrt one). My compiler is rustc 1.25.0-nightly (61452e506 2018-01-09). I've got a couple questions regarding this:
- Is the root cause known? What is it?
- Is there a solution out there?
- If not, where can I go to help solve this?
Hello! I've got this issue, as well -- deps relying on proc-macro2 which do not use default-features = false so proc-macro crate is not found because my target does not have it.
I'm currently compiling rustc for my target and it seems easiest to create proc-macro lib when I'm building. Is there any documentation on how to do this?
For those following various links to this issue, it is important to note that you can compile code using proc_macro to statically-linked musl binaries, BUT you need to cross-compile from a platform that does support dynamic linking.
cargo build --target x86_64-unknown-linux-musl should be sufficient, but you'll need to run that on something like Ubuntu, Fedora, or another full system.
Reopening as this seems to still be a significant issue for people building from musl systems
--target x86_64-unknown-linux-musl doesn't work for me and shouldn't be considered a sufficient workaround.
Many important crates like OpenSSL need to be vendored and cross-compiled to musl (e.g., sfackler/rust-openssl#980 (comment)), but when I tried to do that, complications like feature flags not working with workspaces (e.g, rust-lang/cargo#5015) started cropping up, and I threw up my hands and gave up on musl.
Even if there aren't plans to work on it in the near term, I don't think this issue should be considered fixed until either (1) It is easy to cross-compile apps to musl or (2) it is possible to cis-compile proc_macro targets on musl.
Would it make sense to have --crate-type proc-macro imply -C target-feature=-crt-static (maybe only on musl)? AFAIK, the precise output of --crate-type proc-macro is essentially an implementation detail, since they should only be used as compiler plugins by rustc.
There are likely much better solutions, but… I've just come across https://github.com/dtolnay/watt ; and… would integrating this into rustc as a secondary backend for use only on platforms where regular proc macros aren't supported be a solution? It wouldn't support all the use cases I think, due to regular proc macros being allowed to eg. interact with the FS, but it should already support a number of use cases. And maybe, with time, the API could be extended so that proc macros could all be compiled with wasm, and the current dylib approach could be dropped, thus making porting rustc to a new architecture a lot easier?
@mati865 I'm not sure, I've pinged here following a message from IRC that said this happened just today -- don't know for sure the version of the compiler used, so I've just pinged them
@mati865 If it's part of the brand new Rust 1.43, then yes, it is still an issue.
On Alpine, the heavily patched rust package (1.39) can compile without any trouble. However, on the very same machine, any version of Rust (including 1.39 and 1.43) installed via rustup (and therefore without those patches) is subject to this issue.
@breard-r Rust 1.43 don't include this patch and you can try 1.44 beta from rustup.
Great! Closing as fixed then.
This is still broken for me using rustc 1.44.0-beta.3 (0f0d70055 2020-05-11).
= note: /usr/lib/gcc/x86_64-alpine-linux-musl/9.2.0/../../../../x86_64-alpine-linux-musl/bin/ld: cannot find crti.o: No such file or directory
Full linker arguments
/rcc # cargo build
Compiling thiserror-impl v1.0.19
error: linking with `cc` failed: exit code: 1
|
= note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-Wl,--eh-frame-hdr" "-m64" "-L" "/usr/local/rustup/toolchains/beta-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.0.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.1.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.10.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.11.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.12.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.13.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.14.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.15.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.2.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.3.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.4.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.5.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.6.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.7.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.8.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.9.rcgu.o" "-o" "/rcc/target/debug/deps/libthiserror_impl-c05c536330366b8e.so" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.1x6px3h632f37ynj.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.2thl14zymh2c935g.rcgu.o" "-Wl,--gc-sections" "-Wl,-zrelro" "-Wl,-znow" "-nodefaultlibs" "-L" "/rcc/target/debug/deps" "-L" "/usr/local/rustup/toolchains/beta-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib" "-Wl,-Bstatic" "/rcc/target/debug/deps/libsyn-fe2290862b758c2c.rlib" "/rcc/target/debug/deps/libquote-afe91c35027afeba.rlib" "/rcc/target/debug/deps/libproc_macro2-0913bab1357133a2.rlib" "/rcc/target/debug/deps/libunicode_xid-8b9fbfbf3102211f.rlib" "/usr/local/rustup/toolchains/beta-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libproc_macro-84e8be3f2684e30a.rlib" "-Wl,--start-group" "/usr/local/rustup/toolchains/beta-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-5ee1781929f9655d.rlib" "/usr/local/rustup/toolchains/beta-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-e2ec5d2aca499f1c.rlib" "/usr/local/rustup/toolchains/beta-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libhashbrown-81740b8a57dba39b.rlib" "/usr/local/rustup/toolchains/beta-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_alloc-eb99f2d2958ad0d9.rlib" "/usr/local/rustup/toolchains/beta-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libbacktrace-84349410b417ab27.rlib" "/usr/local/rustup/toolchains/beta-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libbacktrace_sys-b3c09d3262227e0b.rlib" "/usr/local/rustup/toolchains/beta-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_demangle-c58a107a6842bb60.rlib" "/tmp/rustc3EmKiY/libunwind-33abd88ac2c901d5.rlib" "/usr/local/rustup/toolchains/beta-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcfg_if-c210a18d812061d6.rlib" "/tmp/rustc3EmKiY/liblibc-2b136be5748c5959.rlib" "/usr/local/rustup/toolchains/beta-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/liballoc-150a1e69c064728f.rlib" "/usr/local/rustup/toolchains/beta-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_core-e390c28232836237.rlib" "/usr/local/rustup/toolchains/beta-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcore-1cdf993f577c22fa.rlib" "-Wl,--end-group" "/usr/local/rustup/toolchains/beta-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcompiler_builtins-9211daa04cc43fa3.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lc" "-shared"
= note: /usr/lib/gcc/x86_64-alpine-linux-musl/9.2.0/../../../../x86_64-alpine-linux-musl/bin/ld: cannot find crti.o: No such file or directory
collect2: error: ld returned 1 exit status
error: aborting due to previous error
error: could not compile `thiserror-impl`.
This is still broken for me using
rustc 1.44.0-beta.3 (0f0d70055 2020-05-11).= note: /usr/lib/gcc/x86_64-alpine-linux-musl/9.2.0/../../../../x86_64-alpine-linux-musl/bin/ld: cannot find crti.o: No such file or directoryFull linker arguments
/rcc # cargo build Compiling thiserror-impl v1.0.19 error: linking with `cc` failed: exit code: 1 | = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-Wl,--eh-frame-hdr" "-m64" "-L" "/usr/local/rustup/toolchains/beta-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.0.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.1.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.10.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.11.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.12.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.13.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.14.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.15.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.2.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.3.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.4.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.5.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.6.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.7.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.8.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.thiserror_impl.7sf2w6hc-cgu.9.rcgu.o" "-o" "/rcc/target/debug/deps/libthiserror_impl-c05c536330366b8e.so" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.1x6px3h632f37ynj.rcgu.o" "/rcc/target/debug/deps/thiserror_impl-c05c536330366b8e.2thl14zymh2c935g.rcgu.o" "-Wl,--gc-sections" "-Wl,-zrelro" "-Wl,-znow" "-nodefaultlibs" "-L" "/rcc/target/debug/deps" "-L" "/usr/local/rustup/toolchains/beta-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib" "-Wl,-Bstatic" "/rcc/target/debug/deps/libsyn-fe2290862b758c2c.rlib" "/rcc/target/debug/deps/libquote-afe91c35027afeba.rlib" "/rcc/target/debug/deps/libproc_macro2-0913bab1357133a2.rlib" "/rcc/target/debug/deps/libunicode_xid-8b9fbfbf3102211f.rlib" "/usr/local/rustup/toolchains/beta-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libproc_macro-84e8be3f2684e30a.rlib" "-Wl,--start-group" "/usr/local/rustup/toolchains/beta-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-5ee1781929f9655d.rlib" "/usr/local/rustup/toolchains/beta-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-e2ec5d2aca499f1c.rlib" "/usr/local/rustup/toolchains/beta-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libhashbrown-81740b8a57dba39b.rlib" "/usr/local/rustup/toolchains/beta-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_alloc-eb99f2d2958ad0d9.rlib" "/usr/local/rustup/toolchains/beta-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libbacktrace-84349410b417ab27.rlib" "/usr/local/rustup/toolchains/beta-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libbacktrace_sys-b3c09d3262227e0b.rlib" "/usr/local/rustup/toolchains/beta-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_demangle-c58a107a6842bb60.rlib" "/tmp/rustc3EmKiY/libunwind-33abd88ac2c901d5.rlib" "/usr/local/rustup/toolchains/beta-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcfg_if-c210a18d812061d6.rlib" "/tmp/rustc3EmKiY/liblibc-2b136be5748c5959.rlib" "/usr/local/rustup/toolchains/beta-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/liballoc-150a1e69c064728f.rlib" "/usr/local/rustup/toolchains/beta-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_core-e390c28232836237.rlib" "/usr/local/rustup/toolchains/beta-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcore-1cdf993f577c22fa.rlib" "-Wl,--end-group" "/usr/local/rustup/toolchains/beta-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcompiler_builtins-9211daa04cc43fa3.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lc" "-shared" = note: /usr/lib/gcc/x86_64-alpine-linux-musl/9.2.0/../../../../x86_64-alpine-linux-musl/bin/ld: cannot find crti.o: No such file or directory collect2: error: ld returned 1 exit status error: aborting due to previous error error: could not compile `thiserror-impl`.
It seems you don't install musl-dev
That fixed it, thank you!
If you're bumping into this in 2021, it's likely that you need to run apk add --no-cache musl-dev.