Linker error for no_std binary using musl

Question

Linker error for no_std binary using musl

jgriebler opened this issue 6 years ago · 13 comments

I tried to build a simple no_std "Hello World!" program using the x86_64-unknown-linux-musl target. The source code (in src/main.rs) is here:

#![no_std]
#![no_main]

use libc::{c_char, c_int, exit, puts};

#[no_mangle]
pub extern "C" fn main(_: c_int, _: *const *const c_char) -> c_int {
    unsafe {
        puts(b"Hello World!\0" as *const u8 as *const i8);
    }

    0
}

#[panic_handler]
fn panic_handler(_: &core::panic::PanicInfo) -> ! {
    unsafe {
        exit(1)
    }
}

When using my default x86_64-unknown-linux-gnu target, the program compiles and runs like it should, but with x86_64-unknown-linux-musl, I get this linking error:

error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-Wl,--eh-frame-hdr" "-m64" "-nostdlib" "/home/johannes/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/crt1.o" "/home/johannes/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/crti.o" "-L" "/home/johannes/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib" "/home/johannes/hello/target/x86_64-unknown-linux-musl/release/deps/hello-8e567d3b97d6bd11.hello.c72za6mx-cgu.0.rcgu.o" "-o" "/home/johannes/hello/target/x86_64-unknown-linux-musl/release/deps/hello-8e567d3b97d6bd11" "-Wl,--gc-sections" "-no-pie" "-Wl,-zrelro" "-Wl,-znow" "-Wl,-O1" "-nodefaultlibs" "-L" "/home/johannes/hello/target/x86_64-unknown-linux-musl/release/deps" "-L" "/home/johannes/hello/target/release/deps" "-L" "/home/johannes/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib" "-Wl,-Bstatic" "/home/johannes/hello/target/x86_64-unknown-linux-musl/release/deps/liblibc-8c2b3cf08263e000.rlib" "/home/johannes/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libcore-879310dc3b96af61.rlib" "/home/johannes/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libcompiler_builtins-d0572f7a936161bf.rlib" "-static" "-Wl,-Bdynamic" "/home/johannes/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/crtn.o"
  = note: /usr/bin/ld: /home/johannes/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/crt1.o: in function `_start_c':
          /build/musl-1.1.20/crt/crt1.c:17: undefined reference to `__libc_start_main'
          /usr/bin/ld: /home/johannes/hello/target/x86_64-unknown-linux-musl/release/deps/hello-8e567d3b97d6bd11.hello.c72za6mx-cgu.0.rcgu.o: in function `main':
          hello.c72za6mx-cgu.0:(.text.main+0xa): undefined reference to `puts'
          collect2: error: ld returned 1 exit status

I can build a normal (std-using) "Hello World!" program for musl without problems, only this no_std version doesn't work.

I am also able to work around this problem by instead moving the source code to src/lib.rs and compiling it as a library with crate-type = ["staticlib"], then linking manually with musl-gcc. The error only occurs when trying to build a binary directly.

My Cargo.toml. Uncomment the two lines and move main.rs to lib.rs for the workaround.

[package]
name = "hello"
version = "0.1.0"
authors = ["me"]
edition = "2018"

#[lib]
#crate-type = ["staticlib"]

[dependencies]
libc = { version = "0.2", default-features = false }

[profile.dev]
panic = "abort"

[profile.release]
panic = "abort"

Answer 1 · 2019-02-05T12:10:58.000Z

Which Rust version are you using rustc -vV?

On latest nightly your example fails with:

error[E0152]: duplicate lang item found: `panic_impl`.
  --> src/main.rs:16:1
   |
16 | / fn panic_handler(_: &core::panic::PanicInfo) -> ! {
17 | |     unsafe {
18 | |         exit(1)
19 | |     }
20 | | }
   | |_^
   |
   = note: first defined in crate `std`.

Without panic_handler(_) it builds for both gnu and musl.

Answer 2 · 2019-02-05T12:41:13.000Z

Ah, I forgot to mention that libc needs default-features = false so that it doesn't link std implicitly. I'll attach my Cargo.toml to the issue description.

I tried it with both 1.32.0 and the latest nightly.

Answer 3 · 2019-02-05T14:40:23.000Z

When using native no_std gnu target required symbols come from "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-lutil" linker arguments.
When cross compiling to musl Rust provides those symbols built in liblibc to make it easier but with no_std liblibc is not included anymore resulting in undefined symbols error.

Manually adding ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/liblibc-ad64cf40491cbcc1.rlib to the linker made it compile.

So the question to Rust (libs?) team is whether liblibc should be given to the linker for no_std builds or something else would be preferred.

Gist with linker args for different builds: https://gist.github.com/mati865/210dbf7f27b15ffa96bf9e0b79f5d0b5

Answer 4 · 2019-08-07T11:43:50.000Z

the program compiles and runs like it should, but with x86_64-unknown-linux-musl, I get this linking error:

This is expected behavior. This program is not linking musl at all, to do that, one currently needs to enable the private rustc-dep-of-std libc cargo feature which requires other hacks like linking the rustc-hack crates from https://github.com/rust-lang/rust/tree/master/src/tools (not from crates.io) . It is probably easier to just manually patch libc to always link musl (EDIT: see below for a possibly easier workaround).

If you want to statically link musl to your binary, the easiest way is to not make it a #![no_std] binary, such that libstd links it properly for you.

Sadly, solving this issue isn't trivial. I suppose we could expose a cargo feature from libc that allows users to force the linking of the C library such that you could use that here, but there are other things to consider here, like how this interacts with -C target-feature=-crt-static.

Answer 5 · 2019-08-07T11:53:20.000Z

An alternative workaround that might be easier is to try adding this to your binary:

#[link(name = "c", kind = "static", cfg(target_feature = "crt-static"))]
#[link(name = "c", cfg(not(target_feature = "crt-static")))]
extern {}

Answer 6 · 2019-08-09T10:39:11.000Z

I'm not sure if this is the right place to ask this, but is that available on stable? For me this only works when using #![feature(link_cfg)], even though the tracking issue linked in the error has been closed for a long time.

But even with that, the workaround isn't working for me. On the gnu target it works fine (as before), but on musl I now get this error:

error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-Wl,--eh-frame-hdr" "-m64" "-nostdlib" "/home/johannes/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/crt1.o" "/home/johannes/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/crti.o" "-L" "/home/johannes/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib" "/home/johannes/hello/target/x86_64-unknown-linux-musl/release/deps/hello-88b5f1c27ffd5796.hello.81h291ix-cgu.0.rcgu.o" "-o" "/home/johannes/hello/target/x86_64-unknown-linux-musl/release/deps/hello-88b5f1c27ffd5796" "-Wl,--gc-sections" "-no-pie" "-Wl,-zrelro" "-Wl,-znow" "-Wl,-O1" "-nodefaultlibs" "-L" "/home/johannes/hello/target/x86_64-unknown-linux-musl/release/deps" "-L" "/home/johannes/hello/target/release/deps" "-L" "/home/johannes/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib" "-Wl,-Bstatic" "-Wl,--whole-archive" "-lc" "-Wl,--no-whole-archive" "/home/johannes/hello/target/x86_64-unknown-linux-musl/release/deps/liblibc-9b25b07157410626.rlib" "/home/johannes/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_core-58a61c6fb52da4c6.rlib" "/home/johannes/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libcore-8f8f2fe3f7b7398f.rlib" "/home/johannes/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libcompiler_builtins-6db28108a6c10236.rlib" "-static" "-Wl,-Bdynamic" "/home/johannes/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/crtn.o"
  = note: /usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../lib/libc.a(rcmd.o): in function `__validuser2_sa':
          (.text+0x5a9): warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
          /usr/bin/ld: /home/johannes/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/crt1.o: in function `_start':
          crt1.c:(.text+0x9): undefined reference to `_DYNAMIC'
          /usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../lib/libc.a(printf_fp.o): in function `__printf_fp_l':
          (.text+0x52a): undefined reference to `__unordtf2'
          /usr/bin/ld: (.text+0x562): undefined reference to `__unordtf2'
          /usr/bin/ld: (.text+0x588): undefined reference to `__letf2'
          /usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../lib/libc.a(printf_fphex.o): in function `__printf_fphex':
          (.text+0xac): undefined reference to `__unordtf2'
          /usr/bin/ld: (.text+0xe2): undefined reference to `__unordtf2'
          /usr/bin/ld: (.text+0xfa): undefined reference to `__letf2'
          /usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../lib/libc.a(iofclose.o): in function `_IO_new_fclose.cold':
          (.text.unlikely+0x4c): undefined reference to `_Unwind_Resume'
          /usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../lib/libc.a(iofclose.o):(.data.rel.local.DW.ref.__gcc_personality_v0[DW.ref.__gcc_personality_v0]+0x0): undefined reference to `__gcc_personality_v0'
          /usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../lib/libc.a(iofflush.o): in function `_IO_fflush.cold':
          (.text.unlikely+0x4b): undefined reference to `_Unwind_Resume'
          /usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../lib/libc.a(iofputs.o): in function `_IO_fputs.cold':
          (.text.unlikely+0x4b): undefined reference to `_Unwind_Resume'
          /usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../lib/libc.a(iofwrite.o): in function `_IO_fwrite.cold':
          (.text.unlikely+0x4b): undefined reference to `_Unwind_Resume'
          /usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../lib/libc.a(iogetdelim.o): in function `_IO_getdelim.cold':
          (.text.unlikely+0x4b): undefined reference to `_Unwind_Resume'
          /usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../lib/libc.a(ioputs.o): in function `_IO_puts.cold':
          (.text.unlikely+0x4c): undefined reference to `_Unwind_Resume'
          /usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../lib/libc.a(wfileops.o):(.text.unlikely+0x4c): more undefined references to `_Unwind_Resume' follow
          /usr/bin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../lib/libc.a(dl-reloc-static-pie.o): in function `_dl_relocate_static_pie':
          (.text+0x1a): undefined reference to `_DYNAMIC'
          /usr/bin/ld: (.text+0x39): undefined reference to `_DYNAMIC'
          /usr/bin/ld: /home/johannes/hello/target/x86_64-unknown-linux-musl/release/deps/hello-88b5f1c27ffd5796: hidden symbol `_DYNAMIC' isn't defined
          /usr/bin/ld: final link failed: bad value
          collect2: error: ld returned 1 exit status

I don't know enough about how linking is done to try to get this to work, but maybe someone else knows what to do with this error.

(Also, just to clarify: I don't actually need this for anything, I just stumbled across this behaviour when toying around with #[panic_handler] when it was stabilised.)

Answer 7 · 2019-08-09T11:43:39.000Z

I'm not sure if this is the right place to ask this, but is that available on stable?

Nope, but neither are #![no_std] binaries IIRC, right ? (that is, you'd need nightly anyways to be able to reach this issue)

On the gnu target it works fine (as before), but on musl I now get this error:

I see x86_64-pc-linux-gnu in those error messages. Is that normal? How are you compiling the #![no_std] binary ? E.g. if you want to compile with cargo build --target=x86_64-unknown-linux-musl I think you need to make sure that the proper C toolchain is used.

(Also, just to clarify: I don't actually need this for anything, I just stumbled across this behaviour when toying around with #[panic_handler] when it was stabilised.)

Makes sense - this is kind of a known issue, but is good to have a proper bug report for it, so thank you for trying so many things here!

Answer 8 · 2019-08-09T13:07:58.000Z

Nope, but neither are #![no_std] binaries IIRC, right ? (that is, you'd need nightly anyways to be able to reach this issue)

It's available on stable since 1.30, when #[panic_handler] was stabilised, though afaik only with panic=abort.

I see x86_64-pc-linux-gnu in those error messages. Is that normal? How are you compiling the #![no_std] binary ? E.g. if you want to compile with cargo build --target=x86_64-unknown-linux-musl I think you need to make sure that the proper C toolchain is used.

Ah, that's interesting. I tried it again with -C linker=musl-gcc and now your workaround works (though my original example still doesn't). Does this mean that #[link(...)] will always use the system cc even on the musl target?

Answer 9 · 2019-08-09T13:31:50.000Z

The problem is that we can't statically link musl to a binary twice, and libstd for musl always links musl statically.

The only thing that the std feature of libc controls is whether libc itself has libstd as a dependency. If it doesn't, that does not mean that libstd won't be linked to the final binary (e.g. if the final binary isn't #![no_std], or if it is #![no_std] but contains an extern crate std, then libstd will be linked). There is no way for libc to know whether libstd will be linked, so it has to assume it will, and never link musl, since if libstd is actually linked, and libc links musl, musl will be statically linked twice, and the build will fail.

So when you have a #![no_std] binary that links libc, and you disable the libc std feature, that does not pull in musl, because you could write extern crate std; and that would break that use case. If you enable the std feature, libc should pull libstd for you, so that you get a valid musl, but this is probably not what you want if you are building a #![no_std] binary. So the only thing you can do is link musl yourself, which is what that the workaround here does.

This is bad, super hard to discover, etc and there should be a better way to do this.

The simplest thing to do would be adding a new cargo feature to the libc crate, that forces linking the C library no matter what. That way, you could add libc with a force_link_c_libs feature as a dependency, and your example would "just work". If you then, by accident, pull in libstd, e.g., by doing extern crate std;, or adding a dependency that does that (e.g. by not using #![no_std]), then your binary would fail to link, and the error message would be an obscure linking error, and there is no easy fix for that.

So I think we probably want a more elaborated way of solving all these issues, that has nice error messages, and always works in the obvious way.

I think, that if you add:

#![feature(rustc_private)]
extern crate libc;

to your example, and remove the libc dependency from your Cargo.toml, your code will also work, because it will use the libc version that the standard library uses, but without pulling in libstd. That libc version is special and different from the one in crates.io, and it will statically link musl for you.

Exploring a solution in this direction might be worth it. That is, you want a #![no_std] binary with a libc that does link the C standard library. It shouldn't be necessary for you to have to pull in libc from crates.io to do that, although it should be possible.

Answer 10 · 2019-08-09T13:38:44.000Z

@mati865 maybe we could move the parts of the libc crate that actually link stuff into a separate crate, e.g., libc_linkage, that always links the stuff. We can then make that crate "unique", by using the Cargo links = "..." key, preventing it from being linked twice into a dependency graph.

We could then make libc via libstd always link it, and just allow configuring for libc from crates.io whether it should link it or not. If a user messes this up, cargo will error saying that there are two libc_link crates in the dependency graph., which is nice. A problem is that cargo will error even if libc_link dynamically links stuff, but I think this might be fine ?

This allows you to pull in different versions of the bindings, by pulling different versions of the libc crate, into a dependency graph, but ensuring that stuff is linked only once.

Answer 11 · 2019-08-09T13:44:46.000Z

Thanks for the explanation, that does make some sense. I've tried your suggestion with rustc_private, but I couldn't get it to work right now. Maybe I still did something wrong.

I've found an even simpler alternative to your previous workaround though: Just passing -lc to rustc (along with the right linker) seems to be enough. That one even works on stable, which probably makes this whole thing mostly a non-issue. I guess it's still warranted to leave this issue open, since having to do that seems rather unintuitive.

Answer 12 · 2019-08-09T14:03:18.000Z

@gnzlbg I never played with links field but from what I've read that could work.

To make sure I understand libc_linkage would be native library that links to libc, libm, libpthread and whatever else either as static or dynamic libs. Then libc crate would use links = "libc_link".
There are targets that do not support creating dylibs, could that be issue?

Answer 13 · 2019-08-09T14:35:28.000Z

Thanks for the explanation, that does make some sense. I've tried your suggestion with rustc_private, but I couldn't get it to work right now. Maybe I still did something wrong.

No, i think this just doesn't work. The libc used by rustc is not the same as the one used by the std library :(

To make sure I understand libc_linkage would be native library that links to libc, libm, libpthread and whatever else either as static or dynamic libs.

I think it can be an rlib, it just needs to contain the #[link ...] extern { ... } blocks to force rustc to pass the appropriate -lc etc. flags.

Then libc crate would use links = "libc_link".

This isn't what I meant. The libc_link crate would have a links = "...unique..." key. libc would have libc_link as a dependency.

I haven't worked out the rest. For example, if libc_link is optional, we could build libstd with it enabled, and if a user tries to enable it while compiling libc from crates.io that will only work if libstd is not linked. However, if a user does not include libstd, and forgets to enable it, the error message will be obscure. This is because this dependency isn't really optional for libc.

So maybe we could publish libc_link along with libcore, liballoc, etc. and have it as a required dependency of libc. Crates can use whatever libc version they want, but they all link to the exact same libc_links. libstd would link against it as well.

That means that just including libc as a dependency would work for #![no_std] binaries, even if libstd is not available. The "con" of this approach, is that changes to libc_links have to go through the release process. Maybe we could offer a way to override the libc_link crate used, e.g., via a -C ... argument or something. I've wanted to be able to select a different implementation of libm in the past, so this might be a step towards that direction.