Switch to lld makes difficult cdylib that is both a so and an executable
Closed this issue · 17 comments
Code
I tried this code:
// Cargo.toml
[lib]
crate-type = ["cdylib"]
// build.rs
if os == "linux" {
println!("cargo:rustc-link-arg=-Wl,-pie");
println!("cargo:rustc-link-arg=/usr/lib/x86_64-linux-gnu/Scrt1.o");
println!("cargo:rustc-link-arg=-Wl,--dynamic-linker=/lib64/ld-linux-x86-64.so.2");
println!("cargo:rustc-link-arg=-Wl,--allow-multiple-definition"); // when compiling for tests
} else if os == "macos" {
println!("cargo:rustc-link-arg=-Wl,-execute");
println!("cargo:rustc-link-arg=-Wl,-e,_main");
println!("cargo:rustc-link-arg=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib/crt1.o");
}
// lib.rs
/// 🪄 dynlib become an executable!
/// Entrypoint linux
#[cfg(not(test))]
#[allow(clippy::similar_names, clippy::panic, reason = "standard")]
#[expect(unsafe_code, reason = "ffi")]
#[unsafe(no_mangle)]
pub extern "C" fn main(_argc: i32, _argv: *const *const i8, _env: *const *const i8) -> i32 {
// optional: parse argv if you need them
match exec_async() {
Ok(()) => 0,
Err(e) => panic!("{e}"),
}
}
/// 🪄 dynlib become an executable!
/// Entrypoint macosx
#[cfg(not(test))]
#[allow(clippy::similar_names, reason = "standard")]
#[expect(unsafe_code, reason = "ffi")]
#[unsafe(no_mangle)]
pub extern "C" fn _main(argc: i32, argv: *const *const i8, env: *const *const i8) -> i32 {
main(argc, argv, env)
}I expected to see this happen:
In rust 1.89 with BFD linker this works great, I can do ./target/*/*.so and the .so behave like a normal executabe.
Instead, this happened:
In rust 1.90 with LLD linker this doesn't work anymore,
note: rust-lld: error: -shared and -pie may not be used together
after removing -pie flag it compiles, the .so has a _start symbol but it call to a null value
Version it worked on
It most recently worked on: 1.89
Version with regression
rustc --version --verbose:
rustc 1.90.0 (1159e78c4 2025-09-14)
binary: rustc
commit-hash: 1159e78c4747b02ef996e55082b704c09b970588
commit-date: 2025-09-14
host: x86_64-unknown-linux-gnu
release: 1.90.0
LLVM version: 20.1.8
@rustbot modify labels: +regression-from-stable-to-stable -regression-untriaged
I think you'll need to file an issue against lld if this is something that is reasonably expected to work. For that you should try to reduce this further to a single rustc invocation (no Cargo/build.rs), or better for the LLD issue tracker would be a repro in C that works with bfd but not lld.
You should be able to easily work around this for now by passing -Clinker-features=-lld in your build script.
Indeed, if you use custom linker flags and do some magic that might depend on the linker implementation, you'll either have to find a way to make it work with LLD, or opt out of it and use the system BFD linker.
Don't you also need -Wl,-e,_start on Linux to actually set the ELF entrypoint?
As I understand it, _start is provided in /usr/lib/x86_64-linux-gnu/Scrt1.o.
Maybe /lib64/ld-linux-x86-64.so.2 will jump to _start if it exist?
It wasn't necessary to make this works with BFD, I quickly tried to add the line with LLD but got a segfault still
Maybe /lib64/ld-linux-x86-64.so.2 will jump to _start if it exist?
The dynamic linker will jump to whatever address is specified as entrypoint in the ELF header. For executables this defaults to the address of the _start symbol, but I don't know if lld also sets it to _start when -shared is passed.
Would you mind showing a backtrace and disassembly of the place where it jumps to address 0?
Thanks for looking at this,
Here some more informations, first the working version:
// rust 1.89 with bfd
// println!("cargo:rustc-link-arg=-Wl,-pie");
// println!("cargo:rustc-link-arg=/usr/lib/x86_64-linux-gnu/Scrt1.o");
// println!("cargo:rustc-link-arg=-Wl,--dynamic-linker=/lib64/ld-linux-x86-64.so.2");
// println!("cargo:rustc-link-arg=-Wl,--allow-multiple-definition"); // when compiling for tests
// gdb -batch -ex 'disassemble/rs _start' ./libnatae.1.89.bfd.so
Dump of assembler code for function _start:
0x0000000000041560 <+0>: f3 0f 1e fa endbr64
0x0000000000041564 <+4>: 31 ed xor %ebp,%ebp
0x0000000000041566 <+6>: 49 89 d1 mov %rdx,%r9
0x0000000000041569 <+9>: 5e pop %rsi
0x000000000004156a <+10>: 48 89 e2 mov %rsp,%rdx
0x000000000004156d <+13>: 48 83 e4 f0 and $0xfffffffffffffff0,%rsp
0x0000000000041571 <+17>: 50 push %rax
0x0000000000041572 <+18>: 54 push %rsp
0x0000000000041573 <+19>: 45 31 c0 xor %r8d,%r8d
0x0000000000041576 <+22>: 31 c9 xor %ecx,%ecx
0x0000000000041578 <+24>: 48 8d 3d e1 8f fc ff lea -0x3701f(%rip),%rdi # 0xa560 <natae::init::native::main>
0x000000000004157f <+31>: ff 15 83 02 01 00 call *0x10283(%rip) # 0x51808
0x0000000000041585 <+37>: f4 hlt
$ LD_DEBUG=libs ./libnatae.1.89.bfd.so
1728386: find library=libgcc_s.so.1 [0]; searching
1728386: search cache=/etc/ld.so.cache
1728386: trying file=/usr/local/lib/x86_64-linux-gnu/libgcc_s.so.1
1728386:
1728386: find library=libc.so.6 [0]; searching
1728386: search cache=/etc/ld.so.cache
1728386: trying file=/usr/local/lib/x86_64-linux-gnu/libc.so.6
1728386:
1728386:
1728386: calling init: /lib64/ld-linux-x86-64.so.2
1728386:
1728386:
1728386: calling init: /usr/local/lib/x86_64-linux-gnu/libc.so.6
1728386:
1728386:
1728386: calling init: /usr/local/lib/x86_64-linux-gnu/libgcc_s.so.1
1728386:
1728386:
1728386: initialize program: ./libnatae.1.89.bfd.so
1728386:
1728386:
1728386: transferring control: ./libnatae.1.89.bfd.so
1728386:
1728386:
1728386: calling fini: [0]
1728386:
1728386:
1728386: calling fini: /usr/local/lib/x86_64-linux-gnu/libgcc_s.so.1 [0]
1728386:
1728386:
1728386: calling fini: /usr/local/lib/x86_64-linux-gnu/libc.so.6 [0]
1728386:
1728386:
1728386: calling fini: /lib64/ld-linux-x86-64.so.2 [0]
1728386:
$ readelf -d ./libnatae.1.89.bfd.so
Dynamic section at offset 0x51580 contains 29 entries:
Tag Type Name/Value
0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1]
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
0x0000000000000001 (NEEDED) Shared library: [ld-linux-x86-64.so.2]
0x000000000000000c (INIT) 0x5000
0x000000000000000d (FINI) 0x41588
0x0000000000000019 (INIT_ARRAY) 0x4f348
0x000000000000001b (INIT_ARRAYSZ) 16 (bytes)
0x000000000000001a (FINI_ARRAY) 0x4f358
0x000000000000001c (FINI_ARRAYSZ) 8 (bytes)
0x000000006ffffef5 (GNU_HASH) 0x3d8
0x0000000000000005 (STRTAB) 0x8c8
0x0000000000000006 (SYMTAB) 0x400
0x000000000000000a (STRSZ) 788 (bytes)
0x000000000000000b (SYMENT) 24 (bytes)
0x0000000000000015 (DEBUG) 0x0
0x0000000000000003 (PLTGOT) 0x51790
0x0000000000000002 (PLTRELSZ) 48 (bytes)
0x0000000000000014 (PLTREL) RELA
0x0000000000000017 (JMPREL) 0x4eb0
0x0000000000000007 (RELA) 0xd28
0x0000000000000008 (RELASZ) 16776 (bytes)
0x0000000000000009 (RELAENT) 24 (bytes)
0x000000000000001e (FLAGS) BIND_NOW
0x000000006ffffffb (FLAGS_1) Flags: NOW PIE
0x000000006ffffffe (VERNEED) 0xc48
0x000000006fffffff (VERNEEDNUM) 3
0x000000006ffffff0 (VERSYM) 0xbdc
0x000000006ffffff9 (RELACOUNT) 651
0x0000000000000000 (NULL) 0x0
And now the new version:
// rust 1.90 with lld
// println!("cargo:rustc-link-arg=/usr/lib/x86_64-linux-gnu/Scrt1.o");
// println!("cargo:rustc-link-arg=-Wl,--dynamic-linker=/lib64/ld-linux-x86-64.so.2");
// println!("cargo:rustc-link-arg=-Wl,--allow-multiple-definition"); // when compiling for tests
// gdb -batch -ex 'disassemble/rs _start' ./target/debug/libnatae.1.90.lld.so
Dump of assembler code for function _start:
0x000000000004fdc0 <+0>: f3 0f 1e fa endbr64
0x000000000004fdc4 <+4>: 31 ed xor %ebp,%ebp
0x000000000004fdc6 <+6>: 49 89 d1 mov %rdx,%r9
0x000000000004fdc9 <+9>: 5e pop %rsi
0x000000000004fdca <+10>: 48 89 e2 mov %rsp,%rdx
0x000000000004fdcd <+13>: 48 83 e4 f0 and $0xfffffffffffffff0,%rsp
0x000000000004fdd1 <+17>: 50 push %rax
0x000000000004fdd2 <+18>: 54 push %rsp
0x000000000004fdd3 <+19>: 45 31 c0 xor %r8d,%r8d
0x000000000004fdd6 <+22>: 31 c9 xor %ecx,%ecx
0x000000000004fdd8 <+24>: 48 8b 3d c9 34 00 00 mov 0x34c9(%rip),%rdi # 0x532a8
0x000000000004fddf <+31>: ff 15 db 3d 00 00 call *0x3ddb(%rip) # 0x53bc0 /!\ It crashes here, RIP is null
0x000000000004fde5 <+37>: f4 hlt
─── Output/messages ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
─── Assembly ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Cannot access memory at address 0x0
─── Registers ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
rax 0x0000000000000000 rbx 0x0000000000000000 rcx 0x0000000000000000 rdx 0x00007fffffffd6f8 rsi 0x0000000000000001 rdi 0x0000000000000000 rbp 0x0000000000000000
rsp 0x00007fffffffd6d8 r8 0x0000000000000000 r9 0x0000000000000000 r10 0x0000000000000000 r11 0x0000000000000000 r12 0x0000000000000000 r13 0x0000000000000000
r14 0x0000000000000000 r15 0x0000000000000000 rip 0x0000000000000000 eflags [ PF ZF IF RF ] cs 0x00000033 ss 0x0000002b ds 0x00000000
es 0x00000000 fs 0x00000000 gs 0x00000000 fs_base 0x0000000000000000 gs_base 0x0000000000000000
─── Source ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
─── Stack ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
[0] from 0x0000000000000000
[1] from 0x00007ffff7ff8de5 in _start
─── Threads ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
[1] id 1726179 name libnatae.1.90.lld.so from 0x0000000000000000
─── Variables ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#0 0x0000000000000000 in ?? ()
#1 0x00007ffff7ff8de5 in _start ()
$ readelf -d ./libnatae.1.90.lld.so
Dynamic section at offset 0x51080 contains 28 entries:
Tag Type Name/Value
0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1]
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
0x0000000000000001 (NEEDED) Shared library: [ld-linux-x86-64.so.2]
0x000000000000001e (FLAGS) BIND_NOW
0x000000006ffffffb (FLAGS_1) Flags: NOW
0x0000000000000007 (RELA) 0xc88
0x0000000000000008 (RELASZ) 17640 (bytes)
0x0000000000000009 (RELAENT) 24 (bytes)
0x000000006ffffff9 (RELACOUNT) 683
0x0000000000000017 (JMPREL) 0x5170
0x0000000000000002 (PLTRELSZ) 96 (bytes)
0x0000000000000003 (PLTGOT) 0x53bc8
0x0000000000000014 (PLTREL) RELA
0x0000000000000006 (SYMTAB) 0x2f0
0x000000000000000b (SYMENT) 24 (bytes)
0x0000000000000005 (STRTAB) 0x95c
0x000000000000000a (STRSZ) 806 (bytes)
0x000000006ffffef5 (GNU_HASH) 0x938
0x0000000000000019 (INIT_ARRAY) 0x50e68
0x000000000000001b (INIT_ARRAYSZ) 16 (bytes)
0x000000000000001a (FINI_ARRAY) 0x50e60
0x000000000000001c (FINI_ARRAYSZ) 8 (bytes)
0x000000000000000c (INIT) 0x13f24
0x000000000000000d (FINI) 0x13f40
0x000000006ffffff0 (VERSYM) 0x7e8
0x000000006ffffffe (VERNEED) 0x854
0x000000006fffffff (VERNEEDNUM) 3
0x0000000000000000 (NULL) 0x0
ChatGPT says that I also need to link /usr/lib/x86_64-linux-gnu/crti.o and /usr/lib/x86_64-linux-gnu/crtn.o but it didn't worked.
Here is the two files that produces output above
libs.zip
Edit:
This makes the program run as expected:
$ /lib64/ld-linux-x86-64.so.2 ./target/debug/libnatae.so
[2025-09-21T09:56:26Z INFO natae::init::native] starting async...
....
Edit:
$ readelf -p .interp ./libnatae.1.89.bfd.so
String dump of section '.interp':
[ 0] /lib64/ld-linux-x86-64.so.2
$ readelf -p .interp ./libnatae.1.90.lld.so
readelf: Warning: Section '.interp' was not dumped because it does not exist
Yeah, I noticed too that libnatae.1.90.lld.so doesn't have a dynamic linker set. It is missing a PT_INTERP program header. That would explain why you get a jump to 0. The GOT has never been initialized by the dynamic linker.
I'll try to find how to tell LLD to create the header, as -Wl,--dynamic-linker=/lib64/ld-linux-x86-64.so.2 is not enough
Looking at LLD source code, it is clear that LLD will never emit PT_INTERP header when the -shared flag is set
// lld/ELF/SyntheticSections.cpp:4545
static bool needsInterpSection(Ctx &ctx) {
return !ctx.arg.relocatable && !ctx.arg.shared &&
!ctx.arg.dynamicLinker.empty() && ctx.script->needsInterpSection();
}
// lld/ELF/SyntheticSections.cpp:4702
template <class ELFT> void elf::createSyntheticSections(Ctx &ctx) {
// Add the .interp section first because it is not a SyntheticSection.
// The removeUnusedSyntheticSections() function relies on the
// SyntheticSections coming last.
if (needsInterpSection(ctx)) {
for (size_t i = 1; i <= ctx.partitions.size(); ++i) {
InputSection *sec = createInterpSection(ctx);
sec->partition = i;
ctx.inputSections.push_back(sec);
}
}
// lld/ELF/Writer.cpp:2347
// PT_INTERP must be the second entry if exists.
if (OutputSection *cmd = findSection(ctx, ".interp", partNo))
addHdr(PT_INTERP, cmd->getPhdrFlags())->add(cmd);
I was just typing that about LLD not creating .interp when -shared is present 😄
There might be valid reasons why they don't want to support that edge case. Even though PIE might look like a normal shared object with set interpreter, there is more to it. For example, some relaxations are enabled only for the executables.
Do you even need cdylib though?
Creating cdylib with -pie should be mostly the same as creating a regular binary crate.
In rust 1.89 with BFD linker this works great, I can do ./target//.so and the .so behave like a normal executabe.
Because these .so files are just PIEs, named like libraries.
Do you even need cdylib though?
I want my crate to compile as both WASM file and an executable, as far as I understand cargo will emit a wasm file only if the crate is a cdylib. Also I'd prefer not to have a cargo workspace or multiple output targets.
.so files are just PIEs, named like libraries
Exactly! Since rust will create the .so file when not compiling for wasm arch, I'm happy to use it as an normal executable :)
Lets see if llvm team is willing to support this.
Thanks,
Hugues
Ah, I have no experience with WASM, but that sounds like a good justification.
@mati865 I tested Wild to see if it works with my setup:
RUSTFLAGS="-Clinker=clang -Clink-args=--ld-path=wild" cargo build
readelf -l target/debug/libnatae.so
Elf file type is DYN (Shared object file)
Entry point 0x0
There are 11 program headers, starting at offset 64
Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
PHDR 0x0000000000000040 0x0000000000000040 0x0000000000000040
0x0000000000000268 0x0000000000000268 R 0x8
INTERP 0x0000000000000ce8 0x0000000000000ce8 0x0000000000000ce8
0x000000000000001c 0x000000000000001c R 0x1
[Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
LOAD 0x0000000000000000 0x0000000000000000 0x0000000000000000
0x000000000025fa96 0x000000000025fa96 R 0x1000
LOAD 0x000000000025faa0 0x0000000000260aa0 0x0000000000260aa0
0x0000000000739bf5 0x0000000000739bf5 R E 0x1000
LOAD 0x0000000000999698 0x000000000099b698 0x000000000099b698
0x0000000000068338 0x000000000006844f RW 0x1000
DYNAMIC 0x00000000009eb1a0 0x00000000009ed1a0 0x00000000009ed1a0
0x0000000000000210 0x0000000000000210 RW 0x8
NOTE 0x0000000000000d08 0x0000000000000d08 0x0000000000000d08
0x0000000000000070 0x0000000000000070 R 0x8
TLS 0x0000000000999698 0x000000000099b698 0x000000000099b698
0x00000000000001a0 0x00000000000001a0 R 0x8
GNU_EH_FRAME 0x00000000001339a0 0x00000000001339a0 0x00000000001339a0
0x000000000003941c 0x000000000003941c R 0x4
GNU_STACK 0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 RW 0x10
GNU_RELRO 0x0000000000999698 0x000000000099b698 0x000000000099b698
0x0000000000065a78 0x0000000000065a78 R 0x10
Section to Segment mapping:
Segment Sections...
00
01 .interp
02 .interp .note.gnu.property .note.gnu.build-id .note.ABI-tag .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rela.dyn .rodata .eh_frame_hdr .eh_frame .gcc_except_table .debug_gdb_scripts
03 .plt.got .text .init .fini
04 .tdata .init_array .fini_array .data.rel.ro .dynamic .got .data .tm_clone_table .bss
05 .dynamic
06 .note.gnu.property .note.gnu.build-id .note.ABI-tag
07 .tdata .tbss
08 .eh_frame_hdr
09
10 .tdata .init_array .fini_array .data.rel.ro .dynamic .got
program interpreter is correctly set, but the entrypoint is 0x0 which is wrong
Oh, I didn't see that coming. It should be easy to fix.
@izissise it doesn't reproduce for me on Arch Linux:
❯ readelf -Wh target/debug/libcdylib.so | rg Entry
Entry point address: 0x3ca10
❯ readelf -Wl target/debug/libcdylib.so | rg Entry
Entry point 0x3ca10
❯ readelf -p .comment target/debug/libcdylib.so
String dump of section '.comment':
[ 1] GCC: (GNU) 15.2.1 20250813
[ 1c] rustc version 1.90.0 (1159e78c4 2025-09-14)
[ 48] Linker: Wild version 0.6.0
Is it worth to keep this issue open considering LLD is not willing to change? I guess there is nothing Rust project can do about it.
Since I run it with a wrapper script anyway, I settled to exec the SO with /lib64/ld-linux-x86-64.so.2.
I think there should be a better story for project that are cross platform between wasm and native.
Thanks everyone for the help :)