LLD linker does not have required nixpkgs wrapper script and flags
nh2 opened this issue ยท 27 comments
Currently using the lld
linker in nix doesn't work correctly.
Executables that it produces have entries in lld
being not found
(rpaths are missing).
That's because ld
and gold
have wrappers that do the following:
# First, find all -L... switches.
# Second, for each directory in the library search path (-L...),
# see if it contains a dynamic library used by a -l... flag. If
# so, add the directory to the rpath.
But lld
currently doesn't have such a wrapper.
(I found this via See also #24692 (comment).)
Also this type of patch or setting to --enable-new-dtags
will likely be needed, and potentially other changes nixpkgs makes to ld
and gold
.
This would be a nice addition to speed up Haskell builds by reducing time in linking phase.
As part of my work this summer on speeding up Nix Haskell builds for Awake Networks, I'm interested in tackling this issue (and upstreaming a fix). Does it just amount to adding something similar to the code snippet below to pkgs/build-support/cc-wrapper/default.nix
(around line 180)?
if [ -e ${binutils_bin}/bin/ld.gold ]; then
wrap ld.gold ${./ld-wrapper.sh} ${binutils_bin}/bin/ld.gold
fi
or should I be modifying the lld
package itself to use the ld-wrapper.sh
script?
Also, if someone could cook up an example Nix expression that should build iff I have correctly wrapped lld
, that would be quite useful in implementing a fix for this.
@taktoa I can't answer conclusively, but I suspect it won't be as trivial as using the exiting scripts.
The right approach is likely to first do a proper reads through all things nixpkgs does to ld
or gold
, and then check which of those have to be done to lld
as well.
For example, I know that lld
searches through some default paths. We probably want to patch them out, or pass flags so that it doesn't do that.
@taktoa we're interested in this project at https://github.com/input-output-hk as well - maybe we could have quick chat and join some forces :)
we invested a bit of time into parallel GC and general GHC option tuning for scaling the compilation into multiple cores: https://github.com/input-output-hk/iohk-nixops/pull/49/files
Indeed linking takes a hell lot of time, but buildTarget
fix in Cabal master will help a lot, limiting what executables are built for the given package.
@taktoa for this particular change, I'd talk to @vcunat or @wkennington
Assuming the relevant linker options have the same syntax, I'd expect that the same wrapping approach should work. The way cc-wrapper is written now, I'd suggest to first try adding an option that makes it include ld.lld
. (And maybe an option to make it the default ld
, though I don't know if that's the standard way to exchange linkers.)
I have now added the lldClang
stdenv. We could expose this better as needed if people want lld to be installable. Otherwise, without anything actionable I think this can be closed?
for instance, you can run llvm linker:
with import <nixpkgs> { system = "x86_64-linux"; };
let stdenv' = overrideCC stdenv llvmPackages_8.lldClang;
in stdenv'.mkDerivation {
name = "test";
buildCommand = ''
cc ${./test.c} -o test
./test
'';
}
Also install lld using:
nix-env -iA nixpkgs.llvmPackages_8.lldClang.bintools
Otherwise, without anything actionable I think this can be closed?
@matthewbauer The actionable part from the issue description was:
Executables that it produces have entries in
lld
beingnot found
(rpaths are missing).
Butlld
currently doesn't have such a wrapper [that fixes the rpaths].
Concretely, we want to be able to:
- Use a nix-provided
lld
innix-build
andnix-shell
to link executables, also on non-NixOS platforms like Ubuntunix-shell
support means it should work without sandbox, withoutlld
picking up some global paths that it has hardcoded; it should be patched or wrapped to not do that, like binutilsld
is
- Use lld for general nix builds of specific or all packages (
lddClang
sounds like it does exactly that)
If your lldClang
env fixes all those points, then this can be closed; if it does not fix/wrap/patch lld
so that the nix-shell-on-Ubuntu use case works, then we still have to do that.
Can't easily override stdenv in haskell #66267, but could be checked by overriding it in nixpkgs and see if darwin builds.
Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:
- Search for maintainers and people that previously touched the related code and @ mention them in a comment.
- Ask on the NixOS Discourse.
- Ask on the #nixos channel on irc.freenode.net.
I discovered that the following can be used for incremental development:
{ pkgs ? import <nixpkgs> {} }:
let
bintools_lld = pkgs.wrapBintoolsWith {
bintools = pkgs.llvmPackages_10.bintools;
};
in pkgs.mkShell {
buildInputs = [ pkgs.stdenv bintools_lld ];
hardeningDisable = [ "fortify" ];
NIX_CFLAGS_LINK = "-fuse-ld=lld";
}
It would be great if there was an easier way to achieve that, but I'm unsure of what would be a good way to expose lld.
You could use nixpkgs.llvmPackages_10.lldClang.bintools
instead of bintools_lld
, this makes it pretty simple.
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/rust-c-link-time-optimization/13276/1
Using llvmPackages.bintools
fixes similar issue for me.
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/how-rpath-works-or-doesnt-on-nixos/18182/1
Let me restate the solution more clearly:
There are two packages that provide LLD in nixpkgs: pkgs.lld
and pkgs.llvmPackages.bintools
. The former is unwrapped, and doesn't, eg, set rpath. The latter is wrapped, and just works. So, the following fixed the issue I was having with compiling Rust code:
That raises the question whether pkgs.lld
should also provide a wrapped lld so the next person doesn't have to go through the same journey. I'm not sure whether the unwrapped version actually provides any value.
How did you make it work in the end exactly ?
On ubuntu I tried the following
- having
RUSTFLAGS="-Clink-arg=-fuse-ld=lld"
(not inside the cargo config, but as an env var). - As a build input, I added
llvmPackages.bintools
. I still ran into some linker errors.
Just wondering if you have more knowledge on this.
@happysalada https://matklad.github.io/2022/03/14/rpath-or-why-lld-doesnt-work-on-nixos.html includes some notes on debugging. Useful things:
- Compare working and broken linkers
readelf -d target/debug/examples/my-binary | rg PATH
to get the actual RPATHcat (which ld.lld)
to check if it's actually a script- check which C compiler is used as a linker. For me right now,
cc
isclang
for me adding lld_13
to the list of buildInputs did the trick. It makes lld work on darwin as well.
Thanks again for the help!
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/using-mold-as-linker-prevents-libraries-from-being-found/18530/2
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/using-mold-as-linker-prevents-libraries-from-being-found/18530/3
I used the following to make lld
work:
# Define settings
# Toolchain.
gccVersion = "13";
gccPkg = pkgs."gcc${gccVersion}";
llvmVersion = "17";
llvmPkgs = pkgs."llvmPackages_${llvmVersion}";
clangStdEnv = pkgs.stdenvAdapters.overrideCC llvmPkgs.stdenv (
llvmPkgs.clang.override {
bintools = llvmPkgs.bintools;
gccForLibs = gccPkg.cc; # This is the unwrapped gcc.
}
);
clangStdEnv.mkDerivation( ... )
Here's a wrapped bintools
that's cross-platform and includes lld
(tested on linux
and darwin
):
{ clangStdenv, lld, path }:
clangStdenv.cc.bintools.override {
extraBuildCommands = ''
for ld in $(find ${lld}/bin -name "ld*" -printf "%f\n"); do
wrap ${clangStdenv.cc.bintools.targetPrefix}$ld \
${path + /pkgs/build-support/bintools-wrapper/ld-wrapper.sh} \
${lld}/bin/$ld
done
'';
}
@szlend however on linux it shouldn't be using clangStdenv?