tweag/rules_nixpkgs

Binaries created using rules_nixpkgs have rpath for libraries from nix store

malt3 opened this issue · 3 comments

malt3 commented

Describe the bug

I have a go binary that is built in Baze using rules_nixpkgs Go and CC toolchains. The Go binary uses cgo to link cryptsetup from nixpkgs.
The resulting binary uses an RPATH to link the top level libcryptsetup.so.12, while all transitive deps are linked with their absolute store path.

To Reproduce

  • Wrap a C library from nix in a cc_library
  • Link it with any native code (could be a cc_binary or go_binary or something else)
  • Run ldd on the result

Expected behavior

The resulting binary should have only absolute paths in the ldd output, since a nix store is required at runtime anyways.

Actual behavior

The top level library that I link against (libcryptsetup.so.12) uses a location relative to the binary (requiring runfiles to be shipped with the binary itself):

rpath looks like this:

patchelf --print-rpath bootstrapper 
$ORIGIN/../../../../_solib_k8/_U@cryptsetup_Ux86_U64-linux_S_S_Ccryptsetup___Ulib:$ORIGIN/bootstrapper.runfiles/constellation/_solib_k8/_U@cryptsetup_Ux86_U64-linux_S_S_Ccryptsetup___Ulib:/nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib:/nix/store/9fy9zzhf613xp0c3jsjxbjq6yp8afrsv-gcc-12.3.0-lib/lib

ldd bazel-bin/bootstrapper/cmd/bootstrapper/bootstrapper_/bootstrapper
        linux-vdso.so.1 (0x00007ffff7fc8000)
        libm.so.6 => /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib/libm.so.6 (0x00007ffff7ee2000)
        libcryptsetup.so.12 => /home/malte/github.com/edgelesssys/constellation/bazel-bin/bootstrapper/cmd/bootstrapper/bootstrapper_/../../../../_solib_k8/_U@cryptsetup_Ulib_S_S_Ccryptsetup___Ulib/libcryptsetup.so.12 (0x00007ffff7e60000)
        libpthread.so.0 => /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib/libpthread.so.0 (0x00007ffff7e5b000)
        libc.so.6 => /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib/libc.so.6 (0x00007ffff7c73000)
        /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib/ld-linux-x86-64.so.2 => /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib64/ld-linux-x86-64.so.2 (0x00007ffff7fca000)
        libuuid.so.1 => /nix/store/dnhl6kcdmhad5402v0irwx3mdfjyzf6i-util-linux-minimal-2.39.2-lib/lib/libuuid.so.1 (0x00007ffff7c67000)
        libdevmapper.so.1.02 => /nix/store/4iajj5xvjys2bis1g8kvrj43c2khw9aj-lvm2-2.03.22-lib/lib/libdevmapper.so.1.02 (0x00007ffff7c0c000)
        libssl.so.3 => /nix/store/vzajrlhsdv2d39s7v6zv09ggajs05gwj-openssl-3.0.11/lib/libssl.so.3 (0x00007ffff7b5f000)
        libcrypto.so.3 => /nix/store/vzajrlhsdv2d39s7v6zv09ggajs05gwj-openssl-3.0.11/lib/libcrypto.so.3 (0x00007ffff7600000)
        libargon2.so.1 => /nix/store/x3q760hm3rqi72vpqcv4552wyx7x3z9i-libargon2-20190702/lib/libargon2.so.1 (0x00007ffff7b53000)
        librt.so.1 => /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib/librt.so.1 (0x00007ffff7b4e000)
        libdl.so.2 => /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib/libdl.so.2 (0x00007ffff7b49000)
        libjson-c.so.5 => /nix/store/ndc2l8gyajdfpagccpiymacnmpnmn50i-json-c-0.16/lib/libjson-c.so.5 (0x00007ffff7b38000)
        libblkid.so.1 => /nix/store/dnhl6kcdmhad5402v0irwx3mdfjyzf6i-util-linux-minimal-2.39.2-lib/lib/libblkid.so.1 (0x00007ffff7adc000)
        libgcc_s.so.1 => /nix/store/9fy9zzhf613xp0c3jsjxbjq6yp8afrsv-gcc-12.3.0-lib/lib/libgcc_s.so.1 (0x00007ffff7ab9000)
        libudev.so.1 => /nix/store/62sdl1d0qd0qr9llfvx6w4yd2pr1v0bh-systemd-minimal-254.3/lib/libudev.so.1 (0x00007ffff75bf000)
        libcap.so.2 => /nix/store/c3nvhic2lcxbbyln5v4x9bnl9jdcw8m5-libcap-2.69-lib/lib/libcap.so.2 (0x00007ffff7aad000)

Environment

  • NixOS 23.05 (amd64)
  • Latest HEAD: e632be8

Additional context

I'm currently thinking about using `patchelf as a postprocessing step to normalize library paths (and always use absolute paths into the nix store). Not sure if that's a good idea but it is the best I came up with so far.

As an example, I tried "repairing" the binary, by setting a working rpath:

patchelf --set-rpath /nix/store/36r63mp6pi1mymcqzwb0rnc7z2x249i6-cryptsetup-2.6.1/lib:/nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib:/nix/store/9fy9zzhf613xp0c3jsjxbjq6yp8afrsv-gcc-12.3.0-lib/lib bootstrapper-fixed 
ldd bootstrapper-fixed 
        linux-vdso.so.1 (0x00007ffff7fc8000)
        libm.so.6 => /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib/libm.so.6 (0x00007ffff7ee2000)
        libcryptsetup.so.12 => /nix/store/36r63mp6pi1mymcqzwb0rnc7z2x249i6-cryptsetup-2.6.1/lib/libcryptsetup.so.12 (0x00007ffff7e60000)
        libpthread.so.0 => /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib/libpthread.so.0 (0x00007ffff7e5b000)
        libc.so.6 => /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib/libc.so.6 (0x00007ffff7c73000)
        /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib/ld-linux-x86-64.so.2 => /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib64/ld-linux-x86-64.so.2 (0x00007ffff7fca000)
        libuuid.so.1 => /nix/store/dnhl6kcdmhad5402v0irwx3mdfjyzf6i-util-linux-minimal-2.39.2-lib/lib/libuuid.so.1 (0x00007ffff7c67000)
        libdevmapper.so.1.02 => /nix/store/4iajj5xvjys2bis1g8kvrj43c2khw9aj-lvm2-2.03.22-lib/lib/libdevmapper.so.1.02 (0x00007ffff7c0c000)
        libssl.so.3 => /nix/store/vzajrlhsdv2d39s7v6zv09ggajs05gwj-openssl-3.0.11/lib/libssl.so.3 (0x00007ffff7b5f000)
        libcrypto.so.3 => /nix/store/vzajrlhsdv2d39s7v6zv09ggajs05gwj-openssl-3.0.11/lib/libcrypto.so.3 (0x00007ffff7600000)
        libargon2.so.1 => /nix/store/x3q760hm3rqi72vpqcv4552wyx7x3z9i-libargon2-20190702/lib/libargon2.so.1 (0x00007ffff7b55000)
        librt.so.1 => /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib/librt.so.1 (0x00007ffff7b4e000)
        libdl.so.2 => /nix/store/gqghjch4p1s69sv4mcjksb2kb65rwqjy-glibc-2.38-23/lib/libdl.so.2 (0x00007ffff7b49000)
        libjson-c.so.5 => /nix/store/ndc2l8gyajdfpagccpiymacnmpnmn50i-json-c-0.16/lib/libjson-c.so.5 (0x00007ffff7b38000)
        libblkid.so.1 => /nix/store/dnhl6kcdmhad5402v0irwx3mdfjyzf6i-util-linux-minimal-2.39.2-lib/lib/libblkid.so.1 (0x00007ffff7adc000)
        libgcc_s.so.1 => /nix/store/9fy9zzhf613xp0c3jsjxbjq6yp8afrsv-gcc-12.3.0-lib/lib/libgcc_s.so.1 (0x00007ffff7abb000)
        libudev.so.1 => /nix/store/62sdl1d0qd0qr9llfvx6w4yd2pr1v0bh-systemd-minimal-254.3/lib/libudev.so.1 (0x00007ffff75bf000)
        libcap.so.2 => /nix/store/c3nvhic2lcxbbyln5v4x9bnl9jdcw8m5-libcap-2.69-lib/lib/libcap.so.2 (0x00007ffff7aad000)