static links to avoid problems (libcrypt.so.1 not found)
mohammad-akhlaghi opened this issue · 19 comments
I just installed Biber 2.19 using the tlmgr install biber on my 64-bit Arch GNU/Linux. However, it doesn't run because libcrypt.so.1 has been updated to libcrypt.so.2:
$ biber
biber: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory
$ ls /usr/lib/libcrypt.*
/usr/lib/libcrypt.so /usr/lib/libcrypt.so.2 /usr/lib/libcrypt.so.2.0.0To avoid such problems, since Biber is primarily distributed as a pre-built binary, I wanted to recommend to link the pre-built binary statically.
Until this problem is fixed, I tried compiling it from source. But I don't know how to get pp on Arch Linux to compile it from source myself, can you please give some explanation on what pp is and the official software name?
@krumeich - this might mean that the binary build script your end needs updating as it might have not included libcrypt? Usually this is totally self-contained and biber comes with its own libcrypt of the right version. If that gets missed out, it might look for the version it expects and not find it.
pp is just a script that comes with the perl module PAR::Packer that is used to package the binary.
Thanks @plk, I was able to get pp, knowing the name, I got it installed with pacman -S perl-par-packer. But the build script for Linux x86_64 depends on too many hard-coded locations (for example I had to change /usr/local/perl/bin/pp to /usr/bin/vendor_perl/pp (which is where Arch installs it, I discovered this through which pp, which can be included in the script 😉). But I got other errors with wrong absolute locations (like below), so I just gave up :-(
$ ./dist/linux_x86_64/build.sh
USING Unicode::Collate at: /usr/local/perl/lib/5.32.0/Unicode/Collate
/usr/bin/vendor_perl/pp: Input file /usr/local/perl/bin/biber was not foundAbout my initial comment, I should add that I am using TeXLive 2023.
I am now shifting my BibLaTeX source files to BibTeX because I have a deadline I need to catch. But please let me know when this is fixed, I really prefer BibLaTeX and I'd be happy to test it 😉.
That error is because you need to install biber first as a perl script with perl Build.PL;./Build install but if you can do this (you should be able to), you don't need to build the binary anyway and can just use the perl installed script. The binary build is really just a convenience - if you can install perl modules and have the required perl version (5.32+), you can just get the source, do "perl ./Build.PL;./Build installdeps;./Build install" and then run the installed "biber" script.
Thanks a lot @plk! I just got it working by running this command as root (there were two small typos in the example you sent, and I prefer to have && between multiple commands in case one crashes 😉):
$ perl ./Build.PL && ./Build installdeps &&./Build installIt took a little long (maybe half an hour!), but after it finished, the new biber (installed by Perl) worked and I was able to build my LaTeX file with BibLaTeX references 😄.
$ biber --version
biber version: 2.19
$ which biber
/usr/bin/site_perl/biberIt may be good to mention this simple command (and the somewhere close to the top of README, it may have been mentioned, but I couldn't find it after a fast look through the sources).
Thanks a lot for this wonderful tool and the prompt help 👍.
I'll add it to the README - I thought it was a bit more visible but I haven't looked in a while.
Thanks for the heads up, I'll take a look at this tomorrow.
The extracted biber directory in $(biber --cache) contains the following files:
libcrypto.so.3, libgcrypt.so.20
I have no idea whether they contain the functionalities that @mohammad-akhlaghi declares missing. One would have to be a bit more of an expert on Arch Linux, which I am not.
It is correct that the platform specific build script in linux-musl_x86_64/build.sh does not contain libcrypt.so.1 – but then neither does the build script in linux_x86_64. This file includes libcrypto.so.0.9.8.
There is a package libc6-compat for Alpine which provides libcrypt.so.1. I could install that and add the library, but a bit of googling led to the discussion on Stack Exchange. So I'm a bit reluctant to add a library that has been declared deprecated and which is not part of biber for linux_x86_64.
Any ideas or clarification on your side?
I notice that in the aarch64-linux build script, it packs .so.1? It might be that there is a symlink and it packs a different version? I've seen that before under MacOS.
The build.sh script for MUSL actually references some symbolic links. I don't want to update my build script only because Alpine updates a library. On the left hand side you see the name that build.sh references. On the right is the name of the library the symlink points to in the dev environment.
/lib/libz.so -> libz.so.1.2.13
/usr/lib/libcrypto.so -> libcrypto.so.3
/usr/lib/libexslt.so -> libexslt.so.0.8.20
/usr/lib/libgcrypt.so -> libgcrypt.so.20.4.1
/usr/lib/libgpg-error.so -> libgpg-error.so.0.33.1
/usr/lib/liblzma.so -> liblzma.so.5.2.9
/usr/lib/libssl.so -> libssl.so.3
/usr/lib/libxml2.so -> libxml2.so.2.10.3
/usr/lib/libxslt.so -> libxslt.so.1.1.37
In the packaged biber, these links are dereferenced and the libraries are available. This is from the extracted biber directory:
-r-xr-xr-x 1 root root 93616 Mar 6 08:39 libbtparse.so
-rwxr-xr-x 1 root root 3882720 Feb 7 16:19 libcrypto.so.3
-rwxr-xr-x 1 root root 79712 Oct 12 00:54 libexslt.so.0
-rwxr-xr-x 1 root root 1226168 Apr 27 2022 libgcrypt.so.20
-rwxr-xr-x 1 root root 129008 Oct 24 21:46 libgpg-error.so.0
-rwxr-xr-x 1 root root 165640 Dec 1 05:05 liblzma.so.5
-rwxr-x--- 1 root root 3409112 Mar 27 11:36 libperl.so
-rwxr-xr-x 1 root root 602120 Feb 7 16:19 libssl.so.3
-rwxr-xr-x 1 root root 1203632 Oct 28 08:09 libxml2.so.2
-rwxr-xr-x 1 root root 231544 Oct 12 00:54 libxslt.so.1
-rwxr-xr-x 1 root root 100264 Oct 13 21:54 libz.so.1
BTW, the macOS version doesn't contain libcrypt.so.1 (or libcrypt.dylib) either.
@mohammad-akhlaghi Could you try installing the https://archlinux.org/packages/core/x86_64/libxcrypt-compat/ package and see if it provides the libcrypt.so.1 dependency? To me, this seems more a distro-specific problem that an architectural issue.
After some research it turns out that ArchLinux is, after all, not a MUSL-based distribution (see also https://wiki.musl-libc.org/projects-using-musl.html). So the issue has nothing to do with the Alpine build. I'll revert the changes in my fork for the Alpine build and not pursue this any further on my end.
It seems that ArchLinux has dropped a library that is required by some part of biber. I can see two options to proceed: Either package libcrypt.so.1 as part of biber, thereby providing a dependency only for ArchLinux; or adding a note to the README that ArchLinux users have to install the compatibility package libxcrypt-compat linked above. The OP hasn't yet responded yet whether installing this package has solved their problem.
So, @plk, this one's up to you!
Sorry for the late reply, I had a deadline I had to reach, and I was successfully able to use the latest Biber thanks to your help in preparing the proposal. Now, let's get back to debugging this:
Could you try installing the libxcrypt-compat package and see if it provides the libcrypt.so.1 dependency?
It does indeed provide libcrypt.so.1. See below for the versions of libcrypt after the installation (with pacman -S libxcrypt-compat):
ls -l /usr/lib/libcrypt.*
lrwxrwxrwx 1 root root 17 Nov 19 22:01 /usr/lib/libcrypt.so -> libcrypt.so.2.0.0
lrwxrwxrwx 1 root root 17 Nov 19 22:01 /usr/lib/libcrypt.so.1 -> libcrypt.so.1.1.0
-rwxr-xr-x 1 root root 182208 Nov 19 22:01 /usr/lib/libcrypt.so.1.1.0
lrwxrwxrwx 1 root root 17 Nov 19 22:01 /usr/lib/libcrypt.so.2 -> libcrypt.so.2.0.0
-rwxr-xr-x 1 root root 165824 Nov 19 22:01 /usr/lib/libcrypt.so.2.0.0About the dependencies, isn't it possible to do static linking in the system that builds the executable? In this way, you won't have to package any extra library and there will be no dependency problem for anyone (in general, static linking is more portable).
If you install this compat package on the build machine - it should package it in biber. If not, just include an explicit --link line in the build script for that lib.
@plk: I am not sure if that comment was for me. But if so, after I built Biber from source (using the perl ./Build.PL ... command above), I had no problem. The way I have understood, the problem is that the system that Biber is being compiled/linked on (that produces the binaries people download from tlmgr), has an old version of libcrypt. Until all GNU-based operating systems update their GNU C Library, the most portable way that comes to my mind is static linking on that core system. But well, I am not familiar with the packaging system you use, so I understand that this may not be possible.
That was intended for @krumeich - you are using the perl install of biber which isn't packaged and so this doesn't matter so much for you.
I must admit that I'm a bit confused.
@mohammad-akhlaghi What ist the name of the architecture that ArchLinux uses? It's x86_64, isn't it? If so, ArchLinux is a libc based distribution and does not use MUSL. Then it wouldn't matter if I added the compat package on my build machine, because an ArchLinux machine does not use the build I provide.
@krumeich: yes, my architecture is x86_64.
Indeed, but if you do the build on your host system with static linking, the necessary parts of your host's libcrypt will be included within the biber binary that gets uploaded to TeXLive. So none of your users will ever need to worry about the version they have. You can check the dynamically linked libraries of a binary file with the ldd command on any GNU C Library based operating system (ldd is part of GNU C Library). For example, here is what I see when I check TeXLive's biber:
$ ldd --version
ldd (GNU libc) 2.37
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.
$ ldd /usr/local/texlive/2023/bin/x86_64-linux/biber
linux-vdso.so.1 (0x00007fff0ef88000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007fa6d1a9e000)
libnsl.so.1 => /usr/lib/libnsl.so.1 (0x00007fa6d1a84000)
libdl.so.2 => /usr/lib/libdl.so.2 (0x00007fa6d1a7f000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007fa6d1997000)
libcrypt.so.1 => /usr/lib/libcrypt.so.1 (0x00007fa6d1961000)
libutil.so.1 => /usr/lib/libutil.so.1 (0x00007fa6d195c000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007fa6d1773000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fa6d1acb000)But before (when I didn't have the libxcrypt-compat package) I got the problem in my first comment. If you statically link with all the dependency libraries, then ldd should show something like this:
$ ldd /usr/local/texlive/2023/bin/x86_64-linux/biber
linux-vdso.so.1 (0x00007fff0ef88000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fa6d1acb000)This will be portable to any system, even if they do not have the GNU C Library 😉. But I do not know the details of the build system you use for a more detailed suggestion. In a normal GNU build system (./configure && make && sudo make install), we usually have a configure time option called --disable-shared or --enable-shared=no. When a program is configured like this, all static libraries that are present are preferred to their shared versions and we get a much more portable binary (with fewer lines when checking with ldd).
@mohammad-akhlaghi Thanks a lot for elaborating on this. I took the opportunity to learn a lot more about the MUSL-platform (and Alpine Linux). The MUSL libc implementation contains the functionality of libcrypt (hash algorithms). On a freshly installed Alpine system, there isn't a /lib/libcrypt.so.1 or anything similar. /lib is suspiciously empty on that platform (so is /usr/lib):
-rwxr-xr-x 1 root root 616992 Nov 10 18:24 ld-musl-x86_64.so.1
-rwxr-xr-x 1 root root 184008 Oct 23 19:14 libapk.so.3.12.0
lrwxrwxrwx 1 root root 19 Feb 10 16:45 libc.musl-x86_64.so.1 -> ld-musl-x86_64.so.1
-rwxr-xr-x 1 root root 3882720 Feb 7 16:19 libcrypto.so.3
-rwxr-xr-x 1 root root 602120 Feb 7 16:19 libssl.so.3
lrwxrwxrwx 1 root root 14 Feb 10 16:45 libz.so.1 -> libz.so.1.2.13
-rwxr-xr-x 1 root root 100264 Oct 13 21:54 libz.so.1.2.13
When I add the compatibility package libc6-compat, only a handful of symlinks are added to /lib:
-rwxr-xr-x 1 root root 616992 Nov 10 18:24 ld-musl-x86_64.so.1
-rwxr-xr-x 1 root root 184008 Oct 23 19:14 libapk.so.3.12.0
lrwxrwxrwx 1 root root 19 Feb 10 16:45 libc.musl-x86_64.so.1 -> ld-musl-x86_64.so.1
lrwxrwxrwx 1 root root 26 Mar 31 20:11 libc.so.6 -> /lib/libc.musl-x86_64.so.1
lrwxrwxrwx 1 root root 26 Mar 31 20:11 libcrypt.so.1 -> /lib/libc.musl-x86_64.so.1
-rwxr-xr-x 1 root root 3882720 Feb 7 16:19 libcrypto.so.3
lrwxrwxrwx 1 root root 26 Mar 31 20:11 libm.so.6 -> /lib/libc.musl-x86_64.so.1
lrwxrwxrwx 1 root root 26 Mar 31 20:11 libpthread.so.0 -> /lib/libc.musl-x86_64.so.1
lrwxrwxrwx 1 root root 26 Mar 31 20:11 librt.so.1 -> /lib/libc.musl-x86_64.so.1
-rwxr-xr-x 1 root root 602120 Feb 7 16:19 libssl.so.3
lrwxrwxrwx 1 root root 26 Mar 31 20:11 libutil.so.1 -> /lib/libc.musl-x86_64.so.1
lrwxrwxrwx 1 root root 14 Feb 10 16:45 libz.so.1 -> libz.so.1.2.13
-rwxr-xr-x 1 root root 100264 Oct 13 21:54 libz.so.1.2.13
As you can see, these compatibility symlinks (including libcrypt.so.1) point to the libc-MUSL implementation that is there in the first place. So I'm not sure we'd gain a lot by statically linking the standard library.
I do get your point though: If this were a special library not available on the system by default, this would make a lot more sense. But here, it seems to me that this is redundant.
I already have the --link=/lib/libcrypt.so.1 directive in the /dist/linux-musl_x86_64/build.sh, als @plk suggested, but it makes no difference in the packaged version of biber. Biber runs without a hitch.
OK, folks, I'm off on vacation for a week starting tomorrow. I won't have access to the machine that I'm building this on. Let's continue the discussion after that, if it's necessary.