Repaired wheel can't find some of its included libraries
OKaluza opened this issue · 6 comments
Hi, thanks for this really useful software. Unfortunately some of my wheel builds using cibuildwheel 2.12.1 with auditwheel 5.3.0 are broken and I can't find a way to fix them without manually patching the wheel or requiring LD_LIBRARY_PATH to be set:
Testing the package:
$ pip install lavavu-osmesa
...
$ python -m lavavu
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 187, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/usr/lib/python3.10/runpy.py", line 146, in _get_module_details
return _get_module_details(pkg_main_name, error)
File "/usr/lib/python3.10/runpy.py", line 110, in _get_module_details
__import__(pkg_name)
File "/home/owen/python-default/lib/python3.10/site-packages/lavavu/__init__.py", line 10, in <module>
from .lavavu import *
File "/home/owen/python-default/lib/python3.10/site-packages/lavavu/lavavu.py", line 77, in <module>
import LavaVuPython
File "/home/owen/python-default/lib/python3.10/site-packages/lavavu/LavaVuPython.py", line 15, in <module>
import _LavaVuPython
ImportError: libffi-1c6807d3.so.6.0.4: cannot open shared object file: No such file or directory
Running ldd on the module shows three libraries not found:
~/python-default/lib/python3.10/site-packages/lavavu$ ldd _LavaVuPython.cpython-310-x86_64-linux-gnu.so
linux-vdso.so.1 (0x00007fffa2eeb000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f7445f13000)
libtiff-e2a5092b.so.5.2.6 => /home/owen/python-default/lib/python3.10/site-packages/lavavu/./../lavavu_osmesa.libs/libtiff-e2a5092b.so.5.2.6 (0x00007f7445400000)
(...cut)
libffi-1c6807d3.so.6.0.4 => not found
libedit-7f8577df.so.2.0.55 => not found
libtinfo-33626a82.so.5.9 => not found
(...cut)
Yet, inspecting rpath is correct and ls shows the libraries exist there:
$ readelf -d _LavaVuPython.cpython-310-x86_64-linux-gnu.so |head -20
Dynamic section at offset 0x2b90000 contains 37 entries:
Tag Type Name/Value
0x000000000000000f (RPATH) Library rpath: [$ORIGIN:$ORIGIN/../lavavu_osmesa.libs]
(...cut)
$ ls ../lavavu_osmesa.libs/{libtinfo*,libffi*,libedit*}
../lavavu_osmesa.libs/libedit-7f8577df.so.2.0.55 ../lavavu_osmesa.libs/libffi-1c6807d3.so.6.0.4 ../lavavu_osmesa.libs/libtinfo-33626a82.so.5.9
None of these libraries are direct dependencies of my module, they seem to be dependencies of other dependencies (probably from libOSMesa as my main package/module without this library works fine). It seems those libraries aren't able to find their own dependencies in the same directory.
I can get around the issue by setting LD_LIBRARY_PATH=${HOME}/python-default/lib/python3.10/site-packages/lavavu_osmesa.libs or running patchelf --force-rpath --set-rpath '$ORIGIN' *.so* in the lavavu_osmesa.libs directory but I'd like a permanent fix in the wheel.
Am I doing something wrong or is this a bug? Any help would be greatly appreciated.
Could you upload the bad wheel somewhere, or provide instructions to reproduce the issue?
If you try with the previous version $ pip install lavavu-osmesa==1.8.44 the issue above appears exactly as described, I tested just now with lavavu_osmesa-1.8.44-cp310-cp310-manylinux_2_24_x86_64.whl but if I remember correctly, all the manylinux wheels of the 1.8.44 release are broken.
The current release is working because I added a script to fix the broken wheels (https://github.com/lavavu/LavaVu/blob/master/patchwheel.py)
auditwheel lddtree _LavaVuPython.cpython-310-x86_64-linux-gnu.so reveals that the library which needs the missing 3 .so files is libLLVM-3-6d00db57.8.so.1:
"libLLVM-3-6d00db57.8.so.1": {
"realpath": "/tmp/lavavu/lavavu_osmesa.libs/libLLVM-3-6d00db57.8.so.1",
"path": "/tmp/lavavu/lavavu_osmesa.libs/libLLVM-3-6d00db57.8.so.1",
"needed": [
"libffi-1c6807d3.so.6.0.4",
"libedit-7f8577df.so.2.0.55",
"librt.so.1",
"libdl.so.2",
"libtinfo-33626a82.so.5.9",
"libpthread.so.0",
"libz.so.1",
"libstdc++.so.6",
"libm.so.6",
"libgcc_s.so.1",
"libc.so.6",
"ld-linux-x86-64.so.2"
]
},
It seems that the RUNPATH in this library hasn't been changed by auditwheel:
❮ readelf -d /tmp/lavavu/lavavu_osmesa.libs/libLLVM-3-6d00db57.8.so.1 | grep RUNPATH
0x000000000000001d (RUNPATH) Library runpath: [$ORIGIN/../lib]
Stripping the RUNPATH entry fixes the issue and the missing libraries get picked up from the right place:
libffi-1c6807d3.so.6.0.4 => /tmp/lavavu/lavavu/./../lavavu_osmesa.libs/libffi-1c6807d3.so.6.0.4 (0x00007f2f8af6c000)
libedit-7f8577df.so.2.0.55 => /tmp/lavavu/lavavu/./../lavavu_osmesa.libs/libedit-7f8577df.so.2.0.55 (0x00007f2f8ad2e000)
libtinfo-33626a82.so.5.9 => /tmp/lavavu/lavavu/./../lavavu_osmesa.libs/libtinfo-33626a82.so.5.9 (0x00007f2f8ab02000)
I thought auditwheel stripped the RUNPATH entries in dependencies, but I might be wrong. We should probably update the RUNPATH in dependencies as well and point it to the vendor libs path.
@OKaluza I tried building the project here but it failed with a compiler error. I can give it another go when I get more time, but if you could upload the unrepaired wheel somewhere, that would help.
Ah sorry about that, attempting to prepare for distutils being deprecated and broke things, build should be fixed now.
Here is an unrepaired build from my machine: https://github.com/lavavu/LavaVu/releases/download/1.8.45/lavavu_osmesa-1.8.45-cp311-cp311-linux_x86_64.whl
Thanks. I just checked this and I cannot reproduce the issue when repairing this wheel. I am repairing it for manylinux_2_35_x86_64 (this seems to be the lowest version for the required ABIs) and get a seemingly correctly repaired wheel with the three .so correctly located by ldd:
root@fb3d28a7ec3e unzip]# ldd lavavu/_LavaVuPython.cpython-311-x86_64-linux-gnu.so
...
libGL.so.1 => /usr/lib/libGL.so.1 (0x00007fbe670b1000)
libX11.so.6 => /usr/lib/libX11.so.6 (0x00007fbe66f6e000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007fbe66cf1000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007fbe66c04000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007fbe66bdf000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007fbe669f5000)
libzstd-7da67a97.so.1.5.5 => /tmp/wheelhouse/unzip/lavavu/../lavavu_osmesa.libs/libzstd-7da67a97.so.1.5.5 (0x00007fbe6691c000)
liblzma-57832d06.so.5.4.3 => /tmp/wheelhouse/unzip/lavavu/../lavavu_osmesa.libs/liblzma-57832d06.so.5.4.3 (0x00007fbe668e7000)
libjpeg-77ed9821.so.8.2.2 => /tmp/wheelhouse/unzip/lavavu/../lavavu_osmesa.libs/libjpeg-77ed9821.so.8.2.2 (0x00007fbe6685f000)
However, libLLVM is not in the wheel. How exactly do you run auditwheel to get the libLLVM dependency vendored? I took a look at the GH actions in the project but couldn't see anything relevant.
I have the following files in lavavu_osmesa.libs:
libjpeg-77ed9821.so.8.2.2
liblzma-57832d06.so.5.4.3
libpng16-3b625471.so.16.39.0
libtiff-e33d88cd.so.6.0.0
libzstd-7da67a97.so.1.5.5
This is a much smaller set of libraries than in the wheel on PyPI so there is probably a difference in library dependencies in the build environment.
Sorry my mistake, it seems libOSMesa wasn't installed anymore on the machine I built it on, I rebuilt the wheel and it definitely links to libLLVM now: https://github.com/lavavu/LavaVu/releases/download/1.8.45/lavavu_osmesa-1.8.45-cp311-cp311-linux_x86_64.whl
$ ldd _LavaVuPython.cpython-311-x86_64-linux-gnu.so
linux-vdso.so.1 (0x00007ffd53861000)
libpng16.so.16 => /lib/x86_64-linux-gnu/libpng16.so.16 (0x00007f97ffa70000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f97ffa52000)
libtiff.so.6 => /lib/x86_64-linux-gnu/libtiff.so.6 (0x00007f97ff9c4000)
libOSMesa.so.8 => /lib/x86_64-linux-gnu/libOSMesa.so.8 (0x00007f97fea00000)
libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f97fe600000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f97fe917000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f97ff99e000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f97fe200000)
libwebp.so.7 => /lib/x86_64-linux-gnu/libwebp.so.7 (0x00007f97ff591000)
libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x00007f97fe54c000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f97ff96c000)
libLerc.so.4 => /lib/x86_64-linux-gnu/libLerc.so.4 (0x00007f97fe892000)
libjbig.so.0 => /lib/x86_64-linux-gnu/libjbig.so.0 (0x00007f97ff959000)
libjpeg.so.8 => /lib/x86_64-linux-gnu/libjpeg.so.8 (0x00007f97fe4c9000)
libdeflate.so.0 => /lib/x86_64-linux-gnu/libdeflate.so.0 (0x00007f97ff57e000)
libglapi.so.0 => /lib/x86_64-linux-gnu/libglapi.so.0 (0x00007f97fe48f000)
libLLVM-15.so.1 => /lib/x86_64-linux-gnu/libLLVM-15.so.1 (0x00007f97f7000000)
/lib64/ld-linux-x86-64.so.2 (0x00007f97ffac5000)
libffi.so.8 => /lib/x86_64-linux-gnu/libffi.so.8 (0x00007f97fe887000)
libedit.so.2 => /lib/x86_64-linux-gnu/libedit.so.2 (0x00007f97fe455000)
libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007f97fe423000)
libxml2.so.2 => /lib/x86_64-linux-gnu/libxml2.so.2 (0x00007f97f6e16000)
libbsd.so.0 => /lib/x86_64-linux-gnu/libbsd.so.0 (0x00007f97fe872000)
libicuuc.so.72 => /lib/x86_64-linux-gnu/libicuuc.so.72 (0x00007f97f6c17000)
libmd.so.0 => /lib/x86_64-linux-gnu/libmd.so.0 (0x00007f97fe416000)
libicudata.so.72 => /lib/x86_64-linux-gnu/libicudata.so.72 (0x00007f97f4e00000)
The build command was:
LV_PACKAGE=lavavu-osmesa LV_OSMESA=1 LV_LIB_DIRS=/usr/local/lib python setup.py bdist_wheel
However unless libOSMesa is found it will still build but will skip this library and won't have the issue (can set LV_LIB_DIRS and LV_INC_DIRS to point to a specific build)