"Python installation" method fails on musl-based Linux
svenha opened this issue · 9 comments
The method "Python Installation" installs the current and all old versions of larynx (7: 1.0.3 down to 0.3.0) and gruut in this step:
pip3 install -f 'https://synesthesiam.github.io/prebuilt-apps/' -f 'https://download.pytorch.org/whl/cpu/torch_stable.html' larynx
Same for the simpler command:
pip3 install larynx
Of course, this ends in massive version conflicts. The problem first occurred after version 1.0.0.
My python3 is version 3.9.7.
This sounds like there's a version conflict in your virtual environment with something that larynx depends on. pip's response to this is to backtrack versions until something installs.
In a fresh virtual environment on Python 3.9.2, I have no conflicts with pip3 install larynx. Do you have something that depends on a specific version of numpy maybe?
numpy was a good guess because it seems to go wrong when installing numpy (with pip3 install larynx -v):
Using pip 21.3.1 from /home/phablet/larynx_venv/lib/python3.9/site-packages/pip (python 3.9)
Collecting larynx
Using cached larynx-1.1.0.tar.gz (31.7 MB)
Preparing metadata (setup.py): started
Running command python setup.py egg_info
running egg_info
creating /tmp/pip-pip-egg-info-_75u5pdy/larynx.egg-info
writing /tmp/pip-pip-egg-info-_75u5pdy/larynx.egg-info/PKG-INFO
writing dependency_links to /tmp/pip-pip-egg-info-_75u5pdy/larynx.egg-info/dependency_links.txt
writing entry points to /tmp/pip-pip-egg-info-_75u5pdy/larynx.egg-info/entry_points.txt
writing requirements to /tmp/pip-pip-egg-info-_75u5pdy/larynx.egg-info/requires.txt
writing top-level names to /tmp/pip-pip-egg-info-_75u5pdy/larynx.egg-info/top_level.txt
writing manifest file '/tmp/pip-pip-egg-info-_75u5pdy/larynx.egg-info/SOURCES.txt'
reading manifest file '/tmp/pip-pip-egg-info-_75u5pdy/larynx.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file '/tmp/pip-pip-egg-info-_75u5pdy/larynx.egg-info/SOURCES.txt'
Preparing metadata (setup.py): finished with status 'done'
Collecting dataclasses-json~=0.5.0
Using cached dataclasses_json-0.5.6-py3-none-any.whl (25 kB)
Collecting gruut[de,es,fr,it,nl,ru,sv,sw]~=2.1.0
Using cached gruut-2.1.0.tar.gz (72 kB)
Preparing metadata (setup.py): started
Running command python setup.py egg_info
running egg_info
creating /tmp/pip-pip-egg-info-05z9u4c7/gruut.egg-info
writing /tmp/pip-pip-egg-info-05z9u4c7/gruut.egg-info/PKG-INFO
writing dependency_links to /tmp/pip-pip-egg-info-05z9u4c7/gruut.egg-info/dependency_links.txt
writing entry points to /tmp/pip-pip-egg-info-05z9u4c7/gruut.egg-info/entry_points.txt
writing requirements to /tmp/pip-pip-egg-info-05z9u4c7/gruut.egg-info/requires.txt
writing top-level names to /tmp/pip-pip-egg-info-05z9u4c7/gruut.egg-info/top_level.txt
writing manifest file '/tmp/pip-pip-egg-info-05z9u4c7/gruut.egg-info/SOURCES.txt'
reading manifest file '/tmp/pip-pip-egg-info-05z9u4c7/gruut.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file '/tmp/pip-pip-egg-info-05z9u4c7/gruut.egg-info/SOURCES.txt'
Preparing metadata (setup.py): finished with status 'done'
Collecting numpy>=1.20.0
Using cached numpy-1.21.4.zip (10.6 MB)
Installing build dependencies: started
Running command /home/phablet/larynx_venv/bin/python3 /tmp/pip-standalone-pip-5cq45ik1/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-gl96jbwm/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'packaging==20.5; platform_machi
Ignoring packaging: markers 'platform_machine == "arm64"' don't match your environment'
So, this seems to be a numpy problem.
There is a working py3-numpy package on postmarketOS that I could install. How can I use it inside the venv?
You can give your virtual environment access to system packages by creating it with:
python3 -m venv --system-site-packages ...Which CPU architecture are you running on? If it's an ARM device, you may be able to use the pre-compiled numpy wheels here: https://www.piwheels.org/project/numpy/
It's an arm64 dervice, but currently running postmarketOS (based on Alpine Linux) which uses musl as libc. With the option above, it looks like onnxruntime cannot be installed. Interestingly, https://www.piwheels.org/project/onnx/ has only failures.
I tried a fresh installation and closely debugged all steps. The problem is that no tag matches the onnxruntime wheels in https://synesthesiam.github.io/prebuilt-apps/ (there is only armv7l).
The tags that are reported as compatible by pip3 debug --verbose | grep cp39 are as follows; the 4th looks promising?
cp39-cp39-musllinux_1_2_aarch64
cp39-cp39-musllinux_1_1_aarch64
cp39-cp39-musllinux_1_0_aarch64
cp39-cp39-linux_aarch64
cp39-abi3-musllinux_1_2_aarch64
cp39-abi3-musllinux_1_1_aarch64
cp39-abi3-musllinux_1_0_aarch64
cp39-abi3-linux_aarch64
cp39-none-musllinux_1_2_aarch64
cp39-none-musllinux_1_1_aarch64
cp39-none-musllinux_1_0_aarch64
cp39-none-linux_aarch64
cp39-none-any
I tried a wheel from pypi.org named onnxruntime-1.9.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl , but it "is not a supported wheel on this platform". So, its seems that on Alpine Linux, pmOS and friends the following tag is needed: cp39-cp39-musllinux_1_2_aarch64 or similar. As tools like auditwheel added musllinux support two months ago, the situation should improve over next months (?).
BTW: I installed onnx easily (without a wheel). How is the relation between onnx and onnxruntime?
OK. I read a little bit about onnxruntime. The problem here is the missing musl support. This was requested 22 months ago: microsoft/onnxruntime#2909 But it seems that a solution is coming closer, now that more and more related/required tools support musl. Let's see ...
It's an arm64 dervice, but currently running postmarketOS (based on Alpine Linux) which uses musl as libc. With the option above, it looks like onnxruntime cannot be installed. Interestingly, https://www.piwheels.org/project/onnx/ has only failures.
This is because onnxruntime doesn't have a working sdist.
I tried the python install on Arch and had the same error as described above with onnxruntime. I then downloaded the .deb package and run debtapscript on it that changes .deb to .pkg.tar.zst. Then I had to build libopenblas from the AUR and after that Larynx install just fine with pacman -U <package-name>. Then I install gruut with pip that I also had not install! But I still have the same error: "ModuleNotFoundError: No module named 'gruut'" How do I fix this? Is it that Larynx is looking for gruut in the wrong place?
Install .deb files on arch: https://www.maketecheasier.com/install-deb-package-in-arch-linux/