Support for symlinks in wheel files
alcroito opened this issue · 13 comments
Hi, I searched around but I couldn't find any info on this.
Is it by design that wheel files do not preserve symlinks when creating the zip archive?
As stated in https://stackoverflow.com/a/41573364/603646 , there is a way to preserve the symbolic links when creating a zip file via python.
I'd say that it's less by design of wheel, and more by design of Python's packaging system. I'm not sure any part of the packaging infrastructure expects symlinks (certainly they wouldn't be portable reliably to Windows, but I'm sure you know that). I don't know if they'd be preserved in a sdist, or in setup.py install, for example.
I'm going to flat out reject this, mainly because symlinks are poorly supported on Windows. Do you have a pressing use case?
I was mainly investigating this for the purpose of creating binary wheels for PySide2.
For example on macOS we would need to copy over Qt .framework directories, which contain various symlinks that come with a regular macOS bundle (Resources, Libraries, Helpers, Versions/Current, etc). During the setuptools install and wheel creation, all the symlinks get resolved to actual files, leading to wasted space.
Thus I was curious why setuptools / wheels do not support symlinks.
Another reason to support this is that currently libfoo.so -> libfoo.so.n -> libfoo.so.x.y.z symlink chains end up getting packaged in triplicate, exploding package sizes.
They should be supported at least in platform-specific wheels (manylinuxX, darwin, etc.). Though Windows 10 does have more sensible symlink support and Windows 7 is now EOL, maybe this should be revisited.
To me, "reasonable support" means not requiring admin privileges to create symlinks, which they still unfortunately do on the latest Windows.
Latest Windows with developer mode enabled doesn't require admin privileges. Conceded "with developer mode enabled" is not the same as "by default". And the wider point I made above still holds anyway.
Well, we have a wheel that could be 54M, but once we set bdist_wheel on it, ends up being 175MB because of the symlink dereferencing. I've done enough hackery on our build system to appease the wheel system already that more hacks don't really fill me with more joy to support. Especially ones that lay down brand new codepaths (such as skipping SONAME generation) because they're, quite frankly, not a build mode I want laying around in case someone thinks they can get away with such a setting outside of the wheels or having to field questions like "hey, can you expose that option outside of the wheels?".
We already can't support sdist because our build system is too complicated for setup.py to be any kind of useful entry point into it and setup.py install probably doesn't work either. All we need is to make a wheel that is platform-specific anyways (basically, wheels are nothing but a distribution mechanism for us, the rest of the Python setuptools is of no use for this project). We're not a Python project primarily, and never will be, but AFAICT, setup.py is the only supported way to make a wheel, so that's all it can be used for in our project.
@arcivanov how would one use wheel-axle with python3 -m build wheel which is now recommended? Posting here since this GitHub issue shows first in Google when looking for symlinks being ignored (which triples our wheel size for us due to our shared libraries versioning, probably similar to @mathstuf)
build package isn't PyPA and they are switching their build backend from setuptools to Flit, which diverges even further from PyPA.
More to come...
@agirault Here's the example of how to make wheel-axle work with build: https://github.com/karellen/wheel-axle-example
bash-5.2$ python3 -m build -n --wheel
* Getting build dependencies for wheel...
running egg_info
writing src/test_axle_1.egg-info/PKG-INFO
writing dependency_links to src/test_axle_1.egg-info/dependency_links.txt
writing top-level names to src/test_axle_1.egg-info/top_level.txt
reading manifest file 'src/test_axle_1.egg-info/SOURCES.txt'
writing manifest file 'src/test_axle_1.egg-info/SOURCES.txt'
* Building wheel...
running bdist_wheel
running build
running build_py
reproducing link src/bar/foo.so (../../../foo.so) -> build/lib/bar
running build_scripts
copying scripts/script1 -> build/scripts-3.11
changing mode of build/scripts-3.11/script1 from 644 to 755
installing to build/bdist.linux-x86_64/wheel
running install
Distribution option extra_path is deprecated. See issue27919 for details.
running install_lib
creating build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/bar
copying build/lib/bar/__init__.py -> build/bdist.linux-x86_64/wheel/bar
registering link build/lib/bar/foo.so (../../../foo.so) -> build/bdist.linux-x86_64/wheel/bar/foo.so
running install_headers
creating build/bdist.linux-x86_64/wheel/test_axle_1-0.0.1.data
creating build/bdist.linux-x86_64/wheel/test_axle_1-0.0.1.data/headers
copying headers/header1.h -> build/bdist.linux-x86_64/wheel/test_axle_1-0.0.1.data/headers
registering link headers/header2.h (header1.h) -> build/bdist.linux-x86_64/wheel/test_axle_1-0.0.1.data/headers
running install_data
creating build/bdist.linux-x86_64/wheel/test_axle_1-0.0.1.data/data
creating build/bdist.linux-x86_64/wheel/test_axle_1-0.0.1.data/data/lib
copying data/lib/foo.1.so -> build/bdist.linux-x86_64/wheel/test_axle_1-0.0.1.data/data/lib
registering link data/lib/foo.so (foo.1.so) -> build/bdist.linux-x86_64/wheel/test_axle_1-0.0.1.data/data/lib
running install_egg_info
running egg_info
writing src/test_axle_1.egg-info/PKG-INFO
writing dependency_links to src/test_axle_1.egg-info/dependency_links.txt
writing requirements to src/test_axle_1.egg-info/requires.txt
writing top-level names to src/test_axle_1.egg-info/top_level.txt
reading manifest file 'src/test_axle_1.egg-info/SOURCES.txt'
writing manifest file 'src/test_axle_1.egg-info/SOURCES.txt'
Copying src/test_axle_1.egg-info to build/bdist.linux-x86_64/wheel/test_axle_1-0.0.1-py3.11.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/test_axle_1-0.0.1.data/scripts
copying build/scripts-3.11/script1 -> build/bdist.linux-x86_64/wheel/test_axle_1-0.0.1.data/scripts
registering link build/scripts-3.11/script2 (script1) -> build/bdist.linux-x86_64/wheel/test_axle_1-0.0.1.data/scripts/script2
changing mode of build/bdist.linux-x86_64/wheel/test_axle_1-0.0.1.data/scripts/script1 to 755
creating build/bdist.linux-x86_64/wheel/test_axle_1-0.0.1.pth
creating build/bdist.linux-x86_64/wheel/test_axle_1-0.0.1.dist-info/WHEEL
creating '/home/user/Documents/src/wheel-axle-example/dist/tmpa1miw0b7/test_axle_1-0.0.1-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'test_axle_1-0.0.1.pth'
adding 'bar/__init__.py'
adding 'test_axle_1-0.0.1.data/data/lib/foo.1.so'
adding 'test_axle_1-0.0.1.data/headers/header1.h'
adding 'test_axle_1-0.0.1.data/scripts/script1'
adding 'test_axle_1-0.0.1.dist-info/METADATA'
adding 'test_axle_1-0.0.1.dist-info/WHEEL'
adding 'test_axle_1-0.0.1.dist-info/axle.lck'
adding 'test_axle_1-0.0.1.dist-info/symlinks.txt'
adding 'test_axle_1-0.0.1.dist-info/top_level.txt'
adding 'test_axle_1-0.0.1.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
Successfully built test_axle_1-0.0.1-py3-none-any.whl