pypa/cibuildwheel

Tracking issue for broken musllinux wheels

henryiii opened this issue · 4 comments

henryiii commented

See pypa/manylinux#1225 - all muslllinux wheels are broken on Alpine 3.14; the version of Python used in Alpine 3.14 uses the wrong binary names inside the wheels (ending in -musl instead of -gnu) So any wheels produced on musllinux_1_1 do not work on Alpine 3.14.

The patch has not been accepted upstream; -gnu is the correct, musllinux name, and Alpine 3.14 is currently at fault. CPython 3.11 might change the name to -musl, but it probably will not happen for existing Pythons.

Also see:

Current binaries produced:

psycopg_binary-3.0.4-cp310-cp310-musllinux_1_1_x86_64.whl
 - _psycopg.cpython-310-x86_64-linux-gnu.so

What it should be (Alpine 3.14+ has the patched CPython):

psycopg_binary-3.0.4-cp310-cp310-musllinux_1_1_x86_64.whl
 - _psycopg.cpython-310-x86_64-linux-musl.so

We hoped for a possible workaround, but this cannot be done, since wheels are zip files, and the zip module in Python has never supported symlinks:

psycopg_binary-3.0.4-cp310-cp310-musllinux_1_1_x86_64.whl
 - _psycopg.cpython-310-x86_64-linux-musl.so
 - _psycopg.cpython-310-x86_64-linux-gnu.so -> _psycopg.cpython-310-x86_64-linux-musl.so

Here's the current plan:

  • ALPINE: Add a patch on top of the current patch to make CPython look for -gnu on top of -musl for Alpine 3.15 and 3.14. Reverting the patch would break every Alpine wheel previously locally compiled (like NumPy) and would require rebuilding all shipped packages that depend on Python.
  • ALPINE: Revert the patch for CPython 3.10 in Alpine 3.16, due mid next year.
  • CPYTHON: Take the existing patch (bpo-43112: detect musl as a separate SOABI python/cpython#24502) targeting upstream CPython 3.11 and change search to include abi3-gnu on musl after looking for abi3-musl. The ability to install both binaries into a single folder would be a new "feature" of CPython 3.11.
  • AUDITWHEEL: Optionally this could be checked and normalized by auditwheel (like changing -musl to -gnu on 3.9) if desired. ABI3 wheels targeting <3.11 could be normalized to -gnu. (Currently ABI3 wheels do not have platform tags in the filename, so they work everywhere, but can't be in a shared folder.)
henryiii commented

Looks like wheels are zip files and therefore don't support symlinks due to an old CPython bug. So it is not possible to play nice here. Alpine is applying an unaccepted patch onto CPython to change -gnu to -musl, so it's clearly in the wrong unless something is decided by CPython (maybe for 3.11, or maybe it will just be left alone, since not much is to be gained by changing it). We should probably note the problem, and hopefully Alpine will drop their unsupported patch until this is resolved long-term upstream.

henryiii commented

@ncopa and I spoke yesterday and came to a potential path forward. Quick summary added to the description. This has already been fixed in Alpine; 3.14 and 3.15 can now load musllinux wheels.

  • 🎉2
mayeut commented

Thanks @henryiii & @ncopa for figuring out a path forward. I will try to do my best to get something in auditwheel (been quite busy at work these days without much time/energy left to work on open-source projects).

joerick commented

I thought that the patch in Alpine Linux had fixed the issue? (at least for now? I suppose that suffix will get updated at some point, on a CPython minor version, perhaps)