pypa/manylinux

Tracking issue for PEP 600 Perennial manylinux rollout

mattip opened this issue · 18 comments

mattip commented

PEP 600 has been accepted to adopt the manylinux_2_?? standard, we need a tracking issue for implementation (I'm reusing structure and prose from gh-338, maintainers : please edit this so it makes sense).

I am pushing this forward so we can use a newer docker image for other architectures: the available arm64 compilers on CentOS 7 have a bug with copysign xref gh-494 and the glibc 2.18 versions of transcendental functions have bugs, the NumPy work-arounds are based on long double but ppc64 uses double double xref numpy/numpy#15763. Moving to an image based on glibc2.24 (debian stretch, centos8 uses 2.28) will help move the goalposts on these issues.

Client-side Support:

Publisher-side Support:

  • Update Warehouse (PyPI) to accept perennial wheel uploads (pypi/warehouse#7853)
  • Update auditwheel to emit perennial wheel archives (pypa/auditwheel#264)
  • New release of auditwheel (pypa/auditwheel#270)
  • Provide a manylinux_2_24 docker environment (#999)
    • Including a rewrite of pypa/manylinux branches so that master produces all the currrent manylinux images (thanks @mayeut)
  • Upload the new images to quay.io
  • Confirm that twine will upload perennial wheel archives

Documentation:

Additional projects to consider once core capability support has rolled out:

  • Update pipenv to install perennial wheels (pypa/pipenv#4591)
    • Needed due to pipenv vendoring pip, so the support depends on the version of pipenv being used, rather than the version of pip in the host or target environments)
  • Add an option to target manylinux_2_X in cibuildwheel (pypa/cibuildwheel#595)
  • Add an option to target manylinux_2_X in multibuild (matthew-brett/multibuild#396)
  • Distinguish between images in dockcross
  • Develop list of further relevant helper utilities and update them
mattip commented

The PR against packaging to allow PEP 600 tags (pypa/packaging#293) has been merged, so once that percolates out to pip and wheel people could, theoretically, begin producing PEP600 wheels.

  • 👍1
mattip commented

Pip has been updated, setuptools and wheel still have not.

mattip commented

A version of wheel with packaging as a dependency has been released. I guess the next step is to decide on the first PEP 600 format, create images, and adapt auditwheel to the image. I propose using glibc 2.24 via debian stretch, since that will push NumPy past these issues:

di commented

I've merged pypi/warehouse#7853 (and checked the appropriate box here).

sumanthratna commented

Hi! Are there any updates on implementing PEP 600 and is there an ETA? I imagine this will make people far less reluctant to use Alpine for Docker images.

edit: oops, at the end of the PEP I saw alpine and didn't realize that it was referring to rejected ideas :)

merwok commented

PEP 600 motivation mentions:

Non-goals include: handling non-glibc-based platforms;

so I don’t think this is related to Alpine (which is not always a good choice btw — https://pythonspeed.com/articles/alpine-docker-python/)

  • 👍2
mayeut commented

@mattip,

I was confident that packaging and pip now had support for PEP600 following your PR in packaging but it appears that 20.4 (the latest tag) is missing your PR. 14 new commits to master since 20.4, including PEP600 support. I was starting some preliminary work on auditwheel & manylinux using debian 9 (stretch) to check what could be done and how easy it could be (should be pretty easy for auditwheel, rewriting a single branch for manylinux is more challenging, I just hacked something together for now).

Error I'm getting in auditwheel when pip installing in python:3.7 container:

ERROR: sample_extension-0.1.0-cp37-cp37m-manylinux_2_24_x86_64.whl is not a supported wheel on this platform.

The image uses debian 10 and an up-to-date pip so should not cause any issue if support for PEP600 was there (which I checked on the image at first when seeing the issue):

root@6a73746b50f3:/# /lib/x86_64-linux-gnu/libc-2.28.so 
GNU C Library (Debian GLIBC 2.28-10) stable release version 2.28.
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 8.3.0.
libc ABIs: UNIQUE IFUNC ABSOLUTE
For bug reporting instructions, please see:
<http://www.debian.org/Bugs/>.
root@6a73746b50f3:/# pip3 -V
pip 20.2.4 from /usr/local/lib/python3.7/site-packages/pip (python 3.7)
root@6a73746b50f3:/# pip3 debug --verbose
WARNING: This command is only meant for debugging. Do not use this with automation for parsing and getting these details, since the output and options of this command may change without notice.
pip version: pip 20.2.4 from /usr/local/lib/python3.7/site-packages/pip (python 3.7)
sys.version: 3.7.9 (default, Oct 13 2020, 20:59:51) 
[GCC 8.3.0]
sys.executable: /usr/local/bin/python3
sys.getdefaultencoding: utf-8
sys.getfilesystemencoding: utf-8
locale.getpreferredencoding: UTF-8
sys.platform: linux
sys.implementation:
  name: cpython
'cert' config value: Not specified
REQUESTS_CA_BUNDLE: None
CURL_CA_BUNDLE: None
pip._vendor.certifi.where(): /usr/local/lib/python3.7/site-packages/pip/_vendor/certifi/cacert.pem
pip._vendor.DEBUNDLED: False
vendored library versions:
  appdirs==1.4.4
  CacheControl==0.12.6
  colorama==0.4.3
  contextlib2==0.6.0.post1 (Unable to locate actual module version, using vendor.txt specified version)
  distlib==0.3.1
  distro==1.5.0 (Unable to locate actual module version, using vendor.txt specified version)
  html5lib==1.1
  ipaddress==1.0.23
  msgpack==1.0.0 (Unable to locate actual module version, using vendor.txt specified version)
  packaging==20.4
  pep517==0.8.2
  progress==1.5
  pyparsing==2.4.7
  requests==2.24.0
  certifi==2020.06.20
  chardet==3.0.4
  idna==2.10
  urllib3==1.25.9
  resolvelib==0.4.0
  retrying==1.3.3 (Unable to locate actual module version, using vendor.txt specified version)
  setuptools==44.0.0 (Unable to locate actual module version, using vendor.txt specified version)
  six==1.15.0
  toml==0.10.1
  webencodings==0.5.1 (Unable to locate actual module version, using vendor.txt specified version)
Compatible tags: 78
  cp37-cp37m-manylinux2014_x86_64
  cp37-cp37m-manylinux2010_x86_64
  cp37-cp37m-manylinux1_x86_64
  cp37-cp37m-linux_x86_64
  cp37-abi3-manylinux2014_x86_64
  cp37-abi3-manylinux2010_x86_64
  cp37-abi3-manylinux1_x86_64
  cp37-abi3-linux_x86_64
  cp37-none-manylinux2014_x86_64
  cp37-none-manylinux2010_x86_64
  cp37-none-manylinux1_x86_64
  cp37-none-linux_x86_64
  cp36-abi3-manylinux2014_x86_64
  cp36-abi3-manylinux2010_x86_64
  cp36-abi3-manylinux1_x86_64
  cp36-abi3-linux_x86_64
  cp35-abi3-manylinux2014_x86_64
  cp35-abi3-manylinux2010_x86_64
  cp35-abi3-manylinux1_x86_64
  cp35-abi3-linux_x86_64
  cp34-abi3-manylinux2014_x86_64
  cp34-abi3-manylinux2010_x86_64
  cp34-abi3-manylinux1_x86_64
  cp34-abi3-linux_x86_64
  cp33-abi3-manylinux2014_x86_64
  cp33-abi3-manylinux2010_x86_64
  cp33-abi3-manylinux1_x86_64
  cp33-abi3-linux_x86_64
  cp32-abi3-manylinux2014_x86_64
  cp32-abi3-manylinux2010_x86_64
  cp32-abi3-manylinux1_x86_64
  cp32-abi3-linux_x86_64
  py37-none-manylinux2014_x86_64
  py37-none-manylinux2010_x86_64
  py37-none-manylinux1_x86_64
  py37-none-linux_x86_64
  py3-none-manylinux2014_x86_64
  py3-none-manylinux2010_x86_64
  py3-none-manylinux1_x86_64
  py3-none-linux_x86_64
  py36-none-manylinux2014_x86_64
  py36-none-manylinux2010_x86_64
  py36-none-manylinux1_x86_64
  py36-none-linux_x86_64
  py35-none-manylinux2014_x86_64
  py35-none-manylinux2010_x86_64
  py35-none-manylinux1_x86_64
  py35-none-linux_x86_64
  py34-none-manylinux2014_x86_64
  py34-none-manylinux2010_x86_64
  py34-none-manylinux1_x86_64
  py34-none-linux_x86_64
  py33-none-manylinux2014_x86_64
  py33-none-manylinux2010_x86_64
  py33-none-manylinux1_x86_64
  py33-none-linux_x86_64
  py32-none-manylinux2014_x86_64
  py32-none-manylinux2010_x86_64
  py32-none-manylinux1_x86_64
  py32-none-linux_x86_64
  py31-none-manylinux2014_x86_64
  py31-none-manylinux2010_x86_64
  py31-none-manylinux1_x86_64
  py31-none-linux_x86_64
  py30-none-manylinux2014_x86_64
  py30-none-manylinux2010_x86_64
  py30-none-manylinux1_x86_64
  py30-none-linux_x86_64
  cp37-none-any
  py37-none-any
  py3-none-any
  py36-none-any
  py35-none-any
  py34-none-any
  py33-none-any
  py32-none-any
  py31-none-any
  py30-none-any
root@6a73746b50f3:/# python
Python 3.7.9 (default, Oct 13 2020, 20:59:51) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from packaging import tags, __version__
>>> print(__version__)
20.4
>>> print ('\n'.join([str(t) for t in tags.sys_tags()]))
cp37-cp37m-manylinux2014_x86_64
cp37-cp37m-manylinux2010_x86_64
cp37-cp37m-manylinux1_x86_64
cp37-cp37m-linux_x86_64
cp37-abi3-manylinux2014_x86_64
cp37-abi3-manylinux2010_x86_64
cp37-abi3-manylinux1_x86_64
cp37-abi3-linux_x86_64
cp37-none-manylinux2014_x86_64
cp37-none-manylinux2010_x86_64
cp37-none-manylinux1_x86_64
cp37-none-linux_x86_64
cp36-abi3-manylinux2014_x86_64
cp36-abi3-manylinux2010_x86_64
cp36-abi3-manylinux1_x86_64
cp36-abi3-linux_x86_64
cp35-abi3-manylinux2014_x86_64
cp35-abi3-manylinux2010_x86_64
cp35-abi3-manylinux1_x86_64
cp35-abi3-linux_x86_64
cp34-abi3-manylinux2014_x86_64
cp34-abi3-manylinux2010_x86_64
cp34-abi3-manylinux1_x86_64
cp34-abi3-linux_x86_64
cp33-abi3-manylinux2014_x86_64
cp33-abi3-manylinux2010_x86_64
cp33-abi3-manylinux1_x86_64
cp33-abi3-linux_x86_64
cp32-abi3-manylinux2014_x86_64
cp32-abi3-manylinux2010_x86_64
cp32-abi3-manylinux1_x86_64
cp32-abi3-linux_x86_64
py37-none-manylinux2014_x86_64
py37-none-manylinux2010_x86_64
py37-none-manylinux1_x86_64
py37-none-linux_x86_64
py3-none-manylinux2014_x86_64
py3-none-manylinux2010_x86_64
py3-none-manylinux1_x86_64
py3-none-linux_x86_64
py36-none-manylinux2014_x86_64
py36-none-manylinux2010_x86_64
py36-none-manylinux1_x86_64
py36-none-linux_x86_64
py35-none-manylinux2014_x86_64
py35-none-manylinux2010_x86_64
py35-none-manylinux1_x86_64
py35-none-linux_x86_64
py34-none-manylinux2014_x86_64
py34-none-manylinux2010_x86_64
py34-none-manylinux1_x86_64
py34-none-linux_x86_64
py33-none-manylinux2014_x86_64
py33-none-manylinux2010_x86_64
py33-none-manylinux1_x86_64
py33-none-linux_x86_64
py32-none-manylinux2014_x86_64
py32-none-manylinux2010_x86_64
py32-none-manylinux1_x86_64
py32-none-linux_x86_64
py31-none-manylinux2014_x86_64
py31-none-manylinux2010_x86_64
py31-none-manylinux1_x86_64
py31-none-linux_x86_64
py30-none-manylinux2014_x86_64
py30-none-manylinux2010_x86_64
py30-none-manylinux1_x86_64
py30-none-linux_x86_64
py37-none-any
py3-none-any
py36-none-any
py35-none-any
py34-none-any
py33-none-any
py32-none-any
py31-none-any
py30-none-any
>>>
di commented

A new version of pypa/packaging should be released shortly: pypa/packaging#345

  • 👍2
  • 🎉1
  • 🚀1
mayeut commented

As mentioned in a previous comment, I started hacking something together to add manylinux_2_24.
I am now a bit further, rewriting scripts for a single branch. I wanted to leave behind, or even drop altogether, manylinux1 in the process and thus, I did a little digging on its usage.

While not directly on manylinux images, here are some stats on manylinux wheel production and consumption.
One thing to have in mind is that we started publication of the manylinux2010 image roughly 18 months ago.

On production side, I made some stats on all latest release of projects producing manylinux wheels, active (having at least 1 release in the past year) and for which at least 1 manylinux wheel has been downloaded over the past month. As with all stats, it might be biased by my process and/or interpretation.
I checked some metrics for projects having their latest release going back 12 months, 6 months, 3 months & 1 month.

  12 months 6 months 3 months 1 month
number of projects with manylinux wheels 2007 1543 1109 608
manylinux1 only wheels 45,4% 42,5% 40,5% 36,8%
manylinux2010 only wheels 20,0% 20,3% 21,1% 21,7%
manylinux1 and manylinux2010 wheels(optionnaly manylinux2014, probably for aarch64/ppc64le/s390x) 18,3% 18,3% 17,8% 18,3%
manylinux2014 only 11,5% 13,0% 14,2% 15,0%
manylinux2010 and manylinux2014 wheels 1,8% 2,1% 2,3% 3,1%
manylinux1 and manylinux2014 wheels 3,0% 3,8% 4,1% 5,1%

One thing to take into account is that I did not check the platform and this includes all platforms supported by manylinux. The only line giving any doubt between x86_64/i686 and the newer platforms that relates to manylinux1 is the last line. I checked the 60 projects to see if manylinux2014 was used to build x86_64/i686 or only newer platforms. It appears to be 50/50.

That leaves roughly 40% of projects still using the manylinux1 image. But why ? Let's look at the consumer side which might give part of an explanation.

This time, I did stats on downloads using pip on PyPI over the past month for all kinds of download, manylinux1 wheel only and any manylinux wheel for Linux systems running glibc:

  all downloads any manylinux wheel download only manylinux1 wheel download only manylinux1 wheel corrected
not i686/x86_64 0,92% 0,04% 0,00% 0,00%
not manylinux1 compatible 2,3% 0,4% 0,5% 0,00%
only manylinux1 compatible 1,5% 0,4% 0,5% 0,02%
uncertain N/A N/A N/A 0,06%
manylinux2010+ compatible 95,3% 99,2% 99,1% 99,92%
manylinux2010+ compatible/pip19.0+ 76,2% 81,6% 76,9% 76,9%
manylinux2014+ compatible 94,8% 98,7% 98,6% 99,4%
manylinux2014+ compatible/pip19.3+ 61,4% 64,7% 60,2% 60,2%
manylinux_2_23+ compatible 81,0% 85,4% 83,3% 84,0%
manylinux_2_23+ compatible/pip20.3 0,0% 0,0% 0,0% 0,0%
manylinux_2_24+ compatible 60,5% 65,2% 63,1% 63,1%
manylinux_2_24+ compatible/pip20.3 0,0% 0,0% 0,0% 0,0%

The first weird thing to notice is that there are quite a lot of manylinux wheel downloads for system which are reported not to be compatible with any manylinux policy. This is why there's a 4th column and an "uncertain" row. Looking at the details, the number of downloads for glibc 2.2.5, 2.7 and 2.9 seemed too high thus, I did some other stats for manylinux1 on those glibc versions but also including the distro name and version. The "uncertain" row marks distros I could not map to a specific glibc version. Most of the high download count was accounted for by Ubuntu 14.04/16.04, CentOS 6/7, debian 8/9/10 running pip 8.1.0/8.1.1/8.1.2

With all of those number, my feeling is that more than 99.96% of the systems that are manylinux1 compatible and active are also manylinux2010 compatible but, unfortunately, this doesn't mean we can drop manylinux1 just yet. It would seem that updating pip is not a common enough practice as roughly 25% of downloads are using a version of pip< 19.0 and roughly 40% older than 19.3.

I'm now thinking of integrating the outdated manylinux1 image in the single branch rewrite I'm doing (with hacks specific for that image given its age...)

mattip commented

Thanks for the deep analysis!! NumPy is planning to drop manylinux1 for the release after the next one, which means somewhere around July 2021, and I think once they do that the rest of the scientific python stack will follow suit. So if it is a mess to integrate the image into the rewrite, perhaps it is not worth it.

mayeut commented

NumPy is planning to drop manylinux1 for the release after the next one, which means somewhere around July 2021, and I think once they do that the rest of the scientific python stack will follow suit. So if it is a mess to integrate the image into the rewrite, perhaps it is not worth it.

Thanks for the information. That's very useful to have that in mind. I probably won't bother to integrate the manylinux1 image in the rewrite then. I guess manylinux2010 support will be better once numpy drops support and, hopefully, we'll be able to drop the manylinux1 image at the same time.

FYI, as numpy is the most downloaded manylinux package, I also did a quick analysis last week-end specifically on numpy >= 1.19.0downloads for the past week (not enough free quota left on BigQuery to do the full month):

- aarch64 => 2.62%
- i686 => 3.73%
    - manylinux1_i686 => 1.86%
        - glibc < 2.12 => 0,01% (my interpretation of finer grained data: majority is linked to builds on `manylinux1` image)
        - glibc >= 2.12 && pip < 19.0 => 0.53%
        - glibc >= 2.12 && pip >= 19.0 => 1.33% (I don't understand why we have this high number here)
    - manylinux2010_i686 => 1.87%
- x86_64 => 93.65%
    - manylinux1_x86_64 => 32.30%
        - glibc < 2.12 => 0.03% (my interpretation of finer grained data: majority is linked to builds on `manylinux1` image)
        - glibc >= 2.12 && pip < 19.0 => 30.91%
        - glibc >= 2.12 && pip >= 19.0 => 1.36% (I don't understand why we have this high number here)
    - manylinux2010_x86_64 => 61.36%
mayeut commented

twine can already upload a manylinux_2_24 wheel:

root@74e4524a8bbd:~/pybase64# /opt/python/cp37-cp37m/bin/twine --version
twine version 3.2.0 (pkginfo: 1.6.1, requests: 2.24.0, setuptools: 50.3.2,
requests-toolbelt: 0.9.1, tqdm: 4.51.0)
root@74e4524a8bbd:~/pybase64# /opt/python/cp37-cp37m/bin/twine upload --skip-existing -r pybase64 wheelhouse/*
Uploading distributions to https://test.pypi.org/legacy/
Uploading pybase64-1.0.2-cp37-cp37m-manylinux1_x86_64.whl
100%|██████████████████████████████████████████████████████████████████████| 123k/123k [00:01<00:00, 77.4kB/s]
  Skipping pybase64-1.0.2-cp37-cp37m-manylinux1_x86_64.whl because it appears to already exist
Uploading pybase64-1.0.2-cp37-cp37m-manylinux_2_24_x86_64.whl
100%|██████████████████████████████████████████████████████████████████████| 123k/123k [00:03<00:00, 32.9kB/s]

View at:
https://test.pypi.org/project/pybase64/1.0.2/
mattip commented

pip and wheel have released versions that vendor the PEP-600 compliant packaging version.

  • 🚀3
mayeut commented

Following @mattip comment about numpy, the first PEP-600 compliant auditwheel & manylinux are targeting glibc2.24 (debian stretch).

Are there any other short-term needs ?

If so, please do open relevant issues on auditwheel & do comment on #822 to extend manylinux images.

mayeut commented

auditwheel 3.3.0 has been released with support for manylinux_2_24 policy.

  • 🎉4
mattip commented

Edited the checklist to mention #999

mayeut commented

Edited the checklist to mention matthew-brett/multibuild#396

mayeut commented

Let's close this one. Given the shortcomings of manylinux_2_24, it probably won't get into dockcross.