Determine whether AIX platform special case is necessary
astrofrog opened this issue · 11 comments
At the moment, the environment markers in setup.cfg include a condition based on the platform being AIX. This was copied from scipy and I think was due to an issue with the fortran extensions there (scipy/scipy#10431) - but the question is whether this is something that is likely needed for other packages? (I don't understand the original issue enough to know how widespread this issue will be)
I think this probably comes down to whether the scipy package will use this meta-package or not? If not, then we might want to not have the special-casing based on AIX?
I'd leave it as is, because almost all users will anyway install SciPy and that way the wheel build for NumPy doesn't have to be done twice (it will be cached by pip). Plus there are other packages with Fortran code.
Why not allow things to evolve? If support for AIX systems is important to certain developers, could we release AIX wheels? We do this at for conda-forge releases.
Mind you, it is a little tedious.
There's no standard for AIX wheels, so we have nowhere to put them. @aixtools is working on wheels built with xlc I believe, to be distributed somewhere AIX-specific.
Got it.
Actually, Python-3.9 will have a pep425 tag for AIX that is sufficient to support wheels.
The confusion comes from the different ways binary compatibility is resolved in, say manylinux which focuses on having wheels built on a docker image with compatible libraries and the macos way that looks at the macos versions and does some tests.
AIX has compatibility (forward-binary) built-in when using the AIX (IBM provided) libraries, aka /usr/lib/libc.a. There have been issues with the openssl libraries (/usr/lib/libssl.a and /usr/lib/libcrypto.a) - but these are watched more carefully - and fixed by IBM support - if there is an issue.
The 'open' issue - for AIX - is getting some recognition in pypa - that pip will need some AIX specific code. I was working on this a year ago - and finally got everything to work with the lint process. However, they changed the master so much that I had to start all over again (also CPython made their changes).
As long as nothing gets merged - it is just re-inventing the wheel over and over again.
FYI: adding the tag to CPython is not complex - and I add it to my packaging of Python3.6, 3.7 and 3.8. And I have a small python program that renames .whl files so that the running version of Python can take wheels for an earlier version of AIX.
I am working on an ansible on AIX primer - which shows how I use the wheels I made for ansible to setup a user-oriented virtualenv for Ansible. See http://download.aixtools.net/python/wheels/py_wheel_setup.py for how wheels get recognized and 'renamed' (pipe output to ksh).
I can, I suppose, do something similar for projects like scipy and numpy. As a reminder: Ansible needs bdist files for MarkUpSafe, PyYAML, cffi and cryptography.
As to setup.py/setup.cfg - without reading it - another issue with AIX is which (fortran) compiler are you using. I forget which is which - but one of the two prepends (and extra?) underscore ('_') to the function name.
In the 'wheels' I have built for scipy and numpy - when I was experimenting - I built my own libraries - static format - so that these routines would not try and access an incompatible external library (e.g., built with gcc while I use xlc/xlf "try and but" versions).
It is difficult to get a run-time environment right on AIX - because, imho, most people are not aware of the differences that get introduced by using different build environments - unlike other platforms that tend to have only one generally accepted tool set.
Back to wheels - I really like them because of the virtualenv support. A Python user/developer does not need to be root to install anything - and compilers are not needed. Easier to have a stable environment.
Etc..
p.s. There were a couple of comments re: getting travis CI working on AIX - I have, in all honesty, no idea what this entails - but I have time - and I don't mind a challange - as long as I have someone with knowledge I can call on with questions.
While it is an OLD server - it is my own property - so I have the freedom to blow things up - and try again - without impacting anyone else (and I can give access to anyone I trust).
Michael
Hmm, yeah, working with upstream can sometimes be a challenge. Thanks for your work.
As for how to configure travis for PPC64LE, you can have a look at how conda-forge does it
https://github.com/conda-forge/numpy-feedstock/blob/master/.travis.yml
Not sure if they had to get special permission, but you can try.
Actually, Python-3.9 will have a pep425 tag for AIX that is sufficient to support wheels.
Thanks, good to know. I'm following along with your pip/packaging issues on this topic.
getting travis CI working on AIX
I'm not sure that's something you can push forward, it seems to me that TravisCI needs access to the right hardware and make that available. On Azure it's possible to hook in custom builders, but not in TravisCI as far as I know - and anyway that's something we won't want to rely on I think (too many bad memories from the old Buildbot days).
As for how to configure travis for PPC64LE
That's completely different, you're just pointing at a recipe here. IBM worked with TravisCI to make the ppc64le platform available, and that's missing for AIX (and won't be arriving any time soon I think).
I see so ppc64le != AIX.
This discussion contains some useful info, but there's nothing actionable here - we won't change the AIX-specific constraint for Python 3.7 anymore. It was always fine I think, and either way that's long in the past now.
Thanks all!