tox.ini: Add centos-6, docker-nobootstrap, manylinux-*
Closed this issue · 35 comments
The docker-nobootstrap option uses bootstrap -D; thus, autotools are not required on the host for the build.
This enables building with centos-6 and the manylinux images based on centos-6 (https://github.com/pypa/manylinux).
Examples:
tox -e docker-manylinux-2014-standard
tox -e docker-manylinux-2014-standard-python3.9
tox -e docker-manylinux-2010-standard-i686-python3.8
Depends on #29929
Component: porting
Author: Matthias Koeppe
Branch/Commit: 14da64d
Reviewer: Jonathan Kliem
Issue created by migration from https://trac.sagemath.org/ticket/30195
Description changed:
---
+++
@@ -1 +1,13 @@
-https://github.com/pypa/manylinux
+The `docker-nobootstrap` option uses `bootstrap -D`; thus, autotools are not required on the host for the build.
+
+This enables building with `centos-6` and the `manylinux` images based on `centos-6` (https://github.com/pypa/manylinux).
+
+Examples:
+
+```
+tox -e docker-manylinux-2014-standard
+tox -e docker-manylinux-2014-standard-python3.9
+tox -e docker-manylinux-2010-standard-i686-python3.8
+```
+
+Author: Matthias Koeppe
Branch: u/mkoeppe/tox_ini__add_manylinux
Branch is 1 commit on top of #29929, which also needs review
Last 10 new commits:
25393b0 | Handle SAGE_CHECK_PACKAGES in build/make/Makefile.in, not sage-spkg |
5aca238 | Merge branch 't/30118/handle_sage_check_packages_in_build_make_makefile_in__not_sage_spkg' into t/29929/tox_ini__add_a_macos_environment_without_homebrew__conda |
a17c655 | m4/sage_spkg_collect.m4: Do not include script packages in SAGE_SDIST_PACKAGES |
827923a | bootstrap: Accept 2020s in config.guess version check |
c30ac54 | tox.ini: Add local-nobootstrap |
de6464d | .github/workflows/tox.yml: Add jobs "dist", "local-macos-nohomebrew |
775b7f7 | Trac #30088: add a few explanatory comments to m4/sage_spkg_collect.m4. |
160862f | m4/sage_spkg_collect.m4: Mention sagelib in the comment |
d1a2cbf | Merge branch 't/30088/fix__make_sdist___add_test_run_to_github_actions' into t/29929/tox_ini__add_a_macos_environment_without_homebrew__conda |
585bdac | tox.ini: Add centos-6, docker-nobootstrap, manylinux-* |
Can you provide an example of this at work?
According to the readme, only CPython up to 3.8 are provided with manylinux. Why have you configured python3.9?
Reviewer: Jonathan Kliem
The readme does not seem to be up to date. On an actual docker image:
[root@4185c8d9310b /]# /opt/python/cp3 TAB
cp35-cp35m/ cp36-cp36m/ cp37-cp37m/ cp38-cp38/ cp39-cp39/
That was one of my guesses.
Do the new environments work with github actions? If you haven't tested it yet, I have free capacities and can just test this ticket.
They should work, just add them to tox.yml for testing. I haven't tested them widely yet, only one run of docker-manylinux-2014-standard on my computer. Best to test together with #30173
File "/usr/lib/python3/dist-packages/tox/session.py", line 38, in main
config = prepare(args)
File "/usr/lib/python3/dist-packages/tox/session.py", line 26, in prepare
config = parseconfig(args)
File "/usr/lib/python3/dist-packages/tox/config.py", line 239, in parseconfig
parseini(config, inipath)
File "/usr/lib/python3/dist-packages/tox/config.py", line 760, in __init__
self.make_envconfig(name, section, reader._subs, config)
File "/usr/lib/python3/dist-packages/tox/config.py", line 791, in make_envconfig
res = meth(env_attr.name, env_attr.default)
File "/usr/lib/python3/dist-packages/tox/config.py", line 964, in getargvlist
return _ArgvlistReader.getargvlist(self, s)
File "/usr/lib/python3/dist-packages/tox/config.py", line 1134, in getargvlist
commands.append(cls.processcommand(reader, current_command))
File "/usr/lib/python3/dist-packages/tox/config.py", line 1164, in processcommand
new_word = reader._replace(word)
File "/usr/lib/python3/dist-packages/tox/config.py", line 1008, in _replace
return Replacer(self, crossonly=crossonly).do_replace(value)
File "/usr/lib/python3/dist-packages/tox/config.py", line 1028, in do_replace
return self.RE_ITEM_REF.sub(self._replace_match, x)
File "/usr/lib/python3/dist-packages/tox/config.py", line 1057, in _replace_match
return self._replace_env(match)
File "/usr/lib/python3/dist-packages/tox/config.py", line 1077, in _replace_env
(envkey, envkey))
tox.ConfigError: ConfigError: substitution env:'BOOTSTRAP': unknown environment variable 'BOOTSTRAP' or recursive definition.
##[error]Process completed with exit code 1.
https://github.com/kliem/sage/runs/897566509?check_suite_focus=true
Branch pushed to git repo; I updated commit sha1. New commits:
756f310 | tox.ini: Initialize the BOOTSTRAP variable |
Thanks for catching this
Centos 6 seems to work in the way that it fails to buil mpir.
There are many errors with manylinux like
File "/usr/lib/python3/dist-packages/tox/config.py", line 1008, in _replace
return Replacer(self, crossonly=crossonly).do_replace(value)
File "/usr/lib/python3/dist-packages/tox/config.py", line 1028, in do_replace
return self.RE_ITEM_REF.sub(self._replace_match, x)
File "/usr/lib/python3/dist-packages/tox/config.py", line 1057, in _replace_match
return self._replace_env(match)
File "/usr/lib/python3/dist-packages/tox/config.py", line 1071, in _replace_env
envvalue = self.reader.get_environ_value(envkey)
File "/usr/lib/python3/dist-packages/tox/config.py", line 903, in get_environ_value
return self._setenv.get(name)
File "/usr/lib/python3/dist-packages/tox/config.py", line 286, in get
self.resolved[name] = res = self.reader._replace(val)
File "/usr/lib/python3/dist-packages/tox/config.py", line 1008, in _replace
return Replacer(self, crossonly=crossonly).do_replace(value)
File "/usr/lib/python3/dist-packages/tox/config.py", line 1028, in do_replace
return self.RE_ITEM_REF.sub(self._replace_match, x)
File "/usr/lib/python3/dist-packages/tox/config.py", line 1057, in _replace_match
return self._replace_env(match)
File "/usr/lib/python3/dist-packages/tox/config.py", line 1077, in _replace_env
(envkey, envkey))
tox.ConfigError: ConfigError: substitution env:'BASE_IMAGE': unknown environment variable 'BASE_IMAGE' or recursive definition.
https://github.com/kliem/sage/runs/899346008?check_suite_focus=true
and
Step 14/18 : RUN ${BOOTSTRAP}
---> Running in d5c9ccb6ceb3
rm -rf config configure build/make/Makefile-auto.in
rm -f src/doc/en/installation/*.txt
rm -rf src/doc/en/reference/spkg/*.rst
rm -f src/doc/en/reference/repl/*.txt
Traceback (most recent call last):
File "/sage/build/bin/sage-download-file", line 27, in ?
from sage_bootstrap.download.cmdline import run_safe
File "/sage/build/bin/../sage_bootstrap/download/__init__.py", line 7, in ?
from sage_bootstrap.download.transfer import Download
File "/sage/build/bin/../sage_bootstrap/download/transfer.py", line 43
n = 0 if chunks_so_far == 0 else self.length // 2
^
SyntaxError: invalid syntax
Error: downloading configure-339b4599590f3dc3acec5eace4b7a82f210d3219.tar.gz failed
https://github.com/kliem/sage/runs/899346035?check_suite_focus=true
Should they be fixed here?
I'll look into the tox config error first.
Replying to @kliem:
Centos 6 seems to work in the way that it fails to buil
mpir.
Right... errors from an ancient assembler version.
tmp-mul_1.s: Assembler messages:
tmp-mul_1.s:94: Error: no such instruction: `mulx (%rsi),%r9,%r8'
tmp-mul_1.s:95: Error: no such instruction: `mulx 8(%rsi),%r11,%r10'
tmp-mul_1.s:96: Error: no such instruction: `mulx 16(%rsi),%rcx,%r12'
We'll definitely not try to support centos-6 and shouldn't be adding it to tox.yml.
Replying to @kliem:
Centos 6 seems to work in the way that it fails to buil
mpir.There are many errors with manylinux like
tox.ConfigError: ConfigError: substitution env:'BASE_IMAGE': unknown environment variable 'BASE_IMAGE' or recursive definition.
This is just caused by me not adding documentation and leaving you to guess what is supported. manylinux is not a configuration - only manylinux-1, manylinux-2010, manylinux-2014 are.
I'll make manylinux an alias for manylinux-2014 and add some documentation.
Branch pushed to git repo; I updated commit sha1. New commits:
8c13ac7 | tox.ini: Make manylinux default to manylinux-2014, add documentation |
Replying to @kliem:
File "/sage/build/bin/../sage_bootstrap/download/transfer.py", line 43 n = 0 if chunks_so_far == 0 else self.length // 2 ^ SyntaxError: invalid syntax Error: downloading configure-339b4599590f3dc3acec5eace4b7a82f210d3219.tar.gz failedhttps://github.com/kliem/sage/runs/899346035?check_suite_focus=true
The added comments also explain this one -- manylinux-1 is too old, after all, and sage cannot build on it
On manylinux-2010-minimal (https://github.com/kliem/sage/runs/899346056), building python3 fails with ./python: error while loading shared libraries: libcrypt.so.2: cannot open shared object file: No such file or directory (related to #29012?)
... but actually it was intended to use the installed python3 in /opt. Looks like I need to make adjustments for -2010.
Branch pushed to git repo; I updated commit sha1. New commits:
4c670d3 | build/pkgs/xz/spkg-configure.m4: Slightly lower the xz version bound set in #26286, for manylinux-2010 |
2f43164 | tox.ini [manylinux]: Use --with-system-python3=force |
2747c37 | tox.ini [manylinux]: Patch the downloaded configure file so that system xz is accepted |
When you have a chance, please test this version again. See above on my comments on the supported tox environments. Also note the images for non-x86 archs can't work on Linux Docker (and thus on GH Actions) because they are not multi-arch images. They work on macOS Docker Desktop because they are running inside of an emulator.
https://github.com/kliem/sage/actions/runs/180807074
That is what I'm testing for now:
tox_system_factor: [ubuntu-focal, debian-buster, centos-8, manylinux-1, manylinux-2010, manylinux-2014, manylinux-2014-python3.7]
manylinux-1 will of course fail.
I'm getting configure failures with minimal. Will this be a subject of this ticket or of another one?
Otherwise it's looking fine.
Branch pushed to git repo; I updated commit sha1. New commits:
14da64d | tox.ini [manylinux]: Do not use --with-system-python3=force for -minimal |
manylinux-2014-python3.7, minimal still fails at configuration.
https://github.com/kliem/sage/runs/908012032?check_suite_focus=true
Maybe I just got the name scheme wrong.
minimal with python3.7 cannot work because python3 dependencies such as sqlite3 are not available from the system and therefore it cannot use system python3.
https://github.com/kliem/sage/actions/runs/181590963
This seems to work well. At least according to the timings we are well in the build process. Any failures are probably subject to different tickets now.
If the run turns out to your satisfaction, you can put it on positive review on my behalf.
Thank you!
Changed branch from u/mkoeppe/tox_ini__add_manylinux to 14da64d