Support minimal system Python version 3.6 (for venv)

Question

Support minimal system Python version 3.6 (for venv)

Closed this issue 5 years ago · 32 comments

With #27824 it is possible to create a venv over an existing Python 3.7 instead of building the python3 spkg.

This is designed to work with Python 3.7 (and < 3.8) since that's what's currently used by sage-the-distribution and is best-tested.

However, it turns out an existing Python 3.6 can work just fine as the minimum Python version with a few small fixes (sage-the-distribution used Python 3.6 until last year, so the degree to which we diverged from supporting it was minimal).

This would be a nice-to-have, as it would speed up the build on various distributions that we still support:

centos-8 (3.6.8),
fedora-26 (3.6.5, but has 3.7.0 too), https://github.com/mkoeppe/sage/runs/743789829 - clean doctests, but exceeds 6h
fedora-27 (3.6.6, but has 3.7.0 too), https://github.com/mkoeppe/sage/runs/743789883 - clean except for sympow, exceeds 6h
fedora-28 (3.6.5, but has 3.7.3 too), https://github.com/mkoeppe/sage/runs/743789896 - with gcc_spkg, build errors for gmpy2, cython, pynac
opensuse-leap-15.0 (3.6.5), -15.1 (3.6.9), -15.2 (3.6.10),
but not (because python3 is too old):
debian-stretch (oldstable) (3.5.3),
ubuntu-xenial (3.5.2)
and not (because a dependency of python3 is too old):
centos-7 (3.6.8): sqlite is too old (https://github.com/mkoeppe/sage/runs/743789990)

(All this is according to https://repology.org/project/python/versions as of 2020-06-05, not verified.)

Maybe since Sage is less tested with 3.6, there could be a warning against using it.

See also: #27754 (Upgrade: Python 3.8.x)

CC: @jhpalmieri @dimpase @kiwifb @orlitzky

Component: python3

Author: Erik Bray, Matthias Koeppe

Branch: abb5607

Reviewer: Matthias Koeppe, Michael Orlitzky

Issue created by migration from https://trac.sagemath.org/ticket/29033

Answer 1 · 2020-01-17T20:31:12.000Z

comment:2

Related to the question of what Python versions are good is #26457 - "Do not depend on a patched Python"

Answer 2 · 2020-04-14T19:41:51.000Z

comment:3

Batch modifying tickets that will likely not be ready for 9.1, based on a review of the ticket title, branch/review status, and last modification date.

Answer 3 · 2020-04-22T05:33:05.000Z

Description changed:

--- 
+++ 
@@ -1,4 +1,4 @@
-With #29032 it is possible to use an existing Python 3.7 on the system to build and run Sage.  This is designed to work with Python 3.7 (and < 3.8) since that's what's currently used by sage-the-distribution and is best-tested.
+With #27824 it is possible to use an existing Python 3.7 on the system to build and run Sage.  This is designed to work with Python 3.7 (and < 3.8) since that's what's currently used by sage-the-distribution and is best-tested.
 
 However, it turns out an existing Python 3.6 can work just fine as the minimum Python version with a few small fixes (sage-the-distribution used Python 3.6 until last year, so the degree to which we diverged from supporting it was minimal).

Answer 4 · 2020-04-22T05:33:05.000Z

Changed dependencies from #29032 to none

Answer 5 · 2020-04-22T05:33:05.000Z

comment:4

This would need rebasing

Answer 6 · 2020-05-28T06:08:22.000Z

Description changed:

--- 
+++ 
@@ -1,4 +1,6 @@
-With #27824 it is possible to use an existing Python 3.7 on the system to build and run Sage.  This is designed to work with Python 3.7 (and < 3.8) since that's what's currently used by sage-the-distribution and is best-tested.
+With #27824 it is possible to create a venv over an existing Python 3.7 instead of building the python3 spkg.  
+
+This is designed to work with Python 3.7 (and < 3.8) since that's what's currently used by sage-the-distribution and is best-tested.
 
 However, it turns out an existing Python 3.6 can work just fine as the minimum Python version with a few small fixes (sage-the-distribution used Python 3.6 until last year, so the degree to which we diverged from supporting it was minimal).

Answer 7 · 2020-05-28T06:08:22.000Z

comment:5

Edited description to clarify that this is about the venv python3, not sage-system-python.

Answer 8 · 2020-06-05T21:00:46.000Z

Description changed:

--- 
+++ 
@@ -4,4 +4,13 @@
 
 However, it turns out an existing Python 3.6 can work just fine as the minimum Python version with a few small fixes (sage-the-distribution used Python 3.6 until last year, so the degree to which we diverged from supporting it was minimal).
 
-I think this would be a nice-to-have, if not strictly necessary.  Maybe since Sage is less tested with 3.6, there could be a warning against using it.  But actually I'd be happy to keep running a Python 3.6 patchbot for a while until/unless it becomes untenable to support Python 3.6.
+This would be a nice-to-have, as it would speed up the build on various distributions that we still support: 
+- `centos-7` (3.6.8), `-8` (3.6.8), 
+- `fedora-26` (3.6.5, but has 3.7.0 too), `27` (3.6.6, but has 3.7.0 too), `-28` (3.6.5, but has 3.7.3 too), 
+- `opensuse-leap-15.0` (3.6.5), `opensuse-15.1 (3.6.9), opensuse-15.2 (3.6.10), 
+but not:
+- `debian-oldstable` (3.5.3),
+- `ubuntu-xenial` (3.5.2).
+(All this is according to https://repology.org/project/python/versions, not verified.
+
+Maybe since Sage is less tested with 3.6, there could be a warning against using it.

Answer 9 · 2020-06-05T21:01:32.000Z

Description changed:

--- 
+++ 
@@ -6,8 +6,8 @@
 
 This would be a nice-to-have, as it would speed up the build on various distributions that we still support: 
 - `centos-7` (3.6.8), `-8` (3.6.8), 
-- `fedora-26` (3.6.5, but has 3.7.0 too), `27` (3.6.6, but has 3.7.0 too), `-28` (3.6.5, but has 3.7.3 too), 
-- `opensuse-leap-15.0` (3.6.5), `opensuse-15.1 (3.6.9), opensuse-15.2 (3.6.10), 
+- `fedora-26` (3.6.5, but has 3.7.0 too), `-27` (3.6.6, but has 3.7.0 too), `-28` (3.6.5, but has 3.7.3 too), 
+- `opensuse-leap-15.0` (3.6.5), `-15.1` (3.6.9), `-15.2` (3.6.10), 
 but not:
 - `debian-oldstable` (3.5.3),
 - `ubuntu-xenial` (3.5.2).

Answer 10 · 2020-06-05T21:07:09.000Z

Changed branch from u/embray/build/configure/python3.6 to u/mkoeppe/build/configure/python3.6

Answer 11 · 2020-06-05T21:15:43.000Z

comment:9

Tests at https://github.com/mkoeppe/sage/pull/37/checks

New commits:

`abb5607`	`Allow Sage to work with a system Python 3.6.`

Answer 12 · 2020-06-05T21:15:43.000Z

Changed commit from b4a777e to abb5607

Answer 13 · 2020-06-05T21:20:52.000Z

Description changed:

--- 
+++ 
@@ -11,6 +11,6 @@
 but not:
 - `debian-oldstable` (3.5.3),
 - `ubuntu-xenial` (3.5.2).
-(All this is according to https://repology.org/project/python/versions, not verified.
+(All this is according to https://repology.org/project/python/versions as of 2020-06-05, not verified.)
 
 Maybe since Sage is less tested with 3.6, there could be a warning against using it.

Answer 14 · 2020-06-05T21:21:54.000Z

Description changed:

--- 
+++ 
@@ -14,3 +14,5 @@
 (All this is according to https://repology.org/project/python/versions as of 2020-06-05, not verified.)
 
 Maybe since Sage is less tested with 3.6, there could be a warning against using it.
+
+See also: #27754 (Upgrade: Python 3.8.x)

Answer 15 · 2020-06-06T06:14:39.000Z

Description changed:

--- 
+++ 
@@ -5,12 +5,17 @@
 However, it turns out an existing Python 3.6 can work just fine as the minimum Python version with a few small fixes (sage-the-distribution used Python 3.6 until last year, so the degree to which we diverged from supporting it was minimal).
 
 This would be a nice-to-have, as it would speed up the build on various distributions that we still support: 
-- `centos-7` (3.6.8), `-8` (3.6.8), 
-- `fedora-26` (3.6.5, but has 3.7.0 too), `-27` (3.6.6, but has 3.7.0 too), `-28` (3.6.5, but has 3.7.3 too), 
+- `centos-8` (3.6.8), 
+- `fedora-26` (3.6.5, but has 3.7.0 too), https://github.com/mkoeppe/sage/runs/743789829 - clean doctests, but exceeds 6h
+- `fedora-27` (3.6.6, but has 3.7.0 too), https://github.com/mkoeppe/sage/runs/743789883 - clean except for sympow, exceeds 6h
+- `fedora-28` (3.6.5, but has 3.7.3 too), https://github.com/mkoeppe/sage/runs/743789896 - with `gcc_spkg`, build errors for gmpy2, cython, pynac
 - `opensuse-leap-15.0` (3.6.5), `-15.1` (3.6.9), `-15.2` (3.6.10), 
-but not:
-- `debian-oldstable` (3.5.3),
-- `ubuntu-xenial` (3.5.2).
+but not (because python3 is too old):
+- `debian-stretch` (oldstable) (3.5.3),
+- `ubuntu-xenial` (3.5.2)
+and not (because a dependency of python3 is too old):
+- `centos-7` (3.6.8): `sqlite` is too old (https://github.com/mkoeppe/sage/runs/743789990)
+
 (All this is according to https://repology.org/project/python/versions as of 2020-06-05, not verified.)
 
 Maybe since Sage is less tested with 3.6, there could be a warning against using it.

Answer 16 · 2020-06-06T06:17:24.000Z

comment:14

TODO: Check fedora-28 and centos-8 in https://github.com/mkoeppe/sage/actions/runs/126407840

Answer 17 · 2020-06-06T11:43:50.000Z

comment:15

LGTM if those tests pass.

Answer 18 · 2020-06-07T03:23:02.000Z

comment:16

Tests now at https://github.com/mkoeppe/sage/actions/runs/127369767

Answer 19 · 2020-06-07T18:31:38.000Z

Reviewer: Michael Orlitzky

Answer 20 · 2020-06-07T18:36:13.000Z

comment:18

Thank you!

Answer 21 · 2020-06-07T18:36:51.000Z

Changed author from Erik Bray to Erik Bray, Matthias Koeppe

Answer 22 · 2020-06-07T18:36:51.000Z

Changed reviewer from Michael Orlitzky to Matthias Koeppe, Michael Orlitzky

Answer 23 · 2020-06-21T22:37:13.000Z

Changed branch from u/mkoeppe/build/configure/python3.6 to abb5607

Answer 24 · 2020-07-03T07:24:43.000Z

Changed commit from abb5607 to none

Answer 25 · 2020-07-03T07:24:43.000Z

comment:21

C.UTF-8 locale is a Debian-specific thing (and maybe Fedora). Please don't assume that it is available. On Arch, this causes lots (as in, one every other line in the compilation output) of warnings

/bin/sh: warning: setlocale: LC_ALL: cannot change locale (C.UTF-8)

Answer 26 · 2020-07-03T08:54:26.000Z

comment:22

on another ticket, fixing docbuild on non-EN locales, someone using Arch complained that LANG=C causes a lot of warnings.

Answer 27 · 2020-07-03T11:07:00.000Z

comment:23

Replying to @antonio-rojas:

C.UTF-8 locale is a Debian-specific thing (and maybe Fedora). Please don't assume that it is available.

Sorry about that, I didn't realize that Gentoo's glibc was patched (the patches are in an unusual place). It looks like discussion is still ongoing for glibc:

https://sourceware.org/pipermail/libc-alpha/2020-June/115224.html

Musl, on the other hand, already supports it I think?

https://wiki.musl-libc.org/functional-differences-from-glibc.html

In any case, we should probably revert to C in the meantime.

Answer 28 · 2020-07-03T13:31:56.000Z

comment:24

the locale issue is being dealt with on #30053

Answer 29 · 2020-09-10T10:08:10.000Z

comment:25

This has introduced an unfortunate bug related to a weird search order in AC_PATH_PROGS_FEATURE_CHECK(), which has the outer loop over PATH entries.
Thus, a python on the list [python.foo python.baz ....] of pythons that comes first in PATH and has all the features will be picked, not the first python on the list.

As modifying PATH to put the best python 1st might be error-prone (think about Conda, Homebrew, etc), this should be fixed.

This is of course an autoconf bug, too, as AC_PATH_PROGS() has a sane order, ie. its outer loop is over the list entries.

Answer 30 · 2020-09-10T11:09:05.000Z

comment:26

Replying to @dimpase:

This has introduced an unfortunate bug related to a weird search order in AC_PATH_PROGS_FEATURE_CHECK(), which has the outer loop over PATH entries.
Thus, a python on the list [python.foo python.baz ....] of pythons that comes first in PATH and has all the features will be picked, not the first python on the list.

As modifying PATH to put the best python 1st might be error-prone (think about Conda, Homebrew, etc), this should be fixed.

This is of course an autoconf bug, too, as AC_PATH_PROGS() has a sane order, ie. its outer loop is over the list entries.

see #30546 for a fix

Answer 31 · 2020-12-05T03:03:43.000Z

Follow up regarding centos-7:

(because a dependency of python3 is too old):

centos-7 (3.6.8): sqlite is too old (https://github.com/mkoeppe/sage/runs/743789990)

After #30559, sqlite is no longer the problem, but python3 is still not accepted on this system because of zlib -> libpng:

Checking whether SageMath should install SPKG zlib...
checking for inflateEnd in -lz... yes
checking zlib.h usability... yes
checking zlib.h presence... yes
checking for zlib.h... yes
checking for LIBPNG... no
checking for inflateValidate in -lz... no
configure: no suitable system package found for SPKG zlib
-----------------------------------------------------------------------------
Checking whether SageMath should install SPKG libpng...
checking installing zlib? ... yes; install libpng as well
configure: no suitable system package found for SPKG libpng

Answer 32 · 2020-12-05T03:32:14.000Z

comment:28

#31009 is taking care of this.