Upgrade python to 3.11
Closed this issue · 174 comments
Issues
Depends on #32423
Depends on #33530
Depends on #33878
Depends on #34795
CC: @kiwifb @antonio-rojas @nbruin @dimpase @jhpalmieri
Component: packages: standard
Author: Gonzalo Tornaría, Matthias Koeppe, Andrey Belgorodski
Branch/Commit: ac0105e
Reviewer: Matthias Koeppe, Dima Pasechnik
Issue created by migration from https://trac.sagemath.org/ticket/33842
It builds. First failure: building cython - we need to wait for a version with 3.11 support - https://github.com/cython/cython/commits/0.29.x
New commits:
3d3a5b7 | build/pkgs/python3: Update to 3.11.0b1 |
a2b955e | build/pkgs/python3/patches/cygwin-socket-tcpnodelay-21649.patch: Remove, upstreamed |
Description changed:
---
+++
@@ -1 +1,3 @@
+Issues
+- https://github.com/sagemath/memory_allocator/issues/6
[pyzmq-22.3.0] #include "longintrepr.h"
[pyzmq-22.3.0] ^~~~~~~~~~~~~~~
[pyzmq-22.3.0] 1 error generated.
same error as in memory_allocator
Fix is in 23.0.0 (still beta) https://pyzmq.readthedocs.io/en/latest/changelog.html
https://pypi.org/project/pyzmq/#history
cypari2:
gcc -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -g -O2 -g -O2 -I./cypari2 -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b1/lib/python3.11/site-packages/cysignals -I/usr/local/include -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b1/include/python3.11 -c cypari2/convert.c -o build/temp.macosx-12.4-x86_64-3.11/cypari2/convert.o
cypari2/convert.c:3347:21: error: cannot take the address of an rvalue of type 'Py_ssize_t' (aka 'long')
__pyx_v_sizeptr = &Py_SIZE(((PyObject *)__pyx_v_x));
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b1/var/tmp/sage/build/jupyter_jsmol-0.2.4/src/setupbase.py", line 630, in _compile_pattern
return re.compile(res, flags=flags).match
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b1/lib/python3.11/re/__init__.py", line 225, in compile
return _compile(pattern, flags)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b1/lib/python3.11/re/__init__.py", line 273, in _compile
p = _compiler.compile(pattern, flags)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b1/lib/python3.11/re/_compiler.py", line 759, in compile
p = _parser.parse(p, flags)
^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b1/lib/python3.11/re/_parser.py", line 979, in parse
p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b1/lib/python3.11/re/_parser.py", line 454, in _parse_sub
itemsappend(_parse(source, state, verbose, nested + 1,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b1/lib/python3.11/re/_parser.py", line 840, in _parse
raise source.error('global flags not at the start '
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
re.error: global flags not at the start of the expression at position 52
error: subprocess-exited-with-error
Description changed:
---
+++
@@ -1,3 +1,4 @@
Issues
- https://github.com/sagemath/memory_allocator/issues/6
+- https://github.com/sagemath/cypari2/issues/114
Description changed:
---
+++
@@ -1,4 +1,5 @@
Issues
- https://github.com/sagemath/memory_allocator/issues/6
- https://github.com/sagemath/cypari2/issues/114
+- https://github.com/fplll/fpylll/issues/230
This should be fixed in FPyLLL upstream. I can cut new packages?
Yes please!
Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
7619755 | Merge #32937 |
253a8f2 | Merge tag '9.6.rc4' into t/32423/update_numpy_to_1_22_x__scipy_1_8_x___requires_dropping_python_3_7 |
d188705 | Merge #33782 |
be053cb | Merge #32423 |
ce9a905 | build/pkgs/jupyter_jsmol: Update to 2022.1.0 |
cea777c | build/pkgs/jupyter_packaging: Make it a normal standard package |
731424b | Merge #33866 |
b75fd6e | test memory_allocator 0.1.3a1 |
542326c | Merge #33872 |
e61481e | build/pkgs/fpylll: Update to 0.5.7 |
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
da1b912 | change more version numbers for FP(y)LLL |
a623db7 | Merge #33913 |
8091a7b | update memory allocator to 0.1.3 |
49cf595 | Merge #33872 |
481d782 | test cypari update |
11a0e57 | build/bin/write-dockerfile.sh: Add bootstrap-conda |
b494326 | Merge branch 'u/mkoeppe/fix_tox_docker_builds' of git://trac.sagemath.org/sage into public/33878 |
b826c17 | Merge #33878 |
Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
49a7ca5 | Merge #33866 |
eb586f1 | build/pkgs/soupsieve: Update to 2.3.2.post1 |
b0258f3 | build/pkgs/beautifulsoup4: Update to 4.11.1 |
7580e32 | build/pkgs/nbformat: Update to 5.4.0 |
58b4329 | build/pkgs/matplotlib/dependencies: Remove pytz |
f216a8e | build/pkgs/soupsieve/dependencies: Add hatchling |
843d4b3 | build/pkgs/fastjsonschema: New, dep of nbformat |
254694c | Merge #33866 |
2a7342f | src/sage/repl/display/jsmol_iframe.py: Update extension name following version 2022.1.0 |
0b71f8b | Merge #33530 |
Branch pushed to git repo; I updated commit sha1. New commits:
f3f1b5d | test new changes to pari |
481d782 | test cypari update |
b494326 | Merge branch 'u/mkoeppe/fix_tox_docker_builds' of git://trac.sagemath.org/sage into public/33878 |
2e21858 | Merge #33878 |
d7923b7 | build/pkgs/kiwisolver: Update to 1.4.3 |
b5ceaf0 | build/pkgs/cppy: Update to 1.2.1 |
With 3.11.0b3, another problem with fpylll
gcc -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -g -O2 -g -O2 -Isrc/fpylll/fplll -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b3/lib/python3.11/site-packages/cysignals -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b3/include -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b3/lib/python3.11/site-packages/numpy/core/include -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b3/include/python3.11 -c build/src/fpylll/fplll/integer_matrix.cpp -o build/temp.macosx-12.4-x86_64-cpython-311/build/src/fpylll/fplll/integer_matrix.o -std=c++11 -g -O2
In file included from build/src/fpylll/fplll/integer_matrix.cpp:47:
In file included from /Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b3/include/python3.11/Python.h:38:
/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b3/include/python3.11/pyport.h:47:24: error: cannot cast from type 'int' to pointer type '_object *'
return static_cast<type>(const_cast<expr_type &>(expr));
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
build/src/fpylll/fplll/integer_matrix.cpp:22142:9: note: in instantiation of function template specialization '(anonymous namespace)::_Py_CAST_impl<_object *, int>' requested here
PyTuple_SET_ITEM(args, 0, 0);
^
/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b3/include/python3.11/cpython/tupleobject.h:40:49: note: expanded from macro 'PyTuple_SET_ITEM'
PyTuple_SET_ITEM(_PyObject_CAST(op), index, _PyObject_CAST(value))
^
/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b3/include/python3.11/object.h:107:28: note: expanded from macro '_PyObject_CAST'
#define _PyObject_CAST(op) _Py_CAST(PyObject*, (op))
^
/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0b3/include/python3.11/pyport.h:51:32: note: expanded from macro '_Py_CAST'
# define _Py_CAST(type, expr) _Py_CAST_impl<type>(expr)
^
1 error generated.
error: command '/usr/bin/gcc' failed with exit code 1
Replying to @mkoeppe:
With 3.11.0b3, another problem with fpylll
This problem has disappeared with 3.11.0b4
Branch pushed to git repo; I updated commit sha1. New commits:
74221f2 | build/pkgs/python3: Update to 3.11.0b5 |
Branch pushed to git repo; I updated commit sha1. New commits:
4366cf2 | Merge tag '9.8.beta1' into t/33842/test_ticket__python_3_11 |
Patch for python-3.11 (see python-3.11.patch at void-linux/void-packages@6229f31):
--- a/src/sage/cpython/cython_metaclass.h
+++ b/src/sage/cpython/cython_metaclass.h
@@ -66,7 +66,7 @@
}
/* Now, set t.__class__ to metaclass */
- Py_TYPE(t) = metaclass;
+ Py_SET_TYPE(t, metaclass);
PyType_Modified(t);
}
else
--- a/src/sage/symbolic/ginac/numeric.cpp
+++ b/src/sage/symbolic/ginac/numeric.cpp
@@ -52,7 +52,6 @@
#define register
#define PY_SSIZE_T_CLEAN
#include <Python.h>
-#include <longintrepr.h>
#include "flint/fmpz.h"
#include "flint/fmpz_factor.h"
--- a/src/sage/libs/gmp/pylong.pyx
+++ b/src/sage/libs/gmp/pylong.pyx
@@ -32,7 +32,7 @@
from .mpz cimport *
cdef extern from *:
- Py_ssize_t* Py_SIZE_PTR "&Py_SIZE"(object)
+ void Py_SET_SIZE(object, Py_ssize_t)
int hash_bits """
#ifdef _PyHASH_BITS
_PyHASH_BITS /* Python 3 */
@@ -57,10 +57,8 @@
mpz_export(L.ob_digit, NULL,
-1, sizeof(digit), 0, PyLong_nails, z)
if mpz_sgn(z) < 0:
- # Set correct size (use a pointer to hack around Cython's
- # non-support for lvalues).
- sizeptr = Py_SIZE_PTR(L)
- sizeptr[0] = -pylong_size
+ # Set correct size
+ Py_SET_SIZE(L, -pylong_size)
return L
I think we need to conditionalize this - Py_SET_TYPE, Py_SET_SIZE were added in 3.9, but we still support 3.8
Branch pushed to git repo; I updated commit sha1. New commits:
99de8a3 | Merge tag '9.8.beta2' into t/33842/test_ticket__python_3_11 |
9042e6b | build/pkgs/python3: Update to 3.11.0 |
a379ea3 | build/pkgs/cypari: Update to 2.1.3 |
34c5c94 | build/pkgs/cypari/patches/trashcan.patch: Remove |
de38bac | src/sage: Apply python-3.11.patch from https://github.com/void-linux/void-packages/commit/6229f313450ecae88743b4d5e99da2ed4de44e07 |
2df72df | build/pkgs/cython/patches/trashcan.patch: Remove |
Next error (on startup):
File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/src/sage/misc/decorators.py", line 35, in <module>
from inspect import ArgSpec
ImportError: cannot import name 'ArgSpec' from 'inspect' (/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/local/var/lib/sage/venv-python3.11.0/lib/python3.11/inspect.py)
Also
File "sage/misc/cachefunc.pyx", line 2821, in sage.misc.cachefunc.CachedMethod.__get__
File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-py311/src/sage/misc/sageinspect.py", line 1720, in sage_getargspec
return inspect.ArgSpec(*_sage_getargspec_cython(source))
^^^^^^^^^^^^^^^
AttributeError: module 'inspect' has no attribute 'ArgSpec'
From https://docs.python.org/3.11/whatsnew/3.11.html:
Removed from the inspect module:
-
The getargspec() function, deprecated since Python 3.0; use inspect.signature() or inspect.getfullargspec() instead.
-
The formatargspec() function, deprecated since Python 3.5; use the inspect.signature() function or the inspect.Signature object directly.
Branch pushed to git repo; I updated commit sha1. New commits:
9eb08f3 | inspect.ArgSpec -> inspect.FullArgSpec |
Branch pushed to git repo; I updated commit sha1. New commits:
69db932 | pkgs/*/setup.cfg: Update python versions |
Hangs in this doctest:
Trying (line 1258): @weak_cached_function(cache=0)
def f():
print("doing a computation")
return A()
Expecting nothing
ok [0.00 s]
Trying (line 1262): a = f()
Expecting:
doing a computation
ok [0.00 s]
Trying (line 1267): b = f()
Expecting nothing
ok [0.00 s]
Trying (line 1268): a is b
Expecting:
True
ok [0.00 s]
Trying (line 1274): del a
Expecting nothing
ok [0.00 s]
Trying (line 1275): del b
Expecting nothing
^CKilling test pkgs/sagemath-categories/.tox/sagepython-sagewheels-nopypi-norequirements/lib/python3.11/site-packages/sage/misc/cachefunc.pyx
Branch pushed to git repo; I updated commit sha1. New commits:
9e590b7 | src/sage/features/__init__.py: Guard import sage.misc.cython with try...except |
This likely comes from our weak dictionaries. Running sage -t --verbose src/sage/misc/weak_dict.pyx gives:
Trying (line 53): import sage.misc.weak_dict
Expecting nothing
ok [0.00 s]
Trying (line 54): D = sage.misc.weak_dict.WeakValueDictionary()
Expecting nothing
ok [0.00 s]
Trying (line 55): for v in ValList:
D[Keys(v)] = v
Expecting nothing
ok [0.00 s]
Trying (line 57): len(D)
Expecting:
10
ok [0.00 s]
Trying (line 59): del ValList
Expecting nothing
Killed due to segmentation fault
The layout of dictionaries has changed from 3.10 to 3.11, so src/sage/cpython/dict_del_by_value.pyx will have to be changed
Should build/pkgs/cypari/patches/trashcan.patch really be fully removed? Upstream Cython applied a patch (cython/cython@e337825) which replaces the change in Cython/Utility/ExtensionTypes.c by the upstream trashcan support, but keeps the Cython/Compiler/* trashcan uses added by the original patch untouched.
That sounds like a better solution, please feel free to push to the branch
Shouldn't we make this change in cypari2 itself?
Replying to @sagetrac-git:
Branch pushed to git repo; I updated commit sha1. New commits:
a379ea3 | build/pkgs/cypari: Update to 2.1.3 |
Where is cypari 2.1.3? I can't seem to find it. Although the version in the git repo was bumped to 2.1.3 in oct 3, there is no tag. And https://pypi.io/packages/source/c/cypari2/cypari2-2.1.3.tar.gz (which, if IIUC, is the URL after this patch) gives 404.
I built sagemath with python 3.11 (using the patch I posted earlier + the patch to fix inspect.argspec). Running sage hangs forever, I'm guessing I have to update cypari2 but I can't find the tarball for 2.1.3.
I pushed to sage-on-gentoo master for 9.8.beta3 with a patch for python 3.11 tailored from this branch. This was a bit of a mistake as it broke doc building with python 3.10 (and may be more).
cschwan/sage-on-gentoo#724
What I learned today is that the current branch is not backward compatible :(
Actually, I said it was breaking doc building with 3.10, but has anyone build the doc with python 3.11? I may be jumping the gun on backward compatibility, it may just be broken :(
See comment:35 - current changes here on the branch need to be conditionalized on python >= 3.11
And comment:48 is still open - this requires major work
Replying to Gonzalo Tornaría:
Replying to @sagetrac-git:
Branch pushed to git repo; I updated commit sha1. New commits:
a379ea3 build/pkgs/cypari: Update to 2.1.3Where is cypari 2.1.3?
I built it from the master branch of cypari2 but did not upload it to PyPI because I haven't tested it yet
Replying to Matthias Köppe:
And comment:48 is still open - this requires major work
OK, I think this is stuff that breaks the doc building. I think we can infer that from the error messages.
I upgraded cypari to 2.1.3 and fixed running tests (see #34537 comment:79).
But I can't even get sage started at all... What am I missing?
Is there a patch to cysignals that I should be applying (I'm using 1.11.2).
On startup I get a backtrace that starts with
------------------------------------------------------------------------
/usr/lib/python3.11/site-packages/cysignals/signals.so(+0x8170)[0x7fc24132e170]
/usr/lib/python3.11/site-packages/cysignals/signals.so(+0x8228)[0x7fc24132e228]
/usr/lib/python3.11/site-packages/cysignals/signals.so(+0xaa33)[0x7fc241330a33]
/usr/lib/libc.so.6(+0x3d000)[0x7fc241ec5000]
/builddir/sage-9.7/pkgs/sagemath-standard/build/lib.linux-x86_64-cpython-311/sage/cpython/dict_del_by_value.cpython-311-x86_64-linux-gnu.so(+0x429d)[0x7fc24040e29d]
/builddir/sage-9.7/pkgs/sagemath-standard/build/lib.linux-x86_64-cpython-311/sage/misc/weak_dict.cpython-311-x86_64-linux-gnu.so(+0xc815)[0x7fc24041f815]
So this seems to be comment:48. Somehow I thought you had sage running and only some tests would break...
Replying to Gonzalo Tornaría:
So this seems to be comment:48. Somehow I thought you had sage running and only some tests would break...
No, it does not start. I was testing using sagemath-categories, which loads much less stuff at startup; this helped isolate the problem.
Just to have a minimal test case, on python 3.10:
$ python -c 'from sage.cpython.dict_del_by_value import test_del_dictitem_by_exact_value ; D = {1:2} ; test_del_dictitem_by_exact_value(D,1,2); print(D); test_del_dictitem_by_exact_value(D,2,1); print(D)'
{1: 2}
{}
This hangs with python 3.11.
Note also that dict_del_by_value.pyx is completely independent of the rest of sage. In case this helps with logistics (e.g. I have python 3.11 in a particular chroot, and I've recompiled just the python packages )
Maybe it's easy to have a "stupid slow" implementation of del_dictitem_by_exact_value to keep moving forward?
Just to get going, this seems to do something (possibly a very dumb thing to do, but at least sage starts, I'm getting lots of doctest failures now):
diff --git a/src/sage/misc/weak_dict.pyx b/src/sage/misc/weak_dict.pyx
index d52bedacfa4..930aa31bd1b 100644
--- a/src/sage/misc/weak_dict.pyx
+++ b/src/sage/misc/weak_dict.pyx
@@ -216,7 +216,7 @@ cdef class WeakValueDictEraser:
if D._guard_level:
D._pending_removals.append(r)
else:
- del_dictitem_by_exact_value(<PyDictObject *>D, <PyObject *>r, r.key)
+ pass
cdef class WeakValueDictionary(dict):
This seems to get rid of a lot of deprecation warnings:
diff --git a/src/sage/rings/polynomial/pbori/gbrefs.py b/src/sage/rings/polynomial/pbori/gbrefs.py
index 76e3924715d..70dc795cbab 100644
--- a/src/sage/rings/polynomial/pbori/gbrefs.py
+++ b/src/sage/rings/polynomial/pbori/gbrefs.py
@@ -1,6 +1,6 @@
import gzip
from io import StringIO
-import uu
+import base64 as uu
import re
from types import ModuleType
from .PyPolyBoRi import Polynomial
diff --git a/src/sage/misc/fpickle.pyx b/src/sage/misc/fpickle.pyx
index 502080e2c10..0532e5d2d38 100644
--- a/src/sage/misc/fpickle.pyx
+++ b/src/sage/misc/fpickle.pyx
@@ -44,7 +44,12 @@ def reduce_code(co):
co_args += (co.co_kwonlyargcount, co.co_nlocals,
co.co_stacksize, co.co_flags, co.co_code,
co.co_consts, co.co_names, co.co_varnames, co.co_filename,
- co.co_name, co.co_firstlineno, co.co_lnotab)
+ co.co_name)
+ if sys.version_info.minor >= 11:
+ co_args += (co.co_qualname,)
+ co_args += (co.co_firstlineno, co.co_lnotab)
+ if sys.version_info.minor >= 11:
+ co_args += (co.co_exceptiontable,)
return (code_ctor, co_args)
This is a mess... it's not even clear where the arguments to type.CodeType are documented. I found them by running type.CodeType? on sage.
--- a/src/sage/all.py
+++ b/src/sage/all.py
@@ -104,6 +104,10 @@ warnings.filterwarnings('ignore', category=DeprecationWarning,
message='The distutils(.sysconfig module| package) is deprecated',
module='Cython|distutils|numpy|sage.env|sage.features')
+warnings.filterwarnings('ignore', category=DeprecationWarning,
+ message="'cgi' is deprecated and slated for removal in Python 3.13",
+ module='Cython')
+
################ end setup warnings ###############################
diff --git a/src/sage/misc/sageinspect.py b/src/sage/misc/sageinspect.py
index c753f08232b..619ff6da661 100644
--- a/src/sage/misc/sageinspect.py
+++ b/src/sage/misc/sageinspect.py
@@ -1133,7 +1133,7 @@ def _sage_getargspec_from_ast(source):
return inspect.FullArgSpec(args, vararg, kwarg,
tuple(defaults) if defaults else None,
- kwonlyargs=[], kwonlydefaults={}, annotations={})
+ kwonlyargs=[], kwonlydefaults=None, annotations={})
def _sage_getargspec_cython(source):
@@ -1679,7 +1679,8 @@ def sage_getargspec(obj):
# Note that this may give a wrong result for the constants!
try:
args, varargs, varkw = inspect.getargs(obj.__code__)
- return inspect.FullArgSpec(args, varargs, varkw, obj.__defaults__)
+ return inspect.FullArgSpec(args, varargs, varkw, obj.__defaults__,
+ kwonlyargs=[], kwonlydefaults=None, annotations={})
except (TypeError, AttributeError):
pass
if isclassinstance(obj):
@@ -1738,7 +1739,7 @@ def sage_getargspec(obj):
except AttributeError:
defaults = None
return inspect.FullArgSpec(args, varargs, varkw, defaults,
- kwonlyargs=[], kwonlydefaults={}, annotations={})
+ kwonlyargs=[], kwonlydefaults=None, annotations={})
def formatannotation(annotation, base_module=None):
@@ -1784,7 +1785,7 @@ def formatannotation(annotation, base_module=None):
def sage_formatargspec(args, varargs=None, varkw=None, defaults=None,
- kwonlyargs=(), kwonlydefaults={}, annotations={},
+ kwonlyargs=(), kwonlydefaults=None, annotations={},
formatarg=str,
formatvarargs=lambda name: '*' + name,
formatvarkw=lambda name: '**' + name,
plus lots of doctest fixing in sageinspect.py which I'm not pasting here.
I pushed all my patches in the branch u/tornaria/py311:
$ git log1 trac/u/mkoeppe/test_ticket__python_3_11..trac/u/tornaria/py311
6010f52588e (HEAD -> py311, trac/u/tornaria/py311) fix argspec -> fullargspec
21762fa3bc6 warnings: ignore deprecation of importlib.resources.path/read_binary
98598ba0aa3 doctest: message added more info in python 3.11
07acbc4509f doctest: AssertionError message changed in python 3.11
2a4ced7e1a1 changed argspec.keywords -> argspec.varkw
b09c738c6ed doctests: fix more due to ArgSpec -> FullArgSpec
a65553aae06 warnings: ignore deprecation for 'import sre_constants' in pyparsing
f91d711b3c2 Fix FullArgSpec calls after 9eb08f3afde3266bbd667e196513240a0fe245f4
14c91ec77de doctest: fix in sage.misc.sageinspect due to ArgSpec -> FullArgSpec
186d921e70c warnings: ignore deprecation for 'import cgi' in cython
0d34a14717c fix fpickle for python 3.11
a0078320c55 deprecated uu -> base64
bed2c03e6d5 don't use broken del_dictitem_by_exact_value (temporary?)
Only a a few remaining doctest failures, most (all?) seem related to the hack I did to get through the WeakValueDictionary problem.
If anyone can think of a good workaround (or a proper fix) for WeakValueDictionary... It seems void linux wants to upgrade python to 3.11 next week so we would get stuck with a broken sagemath.
Test failures remaining:
sage/structure/coerce_dict.pyx # 1 doctest failed
sage/modules/free_quadratic_module_integer_symmetric.py # 1 doctest failed
sage/misc/weak_dict.pyx # 11 doctests failed
sage/cpython/dict_del_by_value.pyx # Timed out
sage/symbolic/function.pyx # RuntimeError in doctesting framework
The first four look related to the weak dict issue. The last one, I'm not sure.
Replying to Matthias Köppe:
I think we need to conditionalize this -
Py_SET_TYPE,Py_SET_SIZEwere added in 3.9, but we still support 3.8
Is it really necessary to support 3.8? It seems the way deprecation/removal works in python is meant to support 3 versions (e.g. new feature added in 3.9, old feature removed in 3.11). There is a similar cadence with the deprecation importlib.resources.{path,read_binary}: these are deprecated in 3.11, but the alternative api (importlib.resources.files) was introduced in 3.9, so we can't rewrite the code now if we have to support 3.8.
As a point of reference, debian stable ships 3.9.
Replying to Gonzalo Tornaría:
Replying to Matthias Köppe:
I think we need to conditionalize this -
Py_SET_TYPE,Py_SET_SIZEwere added in 3.9, but we still support 3.8Is it really necessary to support 3.8?
Yes, I think so. Even NEP 29 only drops 3.8 on Apr 14, 2023. https://numpy.org/neps/nep-0029-deprecation_policy.html#support-table
For importlib.resources, a backport package is available.
Replying to Matthias Köppe:
Replying to Gonzalo Tornaría:
Replying to Matthias Köppe:
I think we need to conditionalize this -
Py_SET_TYPE,Py_SET_SIZEwere added in 3.9, but we still support 3.8Is it really necessary to support 3.8?
Yes, I think so. Even NEP 29 only drops 3.8 on Apr 14, 2023. https://numpy.org/neps/nep-0029-deprecation_policy.html#support-table
But Sage ships a vendored python so dropping support for python 3.8 does not make it unusable. I'd guess in a system with python 3.8, other python packages required by sage will also be outdated so the difference between using system python or the vendored in one is not that big.
Report on the remaining doctest failures:
- Obviously caused by the change dicts in python 3.11:
sage/cpython/dict_del_by_value.pyx # Timed out`
- These are caused by "the hack" replacing calls to
del_dictitem_by_exact_valuebypass:
sage/structure/coerce_dict.pyx # 1 doctest failed
sage/modules/free_quadratic_module_integer_symmetric.py # 1 doctest failed
sage/misc/weak_dict.pyx # 11 doctests failed
In fact, I compiled my u/tornaria/py311 branch using python 3.10 (so, with "the hack"), and I get the exact same failures (meaning all my changes are compatible at least with python 3.10).
- A failure that is almost surely unrelated:
sage/symbolic/function.pyx # RuntimeError in doctesting framework
This failure does not happen with python 3.10 (even with "the hack"). I will comment more about this.
About the last failure, here's a one-liner that causes the runtime error:
sage: loads(dumps(function("foo", nargs=0, eval_func=lambda self:0)))()
Unexpected exception formatting exception. Falling back to standard exception
Unexpected exception formatting exception. Falling back to standard exception
Unexpected exception formatting exception. Falling back to standard exception
Error in sys.excepthook:
Traceback (most recent call last):
File "/usr/lib/python3.11/pathlib.py", line 1250, in is_dir
return S_ISDIR(self.stat().st_mode)
^^^^^^^^^
AttributeError: 'str' object has no attribute 'stat'
Original exception was:
Traceback (most recent call last):
File "/usr/lib/python3.11/site-packages/IPython/core/interactiveshell.py", line 3378, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-1-5d60a76861b0>", line 1, in <module>
File "sage/symbolic/function.pyx", line 547, in sage.symbolic.function.Function.__call__ (build/cythonized/sage/symbolic/function.c:6279)
File "sage/symbolic/pynac_function_impl.pxi", line 1, in sage.symbolic.expression.call_registered_function (build/cythonized/sage/symbolic/expression.cpp:111832)
File "sage/symbolic/pynac_function_impl.pxi", line 47, in sage.symbolic.expression.call_registered_function (build/cythonized/sage/symbolic/expression.cpp:111428)
File "<ipython-input-1-5d60a76861b0>", line -1, in <lambda>
NameError: name 'Integer' is not defined
During handling of the above exception, another exception occurred:
...
If anybody has a clue... please help.
In case it's useful, here is another failure, may be caused by the same bug, that does not cause a runtime error:
sage: foo = function("foo", nargs=0, eval_func=lambda self:0)
sage: bar = loads(dumps(foo))
sage: bar == foo
False
Of course, the runtime error is there, lurking, if you just do bar() you get the same runtime error as above. I tried to follow the code for bar() but there's so many indirection layers that my head overflowed in the attempt.
I pushed a new commit to my branch u/tornaria/py311 which fixes the issue from the previous comment:
8934024438e (trac/u/tornaria/py311) (corrected) fix fpickle for python 3.11
It turns out my previous fix for fpickle was incorrect (I missed that co_lnotab had to be changed to co_linetable).
I added a test that will (hopefully) catch other mistakes like this.
Now all that remains (it seems) is for someone to reimplement del_dictitem_by_exact_value(...). Maybe there's a python way to acomplish the same (delete an item in a dict) even if it's a bit slower...
My hack (using pass) seems to behave ok, other than causing memory leaks. I'd happily trade a small slowdown for fixing this memory leaks. I wonder if there are any benchmarks that show how bad this really is...
This seems to work:
diff --git a/src/sage/misc/weak_dict.pyx b/src/sage/misc/weak_dict.pyx
index 930aa31bd1b..724f57f421c 100644
--- a/src/sage/misc/weak_dict.pyx
+++ b/src/sage/misc/weak_dict.pyx
@@ -139,6 +139,10 @@ cdef extern from "Python.h":
PyObject* PyWeakref_GetObject(PyObject *ref)
int PyTuple_SetItem(PyObject *op, Py_ssize_t i, PyObject *newitem) except -1
+ int _PyDict_DelItem_KnownHash(PyObject *mp, PyObject *key,
+ Py_hash_t hash) except -1
+ PyObject* _PyDict_GetItem_KnownHash(PyObject *mp, PyObject *key,
+ Py_hash_t hash)
cdef class WeakValueDictEraser:
"""
@@ -212,11 +216,16 @@ cdef class WeakValueDictEraser:
return
# The situation is the following:
# in the underlying dictionary, we have stored a KeyedRef r
- # under a key k. The attribute r.key is the hash of k.
+ # under a key k. The attribute r.key is (k, hash(k))
if D._guard_level:
D._pending_removals.append(r)
else:
- pass
+ if _PyDict_GetItem_KnownHash(<PyObject *>D,
+ <PyObject *>r.key[0],
+ r.key[1]) == <PyObject *>r:
+ _PyDict_DelItem_KnownHash(<PyObject *>D,
+ <PyObject *>r.key[0],
+ r.key[1])
cdef class WeakValueDictionary(dict):
@@ -548,7 +557,7 @@ cdef class WeakValueDictionary(dict):
add a weak reference to ``v`` under the key ``k`` in the actual
dict underlying ``self``.
"""
- PyDict_SetItem(self, k, KeyedRef(v, self.callback, hash(k)))
+ PyDict_SetItem(self, k, KeyedRef(v, self.callback, (k, hash(k))))
# def __delitem__(self, k):
# we do not really have to override this method.
There's a double lookup, but it doesn't seem possible to avoid it using python API. But this seems to work as expected.
All tests pass on python 3.10 and 3.11. Can someone test 3.8 and 3.9? This is all pushed to my branch.
Note: I didn't touch src/sage/cpython/dict_del_by_value.pyx so expect (and ignore) a doctest timeout for that file on python 3.11.
Just as a heads up, with the last version all doctests pass for me (both python 3.10 and 3.11) except for the following ocassional failure:
**********************************************************************
File "/builddir/sage-9.7/pkgs/sagemath-standard/build/lib.linux-x86_64-cpython-311/sage/rings/finite_rings/residue_field.pyx", line 732, in sage.rings.finite_rings.residue_field.ResidueField_generic.lift_map
Failed example:
R = QQ[3^(1/3)]
Expected nothing
Got:
Exception ignored in: <sage.structure.coerce_dict.TripleDictEraser object at 0x7fa2fc27a050>
Traceback (most recent call last):
File "sage/structure/coerce_dict.pyx", line 979, in sage.structure.coerce_dict.TripleDictEraser.__call__ (build/cythonized/sage/structure/coerce_dict.c:6393)
AssertionError: TripleDictEraser: key match but no weakref match
**********************************************************************
This only happens when the change of the previous comment is applied.
It looks like some kind of race condition. No clue. Someone please help/comment (Nils?).
Here is a current summary of my branch u/tornaria/py311:
$ git log1 --first-parent trac/u/mkoeppe/test_ticket__python_3_11..trac/u/tornaria/py311
7b6fa565f42 (trac/u/tornaria/py311) doctests: message added more info in python 3.11
482dd1ac3d2 doctests: AssertionError message changed in python 3.11
44480f4827e doctests: fixes due to ArgSpec -> FullArgSpec change
08e1161c23c warnings: ignore deprecation of importlib.resources.path/read_binary
664fc008ed5 warnings: ignore deprecation for 'import sre_constants' in pyparsing
db45aebfd6b warnings: ignore deprecation for 'import cgi' in cython
8b0dac2322d Fix FullArgSpec usage after 9eb08f3afde3266bbd667e196513240a0fe245f4
dc8e155994a sage.misc.fpickle: fix for python 3.11
014c2ac9a6f deprecated uu -> base64
76040803c8a dict_del_by_value: add internal definitions for python 3.11
8955607c71c dict_del_by_value: move python internal definitions to a separate file
274230f72fa Merge tag '9.8.beta3' into u/mkoeppe/test_ticket__python_3_11
With this, I have all tests passing, both python 3.10 (tested on this branch) and 3.11 (tested on sage 9.7).
One exception: there is one failure with python 3.10 in src/sage/repl/attach.py due to the last commit (7b6fa56). I used ... to represent "0 or more random lines", but it seems it only represents "1 or more random lines".
In case it is useful, the changes I am applying to 9.7 are all commits listed above (except the merge) plus:
9eb08f3afde inspect.ArgSpec -> inspect.FullArgSpec
de38bac21e2 src/sage: Apply python-3.11.patch ...
Replying to Matthias Köppe:
I think we need to conditionalize this -
Py_SET_TYPE,Py_SET_SIZEwere added in 3.9, but we still support 3.8
This is what Doc/whatsnew/3.11.rst (python source) has to say about it:
* Since :c:func:`Py_TYPE()` is changed to a inline static function,
``Py_TYPE(obj) = new_type`` must be replaced with
``Py_SET_TYPE(obj, new_type)``: see the :c:func:`Py_SET_TYPE()` function
(available since Python 3.9). For backward compatibility, this macro can be
used::
#if PY_VERSION_HEX < 0x030900A4 && !defined(Py_SET_TYPE)
static inline void _Py_SET_TYPE(PyObject *ob, PyTypeObject *type)
{ ob->ob_type = type; }
#define Py_SET_TYPE(ob, type) _Py_SET_TYPE((PyObject*)(ob), type)
#endif
(Contributed by Victor Stinner in :issue:`39573`.)
* Since :c:func:`Py_SIZE()` is changed to a inline static function,
``Py_SIZE(obj) = new_size`` must be replaced with
``Py_SET_SIZE(obj, new_size)``: see the :c:func:`Py_SET_SIZE()` function
(available since Python 3.9). For backward compatibility, this macro can be
used::
#if PY_VERSION_HEX < 0x030900A4 && !defined(Py_SET_SIZE)
static inline void _Py_SET_SIZE(PyVarObject *ob, Py_ssize_t size)
{ ob->ob_size = size; }
#define Py_SET_SIZE(ob, size) _Py_SET_SIZE((PyVarObject*)(ob), size)
#endif
(Contributed by Victor Stinner in :issue:`39573`.)
Replying to Gonzalo Tornaría:
Just as a heads up, with the last version all doctests pass for me (both python 3.10 and 3.11) except for the following ocassional failure:
********************************************************************** ... AssertionError: TripleDictEraser: key match but no weakref match **********************************************************************This only happens when the change of the previous comment is applied.
It looks like some kind of race condition. No clue. Someone please help/comment (Nils?).
A quick word about this. It seems PyDict_GetItem* and friends will be calling PyObject_RichCompareBool to compare keys, for some reason this is what breaks the TripleDictEraser (I guess when it happens inside a erase callback).
Anyway, I bite the bullet and refactored del_dictitem_by_exact_value() in a way that the "internal" python dict definitions needed are all in an external .h file copied verbatim from python sources. This makes it easier to port it in case of changes in python.
76040803c8a dict_del_by_value: add internal definitions for python 3.11
8955607c71c dict_del_by_value: move python internal definitions to a separate file
Here the first commit 8955607 does the refactor using definitions copied from python 3.8 and should be functionally noop (still working for python <= 3.10, still broken for python 3.11). The second commit 7604080 only changes the external .h file to (conditionally) use definitions copied from python 3.11.
Changed branch from u/mkoeppe/test_ticket__python_3_11 to u/tornaria/py311
Last 10 new commits:
7604080 | dict_del_by_value: add internal definitions for python 3.11 |
014c2ac | deprecated uu -> base64 |
dc8e155 | sage.misc.fpickle: fix for python 3.11 |
8b0dac2 | Fix FullArgSpec usage after 9eb08f3afde3266bbd667e196513240a0fe245f4 |
db45aeb | warnings: ignore deprecation for 'import cgi' in cython |
664fc00 | warnings: ignore deprecation for 'import sre_constants' in pyparsing |
08e1161 | warnings: ignore deprecation of importlib.resources.path/read_binary |
44480f4 | doctests: fixes due to ArgSpec -> FullArgSpec change |
482dd1a | doctests: AssertionError message changed in python 3.11 |
7b6fa56 | doctests: message added more info in python 3.11 |
For the record: on void linux we are shipping sagemath 9.7 with python 3.11 (and pari 2.15, gap 4.12) everything seems in working order (x86_64, x86_64-musl, i686, as usual; unfortunately we don't have native builders for aarch64 and some dependencies of sagemath don't cross build).
In case it is useful, our patches:
https://github.com/void-linux/void-packages/tree/master/srcpkgs/sagemath/patches
All of them are in trac: #34669 #33360 #33446 #33842 #34118 #34391 #34465 #34537 #34668.
As per comment:80, something like the following should restore support for python 3.8, but I have not tested it.
diff --git a/src/sage/cpython/cython_metaclass.h b/src/sage/cpython/cython_metaclass.h
index 6487342b71e..da06ab75a6b 100644
--- a/src/sage/cpython/cython_metaclass.h
+++ b/src/sage/cpython/cython_metaclass.h
@@ -8,6 +8,13 @@
* http://www.gnu.org/licenses/
*****************************************************************************/
+/* Compatibility for python 3.8, can be removed later */
+#if PY_VERSION_HEX < 0x030900A4 && !defined(Py_SET_TYPE)
+static inline void _Py_SET_TYPE(PyObject *ob, PyTypeObject *type)
+{ ob->ob_type = type; }
+#define Py_SET_TYPE(ob, type) _Py_SET_TYPE((PyObject*)(ob), type)
+#endif
+
/* Tuple (None, None, None), initialized as needed */
static PyObject* NoneNoneNone;
diff --git a/src/sage/libs/gmp/pylong.pyx b/src/sage/libs/gmp/pylong.pyx
index e772b60e3e0..2f7ed35be9d 100644
--- a/src/sage/libs/gmp/pylong.pyx
+++ b/src/sage/libs/gmp/pylong.pyx
@@ -32,6 +32,14 @@ from cpython.longintrepr cimport _PyLong_New, py_long, digit, PyLong_SHIFT
from .mpz cimport *
cdef extern from *:
+ """
+ /* Compatibility for python 3.8, can be removed later */
+ #if PY_VERSION_HEX < 0x030900A4 && !defined(Py_SET_SIZE)
+ static inline void _Py_SET_SIZE(PyVarObject *ob, Py_ssize_t size)
+ { ob->ob_size = size; }
+ #define Py_SET_SIZE(ob, size) _Py_SET_SIZE((PyVarObject*)(ob), size)
+ #endif
+ """
void Py_SET_SIZE(object, Py_ssize_t)
int hash_bits """
#ifdef _PyHASH_BITS
Branch pushed to git repo; I updated commit sha1. New commits:
ca88753 | restore support for python 3.8 |