pkgs/sagemath-*/tox.ini: Create environment that uses Sage's python as the basepython, fix sagemath-standard deps
mkoeppe opened this issue · 60 comments
Newer versions of tox seem to provision python 3.10 when the environment python is used, but then our wheels don't necessarily fit.
For this reason, the examples shown in the developer's manual hardcoded the Python version (tox -e py39).
We now define tox environments sagepython in tox.ini, which takes Python as SAGE_VENV/bin/python3 and change the manual to use this environment in examples, such as:
./bootstrap && ./sage -sh -c '(cd pkgs/sagemath-standard && SAGE_NUM_THREADS=16 tox -v -v -v -e sagepython-sagewheels-nopypi)'
(to be run after make build)
Depends on #33936
Component: refactoring
Author: Matthias Koeppe
Branch: ca9d6ce
Reviewer: Dima Pasechnik
Issue created by migration from https://trac.sagemath.org/ticket/32716
Author: Matthias Koeppe
Description changed:
---
+++
@@ -1,2 +1,6 @@
-Newer versions of `tox` seem to provision python 3.10 when the environment `python` is used, but then our wheels don't fit.
+Newer versions of `tox` seem to provision python 3.10 when the environment `python` is used, but then our wheels don't necessarily fit.
+For this reason, the examples shown in the developer's manual hardcoded the Python version (`tox -e py39`).
+
+We now define tox environments `sagepython` in `tox.ini`, which takes Python as `SAGE_VENV/bin/python3` and change the manual to use this environment in examples.
+Description changed:
---
+++
@@ -2,5 +2,9 @@
For this reason, the examples shown in the developer's manual hardcoded the Python version (`tox -e py39`).
-We now define tox environments `sagepython` in `tox.ini`, which takes Python as `SAGE_VENV/bin/python3` and change the manual to use this environment in examples.
+We now define tox environments `sagepython` in `tox.ini`, which takes Python as `SAGE_VENV/bin/python3` and change the manual to use this environment in examples, such as:
+```
+./bootstrap && ./sage -sh -c '(cd pkgs/sagemath-standard && SAGE_NUM_THREADS=16 tox -v -v -v -e sagepython-sagewheels-nopypi)'
+```
+command in the ticket description fails with
WARNING: Url 'file:///home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/../../../../venv/var/lib/sage/wheels' is ignored: it is neither a file nor a directory.
ERROR: Could not find a version that satisfies the requirement sage-conf==9.7.beta1
ERROR: No matching distribution found for sage-conf==9.7.beta1
ERROR: invocation failed (exit code 1)
ERROR: could not install deps [-rrequirements.txt]; v = InvocationError('/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/bin/python -m pip install -rrequirements.txt', 1)
sagepython-sagewheels-nopypi finish: getenv /home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi after 0.69 seconds
...
Did you run make build as the manual explains?
Description changed:
---
+++
@@ -7,4 +7,4 @@
```
./bootstrap && ./sage -sh -c '(cd pkgs/sagemath-standard && SAGE_NUM_THREADS=16 tox -v -v -v -e sagepython-sagewheels-nopypi)'
```
-
+(to be run after `make build`)besides make build, I also needed to apply
diff --git a/build/pkgs/sage_conf/install-requires.txt b/build/pkgs/sage_conf/install-requires.txt
index 83ad7e641fa..3d4ff51d565 100644
--- a/build/pkgs/sage_conf/install-requires.txt
+++ b/build/pkgs/sage_conf/install-requires.txt
@@ -1 +1 @@
-sage-conf ~= 9.5.b6
+sage-conf
diff --git a/build/pkgs/sage_setup/install-requires.txt b/build/pkgs/sage_setup/install-requires.txt
index 01a931f5eb5..9d4666c07f4 100644
--- a/build/pkgs/sage_setup/install-requires.txt
+++ b/build/pkgs/sage_setup/install-requires.txt
@@ -1 +1 @@
-sage-setup ~= 9.5.b6
+sage-setupthat 9.5.b6 seemed to come from an obsolete workaround anyway.
And after these changes, it errors out (much later) with
ModuleNotFoundError: No module named 'primecountpy'
ERROR: InvocationError for command /home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/bin/python -c 'import sys; "" in sys.path and sys.path.remove(""); import sage.all; print(sage.all.__file__)' (exited with code 1)
the latter is probably a config error, as the module is there and well:
sage: import primecountpy
sage: primecountpy.primecount.prime_pi(100)
25
Branch pushed to git repo; I updated commit sha1. New commits:
b030e7a | Merge tag '9.7.beta2' into t/32716/pkgs_sagemath_standard_tox_ini__use_sage_s_python_by_default |
Branch pushed to git repo; I updated commit sha1. New commits:
c9ca4e4 | pkgs/sagemath-standard/tox.ini: Set basepython correctly for all supported sagepython-* environments |
Thanks for testing. Indeed it didn't work with sagemath-standard because I made a mistake in the tox configuration. Fixed now
Branch pushed to git repo; I updated commit sha1. New commits:
01bd09c | pkgs/sagemath-{environment,repl}/tox.ini: Add sagepython toxenv |
I'm not a big yaml/tompl expert, but why
basepython = {env:SAGE_VENV}/bin/python3
is repeated 7 times? Can't it be set once and for all?
The basepython variable has useful defaulting behavior: Environment names that include a pyNM factor configure basepython = pythonN.M etc., see https://tox.wiki/en/latest/config.html#tox-environments
I want to override it only for environments with the sagepython factor. Unfortunately I haven't found a more elegant solution than repeating it for the listed environments.
Branch pushed to git repo; I updated commit sha1. New commits:
3839cee | Merge tag '9.7.beta3' into t/32716/pkgs_sagemath_standard_tox_ini__use_sage_s_python_by_default |
on Fedora 34:
sage: import sage_conf
sage: sage_conf.VERSION
'9.7.beta3'
but
...
Skipping link: wrong project name (not sage-conf): file:///home/scratch/scratch2/dimpase/sage/sagetrac-mirror/local/var/lib/sage/venv-python3.9/var/lib/sage/wheels/sphinxcontrib_serializinghtml-1.1.5-py2.py3-none-any.whl
Local files found: /home/scratch/scratch2/dimpase/sage/sagetrac-mirror/local/var/lib/sage/venv-python3.9/var/lib/sage/wheels/sage_conf-9.7b3-py3-none-any.whl
Given no hashes to check 0 links for project 'sage-conf': discarding no candidates
ERROR: Could not find a version that satisfies the requirement sage-conf~=9.5.b6
ERROR: No matching distribution found for sage-conf~=9.5.b6
Exception information:
Traceback (most recent call last):
File "/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 171, in _merge_into_criterion
crit = self.state.criteria[name]
KeyError: 'sage-conf'
it seems that sage-conf is not looked up in the right place, i.e. in venv/.
Do you happen to use --enable-editable in this build?
OK, I guess I have to make changes to build/pkgs/sage_conf/install-requires.txt so it does not reject the wheel
Branch pushed to git repo; I updated commit sha1. New commits:
fe510ef | build/pkgs/sage_{conf,setup}/install-requires: Use ~= 9.7.b3 |
for some reason, in this branch ./configure was hanging (!) at testing for giac version. I did the following obvious change, and it passed
--- a/build/pkgs/giac/spkg-configure.m4
+++ b/build/pkgs/giac/spkg-configure.m4
@@ -23,5 +23,6 @@ SAGE_SPKG_CONFIGURE([giac], [
], [sage_spkg_install_giac=yes])
], [sage_spkg_install_giac=yes])
m4_popdef([GIAC_MIN_VERSION])
+ m4_popdef([GIAC_MAX_VERSION])
])
])That's very strange
Do you want to push this change to the branch here or open a new ticket for it? Either way is fine with me
now something is not right with primecountpy (which works fine with "normal" sage, e.g.
./sage -tp 16 src/sage/functions/prime_pi.pyx passes just fine)
sagepython-sagewheels-nopypi finish: envreport after 0.16 seconds
sagepython-sagewheels-nopypi installed: DEPRECATION: --find-links option in pip freeze is deprecated. pip 21.2 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/9069.,-f file:///home/scratch/scratch2/dimpase/sage/sagetrac-mirror/local/var/lib/sage/venv-python3.9/var/lib/sage/wheels,alabaster==0.7.12,Babel==2.9.1,backcall==0.2.0,certifi==2021.10.8,cffi==1.15.0,charset-normalizer==2.0.4,cvxopt==1.2.7,cycler==0.11.0,cypari2==2.1.2,cysignals==1.11.2,Cython==0.29.30,decorator==5.0.9,docutils==0.17.1,entrypoints==0.3,fonttools==4.28.4,fpylll==0.5.7,gmpy2==2.1.1,idna==3.2,imagesize==1.2.0,importlib-metadata==4.8.2,ipykernel==6.6.0,ipython==7.29.0,ipython-genutils==0.2.0,ipywidgets==7.6.5,jedi==0.18.1,Jinja2==3.1.2,jupyter-client==7.1.0,jupyter-core==4.9.1,kiwisolver==1.3.2,MarkupSafe==2.1.1,matplotlib==3.5.1,matplotlib-inline==0.1.2,memory-allocator==0.1.3,mpmath==1.2.1,nest-asyncio==1.5.4,networkx==2.6.3,numpy==1.21.4,packaging==21.3,parso==0.8.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.0.1,pkgconfig==1.5.5,pplpy==0.8.6,prompt-toolkit==3.0.22,ptyprocess==0.5.1,pycparser==2.21,Pygments==2.10.0,pyparsing==3.0.6,python-dateutil==2.8.2,pytz==2021.3,pyzmq==22.3.0,requests==2.26.0,rpy2==3.3.6,sage-conf==9.7b3,sagemath-standard @ file:///home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/.tmp/package/1/sagemath-standard-9.7b3.zip,scipy==1.7.3,six==1.16.0,snowballstemmer==2.1.0,Sphinx==4.4.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,sympy==1.10.1,tornado==6.1,traitlets==5.1.1,tzlocal==2.1,urllib3==1.26.6,wcwidth==0.2.5,widgetsnbextension==3.5.2,zipp==3.5.0
removing /home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/tmp
sagepython-sagewheels-nopypi start: run-test-pre
sagepython-sagewheels-nopypi run-test-pre: PYTHONHASHSEED='632737325'
sagepython-sagewheels-nopypi finish: run-test-pre after 0.00 seconds
sagepython-sagewheels-nopypi start: run-test
sagepython-sagewheels-nopypi run-test: commands[0] | python -c 'import sys; "" in sys.path and sys.path.remove(""); import sage.all; print(sage.all.__file__)'
setting PATH=/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/bin:/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/src/bin:/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/build/bin:/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/local/var/lib/sage/venv-python3.9/bin:/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/local/bin:/home/scratch2/dimpase/mambaforge/bin:/users/dimpase/perl5/bin:/users/dimpase/bin:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/auto/users/dimpase/bin
[3346474] /home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard$ /home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/bin/python -c 'import sys; "" in sys.path and sys.path.remove(""); import sage.all; print(sage.all.__file__)'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/lib64/python3.9/site-packages/sage/all.py", line 138, in <module>
from sage.algebras.all import *
File "/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/lib64/python3.9/site-packages/sage/algebras/all.py", line 21, in <module>
import sage.algebras.catalog as algebras
File "/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/lib64/python3.9/site-packages/sage/algebras/catalog.py", line 84, in <module>
from sage.algebras.clifford_algebra import CliffordAlgebra as Clifford
File "/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/lib64/python3.9/site-packages/sage/algebras/clifford_algebra.py", line 35, in <module>
from sage.quadratic_forms.quadratic_form import QuadraticForm
File "/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/lib64/python3.9/site-packages/sage/quadratic_forms/quadratic_form.py", line 192, in <module>
class QuadraticForm(SageObject):
File "/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/lib64/python3.9/site-packages/sage/quadratic_forms/quadratic_form.py", line 323, in QuadraticForm
from sage.quadratic_forms.quadratic_form__local_field_invariants import \
File "/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/lib64/python3.9/site-packages/sage/quadratic_forms/quadratic_form__local_field_invariants.py", line 30, in <module>
from sage.functions.all import sgn
File "/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/lib64/python3.9/site-packages/sage/functions/all.py", line 68, in <module>
from .prime_pi import legendre_phi, partial_sieve_function, prime_pi
File "sage/functions/prime_pi.pyx", line 37, in init sage.functions.prime_pi (build/cythonized/sage/functions/prime_pi.c:5947)
from primecountpy.primecount import prime_pi as _prime_pi
ModuleNotFoundError: No module named 'primecountpy'
ERROR: InvocationError for command /home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/bin/python -c 'import sys; "" in sys.path and sys.path.remove(""); import sage.all; print(sage.all.__file__)' (exited with code 1)
sagepython-sagewheels-nopypi finish: run-test after 0.34 seconds
sagepython-sagewheels-nopypi start: run-test-post
sagepython-sagewheels-nopypi finish: run-test-post after 0.00 seconds
_______________________________________________ summary _______________________________________________
ERROR: sagepython-sagewheels-nopypi: commands failed
cleanup /home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/.tmp/package/1/sagemath-standard-9.7b3.zip
iirc I saw this earlier...
Replying to @mkoeppe:
Do you want to push this change to the branch here or open a new ticket for it? Either way is fine with me
as more changes seem to be needed, perhaps just add it here?
ok
Branch pushed to git repo; I updated commit sha1. New commits:
b5fedff | build/pkgs/giac/spkg-configure.m4: Pop what was pushed |
Branch pushed to git repo; I updated commit sha1. New commits:
74f1e10 | src/requirements.txt.m4, src/setup.cfg.m4 (install_requires): Add primecountpy |
Replying to @dimpase:
now something is not right with
primecountpy
I have added it as a runtime dependency. Strange, I thought I did this a long time ago already
now this fails with a similar error for lrcalc:
File "/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/lib64/python3.9/site-packages/sage/combinat/sf/schur.py", line 21, in <module>
import sage.libs.lrcalc.lrcalc as lrcalc
File "/home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/lib64/python3.9/site-packages/sage/libs/lrcalc/lrcalc.py", line 193, in <module>
import lrcalc
ModuleNotFoundError: No module named 'lrcalc'
ERROR: InvocationError for command /home/scratch/scratch2/dimpase/sage/sagetrac-mirror/pkgs/sagemath-standard/.tox/sagepython-sagewheels-nopypi/bin/python -c 'import sys; "" in sys.path and sys.path.remove(""); import sage.all; print(sage.all.__file__)' (exited with code 1)
sagepython-sagewheels-nopypi finish: run-test after 0.51 seconds
sagepython-sagewheels-nopypi start: run-test-post
sagepython-sagewheels-nopypi finish: run-test-post after 0.00 seconds
_______________________________________________ summary _______________________________________________
ERROR: sagepython-sagewheels-nopypi: commands failed
I'm trying the change for lrcalc analogous to the one for primecountpy.
Branch pushed to git repo; I updated commit sha1. New commits:
5ea0035 | src/requirements.txt.m4, src/setup.cfg.m4 (install_requires): Add lrcalc_python |
7b17c2d | Split sage_docbuild.conf to sage.misc.sagedoc_conf |
282270f | Fix docs |
c2766bf | Move skip_member() to sage_docbuild |
4296851 | Move the remnants of skip_member from sagedoc_conf |
45f9ff3 | Merge branch 'develop' |
0a02496 | Change assert back to if |
1c2f41b | Merge #33936 |
Replying to @dimpase:
I'm trying the change for
lrcalcanalogous to the one forprimecountpy.
Indeed, that was missing too, fixed now.
Replying to @dimpase:
I'm trying the change for
lrcalcanalogous to the one forprimecountpy.
that appeared to suffice (but now the branch got updated)
what I don't like are explicit versions of sage-conf and sage-setup in requirements.txt
Obviously there should be an automated way.
Reviewer: Dima Pasechnik
OK, otherwise it looks good. Perhaps deal with versions mentioned in the last comment on another ticket (unless it can be trivially fixed).
Replying to @dimpase:
what I don't like are explicit versions of
sage-confandsage-setupinrequirements.txtObviously there should be an automated way.
src/requirements.txt is already generated from src/requirements.txt.m4
If you mean build/pkgs/sage_conf/install-requires.txt and build/pkgs/sage_setup/install-requires.txt, yes, that will need a better solution but it's way beyond the scope of this ticket
OK
Thanks!
Merge failure on top of:
662aaf2 Trac #32423: Update numpy to 1.22.x, scipy 1.8.x, networkx 2.8.x
d98aa73 Trac #29549: bootstrap: Clean up use of gettextize
24b4eeb Trac #13321: FreeModule.hom stores its matrix over the wrong ring
93357f3 Trac #34104: Build & Test workflow: Run apt-get update first
23e4bbe Trac #34092: Docbuild workflow broken because furo is too new
9440e71 Trac #34088: cygwin python3 missing ensurepip
9879ff4 Trac #33936: Remove runtime dependency on sage_docbuild introduced in #33763
55d05c9 Trac #31403: giac: Make cliquer a dependency, libnauty an optional dependency
843eb03 Updated SageMath version to 9.7.beta4
merge was not clean: conflicts in pkgs/sagemath-standard/tox.ini
Branch pushed to git repo; I updated commit sha1. New commits:
ca9d6ce | Merge tag '9.7.beta4' into t/32716/pkgs_sagemath_standard_tox_ini__use_sage_s_python_by_default |
Changed branch from u/mkoeppe/pkgs_sagemath_standard_tox_ini__use_sage_s_python_by_default to ca9d6ce
Replying to @mkoeppe:
Replying to @dimpase:
what I don't like are explicit versions of
sage-confandsage-setupinrequirements.txtObviously there should be an automated way.
src/requirements.txtis already generated fromsrc/requirements.txt.m4If you mean
build/pkgs/sage_conf/install-requires.txtandbuild/pkgs/sage_setup/install-requires.txt, yes, that will need a better solution but it's way beyond the scope of this ticket
Implemented now in #33817