./configure: Make --with-sage-venv=auto the default
mkoeppe opened this issue · 79 comments
Since Sage 9.4, it is possible to use an installation tree for Python packages separate from SAGE_LOCAL:
https://wiki.sagemath.org/ReleaseTours/sage-9.4#For_developers:_..2Fconfigure_--prefix.3DSAGE_LOCAL_--with-sage-venv.3DSAGE_VENV
In this ticket we make a new setting --with-sage-venv=auto the default, which means:
- if
--prefixhas not been used (soSAGE_LOCAL=local), then setSAGE_VENVtolocal/var/lib/sage/venv-PYTHONVERSION
Users can also pass--with-sage-venvor--with-sage-venv=yes, which means to unconditionally setSAGE_VENVtoSAGE_LOCAL/var/lib/sage/venv-PYTHONVERSION.
(We keep the layout as is in incremental builds.)
By keying the default name to the Python version, we ensure that if the system python version changes (either because of system updates or explicit reconfiguration), we create a fresh venv and automatically rebuild all Python packages, eliminating problems such as the one reported in https://groups.google.com/g/sage-devel/c/hZPHxqn_Cyk/m/fOAZCUWHAQAJ
Moreover, at config.status time, we create a symbolic link SAGE_ROOT/venv -> SAGE_VENV (overwriting an existing symbolic link)
The link is for easy access by users and is not used otherwise. Instead of local/lib/python3/site-packages/ or local/bin/sage, use venv/local/lib/python3/site-packages/ etc.
For symmetry, we also create a symbolic link SAGE_ROOT/prefix -> SAGE_LOCAL. (This generalizes what is already done by tox local.)
Users will still be able to restore the previous behavior by using --with-sage-venv=no, or use a specific path as in --with-sage-venv=/path/to/venv.
CC: @dimpase @jhpalmieri
Component: build: configure
Author: Matthias Koeppe
Reviewer: John Palmieri
Issue created by migration from https://trac.sagemath.org/ticket/32442
Description changed:
---
+++
@@ -1,16 +1,18 @@
Since Sage 9.4, it is possible to use an installation tree for Python packages separate from SAGE_LOCAL:
https://wiki.sagemath.org/ReleaseTours/sage-9.4#For_developers:_..2Fconfigure_--prefix.3DSAGE_LOCAL_--with-sage-venv.3DSAGE_VENV
-In this ticket we make `--with-sage-venv[=yes]` the default, which will mean:
-- set `SAGE_VENV` to `SAGE_LOCAL/var/lib/sage/venv-PYTHONVERSION`
+In this ticket we make a new setting `--with-sage-venv=auto` the default, which means:
+- if `--prefix` has not been used (so `SAGE_LOCAL=local`), then
+set `SAGE_VENV` to `local/var/lib/sage/venv-PYTHONVERSION`
+Users can also pass `--with-sage-venv` or `--with-sage-venv=yes`, which means to unconditionally set `SAGE_VENV` to `SAGE_LOCAL/var/lib/sage/venv-PYTHONVERSION`.
- By keying the name to the Python version, we ensure that if the system python version changes, we create a fresh venv and automatically rebuild all Python packages, eliminating problems such as the one reported in https://groups.google.com/g/sage-devel/c/hZPHxqn_Cyk/m/fOAZCUWHAQAJ
+By keying the default name to the Python version, we ensure that if the system python version changes (either because of system updates or explicit reconfiguration), we create a fresh venv and automatically rebuild all Python packages, eliminating problems such as the one reported in https://groups.google.com/g/sage-devel/c/hZPHxqn_Cyk/m/fOAZCUWHAQAJ
-- at `config.status` time, create a symbolic link `SAGE_ROOT/venv -> SAGE_VENV` (overwriting an existing symbolic link)
+Moreover, at `config.status` time, we create a symbolic link `SAGE_ROOT/venv -> SAGE_VENV` (overwriting an existing symbolic link)
- The link is for easy access by users and is not used otherwise. Instead of `local/lib/python3/site-packages/` or `local/bin/sage`, use `venv/local/lib/python3/site-packages/` etc.
+The link is for easy access by users and is not used otherwise. Instead of `local/lib/python3/site-packages/` or `local/bin/sage`, use `venv/local/lib/python3/site-packages/` etc.
- For symmetry, we also create a symbolic link `SAGE_ROOT/prefix -> SAGE_LOCAL`. (This generalizes what is already done by `tox local`.)
+For symmetry, we also create a symbolic link `SAGE_ROOT/prefix -> SAGE_LOCAL`. (This generalizes what is already done by `tox local`.)
Users will still be able to restore the previous behavior by using `--with-sage-venv=no`, or use a specific path as in `--with-sage-venv=/path/to/venv`.
Description changed:
---
+++
@@ -2,8 +2,7 @@
https://wiki.sagemath.org/ReleaseTours/sage-9.4#For_developers:_..2Fconfigure_--prefix.3DSAGE_LOCAL_--with-sage-venv.3DSAGE_VENV
In this ticket we make a new setting `--with-sage-venv=auto` the default, which means:
-- if `--prefix` has not been used (so `SAGE_LOCAL=local`), then
-set `SAGE_VENV` to `local/var/lib/sage/venv-PYTHONVERSION`
+- if `--prefix` has not been used (so `SAGE_LOCAL=local`), then set `SAGE_VENV` to `local/var/lib/sage/venv-PYTHONVERSION`
Users can also pass `--with-sage-venv` or `--with-sage-venv=yes`, which means to unconditionally set `SAGE_VENV` to `SAGE_LOCAL/var/lib/sage/venv-PYTHONVERSION`.
By keying the default name to the Python version, we ensure that if the system python version changes (either because of system updates or explicit reconfiguration), we create a fresh venv and automatically rebuild all Python packages, eliminating problems such as the one reported in https://groups.google.com/g/sage-devel/c/hZPHxqn_Cyk/m/fOAZCUWHAQAJAuthor: Matthias Koeppe
Branch pushed to git repo; I updated commit sha1. New commits:
117fe3a | build/make/Makefile.in (...-clean): Do not fail trying to uninstall if SAGE_VENV has not been created yet |
Description changed:
---
+++
@@ -4,6 +4,8 @@
In this ticket we make a new setting `--with-sage-venv=auto` the default, which means:
- if `--prefix` has not been used (so `SAGE_LOCAL=local`), then set `SAGE_VENV` to `local/var/lib/sage/venv-PYTHONVERSION`
Users can also pass `--with-sage-venv` or `--with-sage-venv=yes`, which means to unconditionally set `SAGE_VENV` to `SAGE_LOCAL/var/lib/sage/venv-PYTHONVERSION`.
+
+(We keep the layout as is in incremental builds.)
By keying the default name to the Python version, we ensure that if the system python version changes (either because of system updates or explicit reconfiguration), we create a fresh venv and automatically rebuild all Python packages, eliminating problems such as the one reported in https://groups.google.com/g/sage-devel/c/hZPHxqn_Cyk/m/fOAZCUWHAQAJ
Branch pushed to git repo; I updated commit sha1. New commits:
8af6041 | Merge tag '9.5.beta2' into t/32442/__configure__make___with_sage_venv_the_default |
After running make distclean && ./configure:
% git status
On branch t/32442/__configure__make___with_sage_venv_the_default
Your branch is up to date with 'trac/u/mkoeppe/__configure__make___with_sage_venv_the_default'.
Untracked files:
(use "git add <file>..." to include in what will be committed)
venv
nothing added to commit but untracked files present (use "git add" to track)
Oh, and venv is a symlink pointing to local.
Branch pushed to git repo; I updated commit sha1. New commits:
b3f7d34 | .gitignore: Add /venv |
Thanks for catching this!
Replying to @jhpalmieri:
Oh, and
venvis a symlink pointing tolocal.
Yes, in an incremental build, nothing should change (except that this symlink is created).
After a make distclean, the new behavior will show
Oh, now I see that you said you did make distclean...
Maybe this has been going on with venv builds for a while, but the symlinks look messed up to me: my local directory looks like
% ls -l local
total 0
drwxr-xr-x 148 jpalmier staff 4736 Sep 27 16:21 bin
drwxr-xr-x 2 jpalmier staff 64 Sep 27 16:09 etc
drwxr-xr-x 47 jpalmier staff 1504 Sep 27 16:21 include
drwxr-xr-x 73 jpalmier staff 2336 Sep 27 16:44 lib
lrwxr-xr-x 1 jpalmier staff 3 Sep 27 16:09 lib64 -> lib
drwxr-xr-x 4 jpalmier staff 128 Sep 27 16:21 libexec
lrwxr-xr-x 1 jpalmier staff 5 Sep 27 16:09 local -> local
drwxr-xr-x 34 jpalmier staff 1088 Sep 27 16:21 share
drwxr-xr-x 5 jpalmier staff 160 Sep 27 16:13 var
lrwxr-xr-x 1 jpalmier staff 33 Sep 27 16:09 venv-python3.9 -> local/var/lib/sage/venv-python3.9
In particular, local/local points to itself, and so venv-python3.9 doesn't point anywhere. venv-python3.9 should probably point to ../local/..... I don't know what the local symlink is supposed to do.
Everything still builds, so this is obviously not crucial, but it doesn't look right.
Yes, that doesn't look right. Could you post config.log please?
Attachment: config.log
This was after make distclean && ./configure && make ptestlong.
Branch pushed to git repo; I updated commit sha1. New commits:
fc4b641 | configure.ac: Remove conveniene symlinks before (re)creating them |
Try with this version please
I've started, and I now only see one symlink in local: lib64 -> lib. At the top level, I see prefix -> local and venv -> local/var/lib/sage/venv-python3.9.
By the way, make distclean should probably remove the top-level symlink venv. As it is, running make distclean first runs ./configure (at least some of the time) which creates venv.
make distclean should also remove prefix.
Replying to @jhpalmieri:
I've started, and I now only see one symlink in
local:lib64 -> lib. At the top level, I seeprefix -> localandvenv -> local/var/lib/sage/venv-python3.9.
Great, this is the intended layout.
Branch pushed to git repo; I updated commit sha1. New commits:
8bcbf1d | Makefile (distclean): Remove convenience symlinks prefix, venv |
How is it decided which documentation goes in local/share/doc and which goes in venv/share/doc? For example, pplpy and sage are local/share/doc, although their corresponding installation files are in venv/var/lib/sage/installed.
I haven't really paid attention to the documentation. There is no attempt to actively install documentation in venv. The installation of the documentation is not really done by these Python packages themselves but rather by our Sage-specific install scripts. build/pkgs/pplpy/spkg-postinst.in directly refers to SAGE_SHARE (= local/share) for the installation location. I'd say a more systematic consideration of this issue would be something for #29868.
On the other hand, networkx and sagetex put their docs in the venv location. Not a big deal and we can certainly deal with it on another ticket.
Thanks for investigating. These two Python packages install their documentation by themselves on setup.py install (and similar), we are not doing anything Sage-specific there. Perhaps at some point we'll get around to moving #27164 forward...
Can you explain the change in Makefile.in? In what circumstances would the package tree directory not exist?
I think that ./configure --help should be changed from
--with-sage-venv={auto,yes,no,SAGE_VENV}
to
--with-sage-venv={auto (default),yes,no,SAGE_VENV}
In this line from configure.ac:
eval SAGE_VENV="$SAGE_VENV"dnl eval so as to evaluate the embedded ${SAGE_LOCAL}
the placement of dnl... doesn't fit the style of the rest of the file. I honestly don't care. Is there any point to change it?
Replying to @jhpalmieri:
Can you explain the change in
Makefile.in? In what circumstances would the package tree directory not exist?
Running make SCRIPTPACKAGE-clean or ./sage -f SCRIPTPACKAGE right after running ./configure (or before any package is installed into the venv)
Branch pushed to git repo; I updated commit sha1. New commits:
dcb4a08 | configure.ac: Indicate default in help string for --with-sage-venv |
Replying to @jhpalmieri:
the placement of
dnl...doesn't fit the style of the rest of the file. I honestly don't care. Is there any point to change it?
It's correct as is, to suppress unnecessary whitespace in the generated file. There are lots of other places in our autoconf scripts that are not quite so careful about this, in the end it does not matter so much
I think this is ready to go, but I will ask one more question — probably not a question for this ticket, just a general venv question: how do I use a venv? I built Sage using homebrew's Python 3.9, and I wanted to see what happened with /usr/bin/python3, which is version 3.8. But just doing ./configure --with-python=/usr/bin/python3 doesn't accomplish anything because local/bin/python3 is still a symlink pointing to homebrew's Python. What do I need to do to convince Sage to use a different Python, therefore creating a different venv, while keeping the other venv? Do I need to manually remove the Python symlink?
Replying to @jhpalmieri:
But just doing
./configure --with-python=/usr/bin/python3doesn't accomplish anything becauselocal/bin/python3is still a symlink pointing to homebrew's Python.
Try ./configure --with-python=/usr/bin/python3 --with-sage-venv=yes
Should --with-sage-venv=auto detect whether this is an incremental build from an earlier build using --with-sage-venv=auto, i.e. whether SAGE_ENV matches local/var/lib/sage/venv-SOMEPYTHONVERSION? And if so, then should it use the appropriate possibly new directory for the venv? Otherwise the ticket description's "we ensure that if the system python version changes (either because of system updates or explicit reconfiguration), we create a fresh venv" is not accurate: this will only happen if the user says --with-sage-venv=yes, if I understand correctly.
If you start from a distclean source tree, the default --with-sage-venv=auto does not create a python3 in local/bin/. So it will work as advertised.
I think you are testing with a tree that has python3 in local/bin/, either from a build before this ticket, or from a defective earlier version of this ticket.
I think you're right, I think it's working the way it's supposed to. Let me test one or two more things.
Can this be fixed? I did
% ./configure # use homebrew's Python 3.9
% make ptestlong
...
all tests passed
% ./configure --with-python=/usr/bin/python3 # Python 3.8
% make
% ./sage -tp src/sage_setup
...
sage -t --warn-long 107.1 --random-seed=0 src/sage_setup/clean.py # 1 doctest failed
sage -t --warn-long 107.1 --random-seed=0 src/sage_setup/find.py # 2 doctests failed
Details:
File "src/sage_setup/clean.py", line 96, in sage_setup.clean._find_stale_files
Failed example:
for f in stale_iter:
if f.endswith(skip_extensions): continue
if '/ext_data/' in f: continue
print('Found stale file: ' + f)
Expected nothing
Got:
Found stale file: /Users/palmieri/Library/Caches/com.apple.python/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.5.beta2/local/var/lib/sage/venv-python3.8/lib/python3.8/site-packages/sage/all_cmdline.cpython-38.pyc
Found stale file: /Users/palmieri/Library/Caches/com.apple.python/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.5.beta2/local/var/lib/sage/venv-python3.8/lib/python3.8/site-packages/sage/version.cpython-38.pyc
...
[many lines like this]
and
File "src/sage_setup/find.py", line 365, in sage_setup.find.installed_files_by_module
Failed example:
f1
Expected:
'sage/structure/__init__.py'
Got:
'/Users/palmieri/Library/Caches/com.apple.python/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.5.beta2/local/var/lib/sage/venv-python3.8/lib/python3.8/site-packages/sage/structure/__init__.cpython-38.pyc'
**********************************************************************
File "src/sage_setup/find.py", line 367, in sage_setup.find.installed_files_by_module
Failed example:
f2
Expected:
'sage/structure/....pyc'
Got:
'sage/structure/__init__.py'
Okay, then I think this is ready.
Reviewer: John Palmieri
Thank you!
Breaks build on Debian 9:
Testing importing of various modules...
ctypes module imported OK
math module imported OK
hashlib module imported OK
crypt module imported OK
socket module imported OK
zlib module imported OK
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/buildbot/slave/sage_git/build/local/var/lib/sage/venv-python3.9.7/var/tmp/sage/build/python3-3.9.7/src/Lib/sqlite3/__init__.py", line 23, in <module>
from sqlite3.dbapi2 import *
File "/home/buildbot/slave/sage_git/build/local/var/lib/sage/venv-python3.9.7/var/tmp/sage/build/python3-3.9.7/src/Lib/sqlite3/dbapi2.py", line 27, in <module>
from _sqlite3 import *
ModuleNotFoundError: No module named '_sqlite3'
sqlite3 module failed to import
ssl module imported OK
Error: One or more modules failed to import.
real 1m25.694s
user 2m0.796s
sys 0m10.188s
************************************************************************
Error building package python3-3.9.7
************************************************************************
Build log please
I was able to reproduce this using tox -e docker-debian-stretch-minimal.
When I run tox -e docker-debian-stretch-minimal, where are the package log files?
They are in the Docker container.
At the end of the build, tox will tell you name and/or sha256 hash of the image built.
You can to docker run -it IMAGE bash to enter the container and then it's all in /sage/logs
Okay, before this latest change I see the Python problem, as well as:
* package: python3-3.9.7
last build time: Oct 13 17:49
log file: /sage/logs/pkgs/python3-3.9.7.log
* package: fplll-5.4.1
last build time: Oct 13 18:01
log file: /sage/logs/pkgs/fplll-5.4.1.log
build directory: /sage/local/var/tmp/sage/build/fplll-5.4.1
* package: cmake-3.21.0
last build time: Oct 13 18:04
log file: /sage/logs/pkgs/cmake-3.21.0.log
build directory: /sage/local/var/tmp/sage/build/cmake-3.21.0
* package: gfortran-10.3.0
last build time: Oct 13 18:14
log file: /sage/logs/pkgs/gfortran-10.3.0.log
build directory: /sage/local/var/tmp/sage/build/gfortran-10.3.0
The other failures appear to be unrelated to this ticket: I get them with the develop branch.
Replying to @jhpalmieri:
The other failures appear to be related to this ticket: I get them with the
developbranch.
Did you mean to write "unrelated to this ticket"?
Yes, sorry, unrelated. I've edited to make it clearer in case anyone looks at this later. With the most recent branch here, Python builds correctly. I think that's good enough for another positive review.
Thank you!
By the way, I cannot reproduce the failures with the other packages, and they also built correctly on GH Actions https://github.com/sagemath/sage/runs/3863930628?check_suite_focus=true
Maybe the issue was that my Docker was out of date, or maybe it was a memory issue. I upgraded Docker and allocated more RAM to it, and now those packages build.
I do get another failure, though; don't know if it's related to this ticket. With tox -e docker-debian-stretch-minimal, I get:
gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1)
****************************************************
Package 'setuptools' is currently not installed
No legacy uninstaller found for 'setuptools'; nothing to do
python3: error while loading shared libraries: libpython3.9.so.1.0: cannot open shared object file: No such file or directory
********************************************************************************
Error building / installing setuptools for Python
********************************************************************************
real 0m0.021s
user 0m0.011s
sys 0m0.010s
************************************************************************
Error installing package setuptools-58.0.2
************************************************************************
Please email sage-devel (http://groups.google.com/group/sage-devel)
explaining the problem and including the log files
/sage/logs/pkgs/setuptools-58.0.2.log
and
/sage/config.log
Describe your computer, operating system, etc.
If you want to try to fix the problem yourself, *don't* just cd to
/sage/local/var/lib/sage/venv-python3.9.7/var/tmp/sage/build/setuptools-58.0.2 and type 'make' or whatever is appropriate.
Instead, the following commands setup all environment variables
correctly and load a subshell for you to debug the error:
(cd '/sage/local/var/lib/sage/venv-python3.9.7/var/tmp/sage/build/setuptools-58.0.2' && '/sage/sage' --buildsh)
When you are done debugging, you can type "exit" to leave the subshell.
There is a file SAGE_ROOT/venv/local/lib/libpython3.9.so.1.0. If I run ./sage --python then I get the same error.
Yes, I see this error now too.
Looks like on Linux I need to set the linker flags differently.
Branch pushed to git repo; I updated commit sha1. New commits:
3bbc5d8 | build/pkgs/python3/spkg-build.in: Set rpath |
This fixes it for me; this time I ran a complete build
This works for me with a few different builds (standard build on my machine and a few tox builds via Docker).
Changed branch from u/mkoeppe/__configure__make___with_sage_venv_the_default to none