sagemath/sage

Add src/requirements.txt for installation of sagelib in a venv

mkoeppe opened this issue · 60 comments

mkoeppe commented

After #30527, it is possible to install Python packages built by the Sage distribution into user-defined venvs.

In this ticket, we create build/pkgs/sagelib/src/requirements.txt -> src/requirements.txt to facilitate installing sagelib into such a venv. (This file duplicates some information also available in build/pkgs; #29041 will rectify this.)

A virtual environment using it can be created and doctested using:

./sage -sh -c '(cd build/pkgs/sagelib/src && tox -v -v -v)'

(Reusing wheels built by the Sage distribution requires to do sage -i pip2pi first - this is from #30527.)

Related:

  • #28746 Developer's Guide: Add instructions for sagelib development in a conda environment
  • #28752 Make sagelib pip-installable in a conda environment

Follow-up tickets:

  • #30371 In-place (editable) installs of sagelib in a venv
  • #29041 at ./bootstrap time, generate requirements.txt, constraints.txt, setup.cfg [install_requires], Pipfile from build/pkgs
  • add documentation on how to set up the virtual environment, using either plain python -m venv, virtualenv (20.x), or pipenv.

Depends on #30527
Depends on #30013

CC: @tobiasdiez @kiwifb @jhpalmieri @videlec

Component: build

Keywords: sd111

Branch/Commit: u/mkoeppe/add-src-requirements-txt @ 9ad6b36

Issue created by migration from https://trac.sagemath.org/ticket/30578

mkoeppe commented

Changed dependencies from #30527 to #29500, #30527, #22731

mkoeppe commented

Description changed:

--- 
+++ 
@@ -1,6 +1,7 @@
 After #30527, it is possible to install Python packages built by the Sage distribution into user-defined venvs.  
 
 In this ticket, we create `src/requirements.txt` to facilitate this.
+(This duplicates some information also available in `build/pkgs`; #29041 will rectify this.)
 
 We add documentation on how to set up the virtual environment, using either plain `python -m venv`, `virtualenv` (20.x), or `pipenv`.
tobiasdiez commented
comment:4

What do you think about generating the requirements.txt file from a pipfile: https://pipenv-fork.readthedocs.io/en/latest/advanced.html#generating-a-requirements-txt? (Maybe as part of this ticket or as #29041)

Last 10 new commits:

0a64674build/pkgs/gambit/spkg-install.in: Install via bdist_wheel
ca58693build/pkgs/pillow/spkg-install.in: Install via bdist_wheel
207d80fbuild/pkgs/pip2pi: New
2555e19build/make/install: At the end, update the repository index
5a747c4build/bin/sage-pip-{install,uninstall}: Remove pip2 support
e66243fMerge branch 't/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels' into t/30527/pep-503-simple-repository-for-wheels
9937126Merge branch 't/30527/pep-503-simple-repository-for-wheels' into t/30371/public/build/inplace
1a39442build/pkgs/sagelib/src/setup.py: Replace symlink to SAGE_SRC/setup.py by a copy
67c56a6src/requirements.txt, src/MANIFEST.in: Move here from build/pkgs/sagelib/src, leave symlinks behind
1d88afeWIP
tobiasdiez commented

Commit: 1d88afe

mkoeppe commented
comment:5

Yes, I started to play a bit with pipenv, and this looks like a possible route.

I would like to keep that for #29041 though.

On the present ticket I would only want to do the most rudimentary thing - basically to try out the adjustments that sagelib will need (see #22731).

sagetrac-git commented

Changed commit from 1d88afe to ba2a6b9

sagetrac-git commented

Branch pushed to git repo; I updated commit sha1. New commits:

e83725fsage_setup.command.sage_install.sage_install: Rename to sage_install_and_clean, update use it in src/setup.py; refactor through new class sage_install
ba2a6b9src/requirements.txt: Add sage_conf
sagetrac-git commented

Branch pushed to git repo; I updated commit sha1. New commits:

033a95asage_setup.command.sage_install: Fixup: Remove cleaning code (moved to sage_install_and_clean)
sagetrac-git commented

Changed commit from ba2a6b9 to 033a95a

mkoeppe commented

Description changed:

--- 
+++ 
@@ -3,7 +3,10 @@
 In this ticket, we create `src/requirements.txt` to facilitate this.
 (This duplicates some information also available in `build/pkgs`; #29041 will rectify this.)
 
-We add documentation on how to set up the virtual environment, using either plain `python -m venv`, `virtualenv` (20.x), or `pipenv`.
+We add a tox environment to `src/tox.ini`, enabling the command
+`../sage -sh -c 'tox -e develop'` (with #30410: `../sage -tox -e develop`) for building sagelib in a venv.
+
+We also add documentation on how to set up the virtual environment, using either plain `python -m venv`, `virtualenv` (20.x), or `pipenv`.
 
 The existing `src/setup.py` is copied into `build/pkgs/sagelib/src`  to replace the symlink. In this way, the changes to `src/setup.py` made in this ticket do not affect the existing build process of the Sage library.
sagetrac-git commented

Changed commit from 033a95a to 4ed815a

sagetrac-git commented

Branch pushed to git repo; I updated commit sha1. New commits:

4ed815asrc/tox.ini (develop): New
sagetrac-git commented

Branch pushed to git repo; I updated commit sha1. New commits:

85175absrc/setup.py: Switch to setuptools, add setup.cfg, install-requires.txt
sagetrac-git commented

Changed commit from 4ed815a to 85175ab

mkoeppe commented

Changed dependencies from #29500, #30527, #22731 to #29500, #30527, #30658, #22731

sagetrac-git commented

Branch pushed to git repo; I updated commit sha1. New commits:

9ee2110build/bin/sage-dist-helpers: Also use $sudo for storing the wheel file
d7aac84src/doc/en/developer/packaging.rst: Update sdh_... documentation
9b7c7a0build/bin/sage-pip-{install,uninstall}: Fix typo in comment
4135e8bbuild/bin/sage-pip-install: Remove an outdated comment
f2e7075Merge tag '9.2.beta13' into t/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels
c2efa14Merge branch 't/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels' into t/30658/conda_forge_ubuntu_standard__pillow_fails_to_install__follow_up_
ff0c973build/pkgs/pillow/spkg-install.in: Fix up 'setup.py bdist_wheel' command
63490fdMerge branch 't/30658/conda_forge_ubuntu_standard__pillow_fails_to_install__follow_up_' into t/30578/add-src-requirements-txt
sagetrac-git commented

Changed commit from 85175ab to 63490fd

sagetrac-git commented

Branch pushed to git repo; I updated commit sha1. New commits:

5a2491aMerge branch 't/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels' into t/30527/pep-503-simple-repository-for-wheels
4e3a2c3Merge branch 't/30527/pep-503-simple-repository-for-wheels' into t/30578/add-src-requirements-txt
sagetrac-git commented

Changed commit from 63490fd to 4e3a2c3

sagetrac-git commented

Branch pushed to git repo; I updated commit sha1. New commits:

659f723Merge tag '9.2.rc2' into t/30578/add-src-requirements-txt
sagetrac-git commented

Changed commit from 4e3a2c3 to 659f723

mkoeppe commented

Work Issues: rebase on #30779

mkoeppe commented

Changed dependencies from #29500, #30527, #30658, #22731 to #30527, #22731 | #30779

mkoeppe commented

Changed work issues from rebase on #30779 to none

mkoeppe commented

Changed dependencies from #30527, #22731 | #30779 to #30527, #22731, #30780

sagetrac-git commented

Changed commit from 659f723 to 22a92ce

sagetrac-git commented

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

f82f345Merge branch 't/22731/22731' into t/30578/add-src-requirements-txt
7980130Merge branch 't/30779/duplicate_src_setup_py' into t/30780/sage_setup__separate_installing_and_cleaning
9b95628sage_setup.command.sage_install.sage_install: Rename to sage_install_and_clean, update use it in build/pkgs/sagelib/src/setup.py; refactor through new class sage_install
ad9402fsage_setup.command.sage_install: Fixup: Remove cleaning code (moved to sage_install_and_clean)
e2fa705Merge branch 't/30780/sage_setup__separate_installing_and_cleaning' into t/30578/add-src-requirements-txt
3557bb8src/requirements.txt, src/MANIFEST.in: Move here from build/pkgs/sagelib/src, leave symlinks behind
c584408WIP
343ebfbsrc/requirements.txt: Add sage_conf
1e766ecsrc/tox.ini (develop): New
22a92cesrc/setup.py: Switch to setuptools, add setup.cfg, install-requires.txt
mkoeppe commented

Description changed:

--- 
+++ 
@@ -4,7 +4,7 @@
 (This duplicates some information also available in `build/pkgs`; #29041 will rectify this.)
 
 We add a tox environment to `src/tox.ini`, enabling the command
-`../sage -sh -c 'tox -e develop'` (with #30410: `../sage -tox -e develop`) for building sagelib in a venv.
+`../sage -sh -c 'tox -e develop'` (with #30410: `../sage -tox -e develop`) for building sagelib in a venv.  (This requires `sage -i pip2pi` from #30527.)
 
 We also add documentation on how to set up the virtual environment, using either plain `python -m venv`, `virtualenv` (20.x), or `pipenv`.
sagetrac-git commented

Branch pushed to git repo; I updated commit sha1. New commits:

9e9b419src/requirements.txt, src/setup.cfg: Fix up
sagetrac-git commented

Changed commit from 22a92ce to 9e9b419

mkoeppe commented

Description changed:

--- 
+++ 
@@ -4,7 +4,7 @@
 (This duplicates some information also available in `build/pkgs`; #29041 will rectify this.)
 
 We add a tox environment to `src/tox.ini`, enabling the command
-`../sage -sh -c 'tox -e develop'` (with #30410: `../sage -tox -e develop`) for building sagelib in a venv.  (This requires `sage -i pip2pi` from #30527.)
+`../sage -tox -e develop` for building sagelib in a venv.  (This requires `sage -i pip2pi` from #30527.)
 
 We also add documentation on how to set up the virtual environment, using either plain `python -m venv`, `virtualenv` (20.x), or `pipenv`.
mkoeppe commented

Description changed:

--- 
+++ 
@@ -1,15 +1,17 @@
 After #30527, it is possible to install Python packages built by the Sage distribution into user-defined venvs.  
 
-In this ticket, we create `src/requirements.txt` to facilitate this.
+In this ticket, we create `build/pkgs/sagelib/src/requirements.txt -> 
+ src/requirements.txt` to facilitate this.
 (This duplicates some information also available in `build/pkgs`; #29041 will rectify this.)
 
-We add a tox environment to `src/tox.ini`, enabling the command
-`../sage -tox -e develop` for building sagelib in a venv.  (This requires `sage -i pip2pi` from #30527.)
+A virtual environment using it can be created using:
+./sage -sh -c '(cd build/pkgs/sagelib/src && tox -v -v -v)'
+(This requires `sage -i pip2pi` from #30527.)
+
+
+(We also add a tox environment `develop` to `src/tox.ini`, enabling the command `../sage -tox -e develop` for building sagelib in a venv - but this needs further work in #30371.)
 
 We also add documentation on how to set up the virtual environment, using either plain `python -m venv`, `virtualenv` (20.x), or `pipenv`.
-
-The existing `src/setup.py` is copied into `build/pkgs/sagelib/src`  to replace the symlink. In this way, the changes to `src/setup.py` made in this ticket do not affect the existing build process of the Sage library.
-
 
 Related:
 - #28746 Developer's Guide: Add instructions for sagelib development in a conda environment
mkoeppe commented

Description changed:

--- 
+++ 
@@ -1,8 +1,7 @@
 After #30527, it is possible to install Python packages built by the Sage distribution into user-defined venvs.  
 
-In this ticket, we create `build/pkgs/sagelib/src/requirements.txt -> 
- src/requirements.txt` to facilitate this.
-(This duplicates some information also available in `build/pkgs`; #29041 will rectify this.)
+In this ticket, we create `build/pkgs/sagelib/src/requirements.txt -> src/requirements.txt` to facilitate this.
+(This file duplicates some information also available in `build/pkgs`; #29041 will rectify this.)
 
 A virtual environment using it can be created using:
 ./sage -sh -c '(cd build/pkgs/sagelib/src && tox -v -v -v)'
mkoeppe commented

Changed dependencies from #30527, #22731, #30780 to #30527, #22731, #30780, #30013

sagetrac-git commented

Changed commit from 9e9b419 to a4fd284

sagetrac-git commented

Branch pushed to git repo; I updated commit sha1. New commits:

4dd9075build/pkgs/sagelib/src: Add more symlinks to SAGE_ROOT/src
a4fd284WIP
mkoeppe commented

Description changed:

--- 
+++ 
@@ -4,7 +4,10 @@
 (This file duplicates some information also available in `build/pkgs`; #29041 will rectify this.)
 
 A virtual environment using it can be created using:
+
+```
 ./sage -sh -c '(cd build/pkgs/sagelib/src && tox -v -v -v)'
+```
 (This requires `sage -i pip2pi` from #30527.)
sagetrac-git commented

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

f6f1e97Merge branch 't/22731/22731' into t/30578/add-src-requirements-txt
1a93d0dMerge branch 't/30780/sage_setup__separate_installing_and_cleaning' into t/30578/add-src-requirements-txt
a796d49src/requirements.txt, src/MANIFEST.in: Move here from build/pkgs/sagelib/src, leave symlinks behind
aebb1ceWIP
95f02a1src/requirements.txt: Add sage_conf
8e12b7csrc/tox.ini (develop): New
9ddaf26src/setup.py: Switch to setuptools, add setup.cfg, install-requires.txt
8a5d01fsrc/requirements.txt, src/setup.cfg: Fix up
7f9388dbuild/pkgs/sagelib/src: Add more symlinks to SAGE_ROOT/src
d3f3683WIP
sagetrac-git commented

Changed commit from a4fd284 to d3f3683

mkoeppe commented

Author: Matthias Koeppe

mkoeppe commented
comment:30

Now, with #30013 merged, the correct sage script is executed in the venv created by tox.

What remains to be done is to add more packages to requirements.txt - in particular ipywidgets so that the doctester can run.

sagetrac-git commented

Changed commit from d3f3683 to 8c03c6b

sagetrac-git commented

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

842995fsrc/bin/sage-venv-config: Add comment
1a518c0src/bin/sage: Set SAGE_VENV from sage-venv-config
d372ecfsrc/bin/sage-env: Put $SAGE_VENV/bin before $SAGE_LOCAL/bin in PATH
0140f84src/bin/sage-src-env-config.in: New, sourced in src/bin/sage
948fb49Merge branch 'u/mkoeppe/src_bin_sage_env__make_sure__sage_venv_bin_is_at_the_beginning_of_the_path' of git://trac.sagemath.org/sage into t/30578/add-src-requirements-txt
e96dc7asrc/requirements.txt, src/MANIFEST.in: Move here from build/pkgs/sagelib/src, leave symlinks behind
367fd3esrc/tox.ini (develop): New
0c5a1a1WIP
7121cb7src/requirements.txt: Add sage_conf
8c03c6bsrc/setup.py: Switch to setuptools, add setup.cfg, install-requires.txt
mkoeppe commented

Description changed:

--- 
+++ 
@@ -1,19 +1,13 @@
 After #30527, it is possible to install Python packages built by the Sage distribution into user-defined venvs.  
 
-In this ticket, we create `build/pkgs/sagelib/src/requirements.txt -> src/requirements.txt` to facilitate this.
-(This file duplicates some information also available in `build/pkgs`; #29041 will rectify this.)
+In this ticket, we create `build/pkgs/sagelib/src/requirements.txt -> src/requirements.txt` to facilitate installing sagelib into such a venv. (This file duplicates some information also available in `build/pkgs`; #29041 will rectify this.)
 
-A virtual environment using it can be created using:
+A virtual environment using it can be created and doctested using:
 
 ```
 ./sage -sh -c '(cd build/pkgs/sagelib/src && tox -v -v -v)'
 ```
-(This requires `sage -i pip2pi` from #30527.)
-
-
-(We also add a tox environment `develop` to `src/tox.ini`, enabling the command `../sage -tox -e develop` for building sagelib in a venv - but this needs further work in #30371.)
-
-We also add documentation on how to set up the virtual environment, using either plain `python -m venv`, `virtualenv` (20.x), or `pipenv`.
+(Reusing wheels built by the Sage distribution requires to do `sage -i pip2pi` - from #30527.)
 
 Related:
 - #28746 Developer's Guide: Add instructions for sagelib development in a conda environment
@@ -22,3 +16,5 @@
 Follow-up tickets:
 - #30371 In-place (editable) installs of sagelib in a venv
 - #29041 at `./bootstrap` time, generate `requirements.txt`, `constraints.txt`, `setup.cfg [install_requires]`, `Pipfile` from `build/pkgs`
+- add documentation on how to set up the virtual environment, using either plain `python -m venv`, `virtualenv` (20.x), or `pipenv`.
+
sagetrac-git commented

Changed commit from 8c03c6b to e2d498f

sagetrac-git commented

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

55d59f8src/requirements.txt: Add more packages
e2d498fbuild/pkgs/sagelib/src/tox.ini: Update tests and instructions
mkoeppe commented
comment:34

This works; only minor issues in doctests remain.

File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/src/sage/interfaces/gap_workspace.py", line 55, in sage.interfaces.gap_workspace.gap_workspace_file
Failed example:
    name1 = Popen([sys.executable, '-c', cmd], stdout=PIPE).communicate()[0]
Expected nothing
Got:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/build/pkgs/sagelib/src/sage/all.py", line 125, in <module>
        import sage.misc.lazy_import
    ModuleNotFoundError: No module named 'sage.misc.lazy_import'

There are also some doctest errors in the pexpect interfaces for singular and gap.

mkoeppe commented

Description changed:

--- 
+++ 
@@ -1,13 +1,19 @@
 After #30527, it is possible to install Python packages built by the Sage distribution into user-defined venvs.  
 
-In this ticket, we create `build/pkgs/sagelib/src/requirements.txt -> src/requirements.txt` to facilitate installing sagelib into such a venv. (This file duplicates some information also available in `build/pkgs`; #29041 will rectify this.)
+In this ticket, we create `build/pkgs/sagelib/src/requirements.txt -> src/requirements.txt` to facilitate this.
+(This file duplicates some information also available in `build/pkgs`; #29041 will rectify this.)
 
-A virtual environment using it can be created and doctested using:
+A virtual environment using it can be created using:
 
 ```
 ./sage -sh -c '(cd build/pkgs/sagelib/src && tox -v -v -v)'
 ```
-(Reusing wheels built by the Sage distribution requires to do `sage -i pip2pi` - from #30527.)
+(This requires `sage -i pip2pi` from #30527.)
+
+
+(We also add a tox environment `develop` to `src/tox.ini`, enabling the command `../sage -tox -e develop` for building sagelib in a venv - but this needs further work in #30371.)
+
+We also add documentation on how to set up the virtual environment, using either plain `python -m venv`, `virtualenv` (20.x), or `pipenv`.
 
 Related:
 - #28746 Developer's Guide: Add instructions for sagelib development in a conda environment
@@ -16,5 +22,3 @@
 Follow-up tickets:
 - #30371 In-place (editable) installs of sagelib in a venv
 - #29041 at `./bootstrap` time, generate `requirements.txt`, `constraints.txt`, `setup.cfg [install_requires]`, `Pipfile` from `build/pkgs`
-- add documentation on how to set up the virtual environment, using either plain `python -m venv`, `virtualenv` (20.x), or `pipenv`.
-
mkoeppe commented

Description changed:

--- 
+++ 
@@ -1,19 +1,13 @@
 After #30527, it is possible to install Python packages built by the Sage distribution into user-defined venvs.  
 
-In this ticket, we create `build/pkgs/sagelib/src/requirements.txt -> src/requirements.txt` to facilitate this.
-(This file duplicates some information also available in `build/pkgs`; #29041 will rectify this.)
+In this ticket, we create `build/pkgs/sagelib/src/requirements.txt -> src/requirements.txt` to facilitate installing sagelib into such a venv. (This file duplicates some information also available in `build/pkgs`; #29041 will rectify this.)
 
-A virtual environment using it can be created using:
+A virtual environment using it can be created and doctested using:
 
 ```
 ./sage -sh -c '(cd build/pkgs/sagelib/src && tox -v -v -v)'
 ```
-(This requires `sage -i pip2pi` from #30527.)
-
-
-(We also add a tox environment `develop` to `src/tox.ini`, enabling the command `../sage -tox -e develop` for building sagelib in a venv - but this needs further work in #30371.)
-
-We also add documentation on how to set up the virtual environment, using either plain `python -m venv`, `virtualenv` (20.x), or `pipenv`.
+(Reusing wheels built by the Sage distribution requires to do `sage -i pip2pi` - from #30527.)
 
 Related:
 - #28746 Developer's Guide: Add instructions for sagelib development in a conda environment
@@ -22,3 +16,5 @@
 Follow-up tickets:
 - #30371 In-place (editable) installs of sagelib in a venv
 - #29041 at `./bootstrap` time, generate `requirements.txt`, `constraints.txt`, `setup.cfg [install_requires]`, `Pipfile` from `build/pkgs`
+- add documentation on how to set up the virtual environment, using either plain `python -m venv`, `virtualenv` (20.x), or `pipenv`.
+
mkoeppe commented
comment:36

The fpylll package unfortunately does not install because it does not declare its build dependencies correctly. Reported upstream: fplll/fpylll#185

sagetrac-git commented

Changed commit from e2d498f to db148ac

sagetrac-git commented

Branch pushed to git repo; I updated commit sha1. New commits:

db148acsrc/requirements.txt: Disable fpylll
sagetrac-git commented

Branch pushed to git repo; I updated commit sha1. New commits:

9ad6b36build/pkgs/sagelib/src/setup.cfg: Remove
sagetrac-git commented

Changed commit from db148ac to 9ad6b36

mkoeppe commented

Description changed:

--- 
+++ 
@@ -7,7 +7,7 @@
 ```
 ./sage -sh -c '(cd build/pkgs/sagelib/src && tox -v -v -v)'
 ```
-(Reusing wheels built by the Sage distribution requires to do `sage -i pip2pi` - from #30527.)
+(Reusing wheels built by the Sage distribution requires to do `sage -i pip2pi` first - this is from #30527.)
 
 Related:
 - #28746 Developer's Guide: Add instructions for sagelib development in a conda environment
mkoeppe commented

Changed keywords from none to sd111

mkoeppe commented

Changed dependencies from #30527, #22731, #30780, #30013 to #30527, #30013

mkoeppe commented

Changed author from Matthias Koeppe to none

mkoeppe commented
comment:43

Let's skip this ticket and do #30913 directly.