Build sagelib from build/pkgs/sagelib/src, fix `setup.py sdist`, add spkg-src and tox.ini

Question

Build sagelib from build/pkgs/sagelib/src, fix `setup.py sdist`, add spkg-src and tox.ini

Closed this issue 5 years ago · 57 comments

As a follow-up to #29411, we set up a separate directory build/pkgs/sagelib/src with symlinks into SAGE_ROOT/src so that sagelib's setup.py does not have access to the whole tree SAGE_ROOT/src.

We fix setup.py sdist, which can be tested using spkg-src (which builds a pip-installable tarball in upstream/) and tox:

./sage -sh -c '(cd build/pkgs/sagelib/src && tox -v -v)'
./sage -sh -c 'build/pkgs/sagelib/src/.tox/python/bin/sage-ipython'

The call of ./sage -sh is necessary so that SAGE_LOCAL is set.
Moreover, for scripts that use sage-env, also SAGE_ENV_SOURCED is crucial for the correct operation; otherwise sage-env sets things based on SAGE_LOCAL again.

Depends on #29701

CC: @kiwifb @jhpalmieri @dimpase @kliem @isuruf @vbraun

Component: refactoring

Author: Matthias Koeppe

Branch/Commit: ccc67b0

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/29950

Answer 1 · 2020-06-24T02:28:00.000Z

Branch: u/mkoeppe/build_sagelib_using_installed_sage_setup

Answer 2 · 2020-06-24T02:32:23.000Z

Commit: 1ab75a9

Answer 3 · 2020-06-24T02:32:23.000Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`0459006`	`.gitignore: Ignore generated files in build/pkgs/*/src`
`1ab75a9`	`Build sagelib using the installed sage_setup`

Answer 4 · 2020-06-24T02:47:50.000Z

Work Issues: add spkg-src

Answer 5 · 2020-06-24T05:04:05.000Z

Author: Matthias Koeppe

Answer 6 · 2020-06-25T06:32:34.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`3d83c12`	`build/pkgs/sagelib/spkg-src: New`
`a2bde50`	`src/MANIFEST.in: prune .tox`
`4b258d3`	`src/MANIFEST.in: Remove bitrotted file`
`a29cd2d`	`build/pkgs/sagelib/MANIFEST.in: New`

Answer 7 · 2020-06-25T06:32:34.000Z

Changed commit from 1ab75a9 to a29cd2d

Answer 8 · 2020-06-25T06:45:16.000Z

Changed work issues from add spkg-src to Fixup MANIFEST.in

Answer 9 · 2020-06-28T16:25:48.000Z

comment:8

sdist is broken: Files named ** end up in the tar file.

Answer 10 · 2020-06-28T16:26:49.000Z

Changed dependencies from #29411, #29847 to #29411, #29847, #29701

Answer 11 · 2020-06-28T18:18:26.000Z

comment:10

sagemath/cypari2#93

Answer 12 · 2020-06-28T18:26:50.000Z

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`7f8850a`	`src/module_list.py: Document that it is obsolete`
`26a85a4`	`Merge branch 't/29702/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup' into t/29701/replace_use_of_module_list_optionalextension`
`5d5803e`	`src/sage/graphs/planarity.pyx: Add forgotten distutils directive`
`a0be9b6`	`Merge branch 't/29790/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_5__sage_graphs_' into t/29701/replace_use_of_module_list_optionalextension`
`14396af`	`Merge tag '9.2.beta2' into t/29701/replace_use_of_module_list_optionalextension`
`821e337`	`Merge branch 't/29701/replace_use_of_module_list_optionalextension' into t/29950/build_sagelib_using_installed_sage_setup`
`eeb6ba4`	`build/pkgs/sagelib/src/requirements.txt, tox.ini: New`
`ce8b84d`	`build/pkgs/sagelib/src/MANIFEST.in: Prune sage/ext/interpreters`
`a9a1335`	`build/pkgs/sagelib/src/requirements.txt: Add packages required for runtime`
`c3b8f8f`	`Suggest tox -v -v`

Answer 13 · 2020-06-28T18:26:50.000Z

Changed commit from a29cd2d to c3b8f8f

Answer 14 · 2020-06-28T18:29:11.000Z

Description changed:

--- 
+++ 
@@ -1,2 +1,8 @@
-After #29411 and #29847, we set up a separate directory `build/pkgs/sagelib/src` with symlinks into `SAGE_ROOT/src` so that sagelib's `setup.py` does not have access to the whole tree `SAGE_ROOT/src`. In particular, we arrange for the installed `sage_setup` (#29847), rather than the one from the source tree, to be used.
+As a follow-up to #29411, we set up a separate directory `build/pkgs/sagelib/src` with symlinks into `SAGE_ROOT/src` so that sagelib's `setup.py` does not have access to the whole tree `SAGE_ROOT/src`.
 
+We fix `setup.py sdist`, which can be tested using `spkg-src` and `tox`.
+
+```
+./sage -sh -c '(cd build/pkgs/sagelib/src && tox -v -v)'
+```
+

Answer 15 · 2020-06-28T18:29:11.000Z

Changed work issues from Fixup MANIFEST.in to Rebase away from #29847

Answer 16 · 2020-06-28T19:04:05.000Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`cdf501f`	`build/pkgs/sagelib/spkg-src: New`
`ef4f5b8`	`Move src/MANIFEST.in to build/pkgs/sagelib/MANIFEST.in, fix up`
`233f6ba`	`build/pkgs/sagelib/src/requirements.txt, tox.ini: New`
`177cd4c`	`build/pkgs/sagelib/src/MANIFEST.in: Prune sage/ext/interpreters`
`07b29fb`	`build/pkgs/sagelib/src/requirements.txt: Add packages required for runtime`
`ad33705`	`Suggest tox -v -v`
`481e212`	`.gitignore: Ignore generated files in build/pkgs/*/src`
`9955a3f`	`Build sagelib from build/pkgs/sagelib/src`

Answer 17 · 2020-06-28T19:04:05.000Z

Changed commit from c3b8f8f to 9955a3f

Answer 18 · 2020-06-28T19:04:51.000Z

Changed dependencies from #29411, #29847, #29701 to #29411, #29701

Answer 19 · 2020-06-28T19:04:51.000Z

Changed work issues from Rebase away from #29847 to none

Answer 20 · 2020-06-28T19:07:03.000Z

comment:15

    from .maxima import maxima, Maxima
  File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/build/pkgs/sagelib/src/.tox/python/lib/python3.7/site-packages/sage/interfaces/maxima.py", line 1236, in <module>
    script_subdirectory=None)
  File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/build/pkgs/sagelib/src/.tox/python/lib/python3.7/site-packages/sage/interfaces/maxima.py", line 528, in __init__
    raise RuntimeError('You must get the file local/bin/sage-maxima.lisp')
RuntimeError: You must get the file local/bin/sage-maxima.lisp

Answer 21 · 2020-06-28T19:14:35.000Z

Changed dependencies from #29411, #29701 to #29411, #29701, #27171

Answer 22 · 2020-06-28T19:18:51.000Z

Changed dependencies from #29411, #29701, #27171 to #29411, #29701, #21559

Answer 23 · 2020-06-28T19:19:31.000Z

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`875940d`	`trac 29111: more tinkering. Fix doctests in tests/cmdline.py.`
`4a3d36e`	`Move 'sage -app' back to src/bin/sage`
`3a0193c`	`src/bin/sage: Remove handling of 'sage -axiom'`
`6b04075`	`Merge branch 't/29111/specify_a_subset_of_sage_command_line_options_that_are_supported_by_sagelib___rather_than_sage_the_distribution' into t/21559/change-src-bin-installation`
`9c7116b`	`src/bin/sage-list-optional, sage-list-experimental, sage-list-standard: Remove deprecated scripts`
`831cc09`	`Merge branch 't/29920/remove_deprecated_scripts_sage_list_optional__sage_list_experimental__sage_list_standard' into t/21559/change-src-bin-installation`
`a56dc35`	`Merge tag '9.2.beta1' into t/29702/public/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup`
`e3eca85`	`Merge branch 'public/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup' of git://trac.sagemath.org/sage into t/21559/change-src-bin-installation`
`7d29141`	`src/setup.py: Do not install removed script sage-rsyncdist`
`08f5175`	`Merge branch 'u/mkoeppe/change-src-bin-installation' of git://trac.sagemath.org/sage into t/29950/build_sagelib_using_installed_sage_setup`

Answer 24 · 2020-06-28T19:19:31.000Z

Changed commit from 9955a3f to 08f5175

Answer 25 · 2020-06-28T19:31:18.000Z

Description changed:

--- 
+++ 
@@ -1,6 +1,6 @@
 As a follow-up to #29411, we set up a separate directory `build/pkgs/sagelib/src` with symlinks into `SAGE_ROOT/src` so that sagelib's `setup.py` does not have access to the whole tree `SAGE_ROOT/src`.
 
-We fix `setup.py sdist`, which can be tested using `spkg-src` and `tox`.
+We fix `setup.py sdist`, which can be tested using `spkg-src` (which builds a pip-installable tarball in `upstream/`) and `tox`:
 
 ```
 ./sage -sh -c '(cd build/pkgs/sagelib/src && tox -v -v)'

Answer 26 · 2020-06-28T20:36:14.000Z

comment:20

./sage -sh -c '(cd build/pkgs/sagelib/src && tox -r -v -v)'
...
RuntimeError: libSingular not found--a working Singular install in $SAGE_LOCAL is required for Sage to work
ERROR: InvocationError for command /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/build/pkgs/sagelib/src/.tox/python/bin/python -c 'import sys; "" in sys.path and sys.path.remove(""); import sage.all' (exited with code 1)

This is because of:

$ build/pkgs/sagelib/src/.tox/python/bin/python
Python 3.7.7 (v3.7.7:d7c567b08f, Mar 10 2020, 02:56:16) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sage.env
>>> sage.env.SINGULAR_SO
>>> sage.env.GAP_SO
>>> sage.env.SAGE_LOCAL
'/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/build/pkgs/sagelib/src/.tox/python'
>>>

Answer 27 · 2020-06-28T20:48:08.000Z

comment:21

Passing the correct SAGE_LOCAL suffices to fix this, of course. But the singular library location should be configured separately (#29024).

Answer 28 · 2020-06-28T20:48:38.000Z

Changed commit from 08f5175 to 516358b

Answer 29 · 2020-06-28T20:48:38.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`516358b`	`build/pkgs/sagelib/src/tox.ini: Pass SAGE_LOCAL`

Answer 30 · 2020-06-28T22:52:27.000Z

Changed commit from 516358b to eaced09

Answer 31 · 2020-06-28T22:52:27.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`d62da15`	`sage_setup.command.sage_build: Add the extensions to the distribution`
`5e577ab`	`Merge branch 't/29701/replace_use_of_module_list_optionalextension' into t/29950/build_sagelib_using_installed_sage_setup`
`eaced09`	`build/pkgs/sagelib: Add remaining Python dependencies for sage.all`

Answer 32 · 2020-06-28T23:03:33.000Z

Description changed:

--- 
+++ 
@@ -4,5 +4,6 @@
 
 ```
 ./sage -sh -c '(cd build/pkgs/sagelib/src && tox -v -v)'
+./sage -sh -c 'build/pkgs/sagelib/src/.tox/python/bin/sage-ipython'
 ```

Answer 33 · 2020-06-28T23:08:44.000Z

Description changed:

--- 
+++ 
@@ -7,3 +7,5 @@
 ./sage -sh -c 'build/pkgs/sagelib/src/.tox/python/bin/sage-ipython'
 ```
 
+Scripts installed into the tox virtual environment that use `sage-env` do not work yet -- `sage-env` sets things based on `SAGE_LOCAL` again.
+

Answer 34 · 2020-06-28T23:54:45.000Z

Description changed:

--- 
+++ 
@@ -7,5 +7,5 @@
 ./sage -sh -c 'build/pkgs/sagelib/src/.tox/python/bin/sage-ipython'
 ```
 
-Scripts installed into the tox virtual environment that use `sage-env` do not work yet -- `sage-env` sets things based on `SAGE_LOCAL` again.
-
+The call of `./sage -sh` is necessary so that `SAGE_LOCAL` is set. 
+Moreover, for scripts that use `sage-env`, also `SAGE_ENV_SOURCED` is crucial for the correct operation; otherwise `sage-env` sets things based on `SAGE_LOCAL` again.

Answer 35 · 2020-06-28T23:54:57.000Z

comment:27

The sage script relies on PATH to be set correctly when invoking auxiliary scripts such as sage-eval. Hence the following does not work correctly:

$ ./sage -sh -c 'build/pkgs/sagelib/src/.tox/python/bin/sage -c "print(sys.path)"'
['/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/src/bin', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python37.zip', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/lib-dynload', '/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages']

Answer 36 · 2020-06-29T00:10:35.000Z

comment:28

This will be fixed by #30013 (src/bin/sage-env: Make sure SAGE_SCRIPTS_DIR is at the beginning of the PATH).

Answer 37 · 2020-06-29T00:11:18.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`b8e366e`	`build/pkgs/sagelib/spkg-src: chmod +x`

Answer 38 · 2020-06-29T00:11:18.000Z

Changed commit from eaced09 to b8e366e

Answer 39 · 2020-07-02T20:14:58.000Z

comment:31

All dependencies of this ticket are green. Ready for review.

Answer 40 · 2020-07-04T16:30:44.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`c5f7f2f`	`Merge tag '9.2.beta3' into t/21559/change-src-bin-installation`
`0473ef3`	`Merge branch 't/21559/change-src-bin-installation' into t/29950/build_sagelib_using_installed_sage_setup`
`06a3609`	`sage_setup.find.find_python_sources: Add benchmark doctest`
`174626c`	`Merge branch 't/29786/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_4__sage_rings_' into t/29701/replace_use_of_module_list_optionalextension`
`034a7f7`	`Merge branch 'public/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup' of git://trac.sagemath.org/sage into t/29701/replace_use_of_module_list_optionalextension`
`d3c608b`	`Merge tag '9.2.beta3' into t/29701/replace_use_of_module_list_optionalextension`
`7244371`	`Merge branch 't/29701/replace_use_of_module_list_optionalextension' into t/29950/build_sagelib_using_installed_sage_setup`

Answer 41 · 2020-07-04T16:30:44.000Z

Changed commit from b8e366e to 7244371

Answer 42 · 2020-07-04T16:31:17.000Z

comment:33

Merged in latest versions #29701, #21559 (on top of 9.2.beta3). Needs review

Answer 43 · 2020-07-08T23:26:36.000Z

Changed commit from 7244371 to 4344f89

Answer 44 · 2020-07-08T23:26:36.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`3953671`	`Merge tag '9.2.beta4' into t/29111/specify_a_subset_of_sage_command_line_options_that_are_supported_by_sagelib___rather_than_sage_the_distribution`
`19d7f67`	`Merge branch 't/29111/specify_a_subset_of_sage_command_line_options_that_are_supported_by_sagelib___rather_than_sage_the_distribution' into t/21559/change-src-bin-installation`
`4344f89`	`Merge branch 't/21559/change-src-bin-installation' into t/29950/build_sagelib_using_installed_sage_setup`

Answer 45 · 2020-08-03T00:53:28.000Z

Changed commit from 4344f89 to 01b96b0

Answer 46 · 2020-08-03T00:53:28.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`fcad518`	`Merge tag '9.2.beta7' into t/29701/replace_use_of_module_list_optionalextension`
`55c3fbc`	`src/sage_setup/clean.py: Fix doctest`
`01b96b0`	`Merge branch 't/29701/replace_use_of_module_list_optionalextension' into t/29950/build_sagelib_using_installed_sage_setup`

Answer 47 · 2020-08-03T00:54:01.000Z

Changed dependencies from #29411, #29701, #21559 to #29701

Answer 48 · 2020-08-05T08:22:44.000Z

comment:37

lgtm

Answer 49 · 2020-08-05T08:22:44.000Z

Reviewer: Dima Pasechnik

Answer 50 · 2020-08-08T20:50:05.000Z

comment:38

Failing doctest needs work as noted in #29701.

Answer 51 · 2020-08-08T22:50:02.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`8a19fe2`	`build/make/Makefile.in (sagelib-clean): Clean the new build location`

Answer 52 · 2020-08-08T22:50:02.000Z

Changed commit from 01b96b0 to 8a19fe2

Answer 53 · 2020-08-08T23:08:03.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`ccc67b0`	`src/sage_setup: Update cythonized_dir in doctests`

Answer 54 · 2020-08-08T23:08:03.000Z

Changed commit from 8a19fe2 to ccc67b0

Answer 55 · 2020-08-10T16:45:25.000Z

comment:42

ok

Answer 56 · 2020-08-10T16:51:20.000Z

comment:43

Thanks!

Answer 57 · 2020-08-12T19:53:18.000Z

Changed branch from u/mkoeppe/build_sagelib_using_installed_sage_setup to ccc67b0