New make targets "sagelib-tox-...", "sage_docbuild-tox-...", ...

Question

New make targets "sagelib-tox-...", "sage_docbuild-tox-...", ...

mkoeppe opened this issue 5 years ago · 34 comments

For all script packages SPKG that have an embedded source tree -- currently sagelib, sage_docbuild, sage_sws2rst -- we add a new target:

SPKG-tox-% (which calls tox -e % in the package's source tree)

These targets are particularly useful for invoking them within the portability tests of the top-level tox:

$ TARGETS_PRE=config.status tox -e docker-ubuntu-focal-standard -- sage_docbuild-tox invokes the tox testing of the sage_docbuild distribution
$ EXTRA_CONFIGURE_ARGS="--disable-notebook" tox -e docker-ubuntu-focal-standard -- build sagelib-tox-python-sagewheels-nopypi builds (in a Docker container) the Sage distribution (without Jupyter notebook) and then invokes the tox testing of sagelib (added in #30913) in a venv populated using the wheels
$ tox -e local-homebrew-macos-maximal -- build-local cysignals networkx sagelib-tox-python-sagewheels-sitepackages-norequirements-nopypi (added in #30913) builds the non-Python parts of the Sage distribution on top of Homebrew, using a virtual environment that uses homebrew's Python packages
$ tox -e local-conda-forge-macos-src-environment -- sage_conf sagelib-tox-python-sagewheels-sitepackages-norequirements-nobuildisolation-nopypi (needs #30913, #31384) uses the src/environment.yml file to set up a complete environment from conda and then tests sagelib in a venv using tox

Depends on #32899

CC: @jhpalmieri @dimpase @kliem @orlitzky @tobiasdiez

Component: porting

Author: Matthias Koeppe

Branch/Commit: ad12d25

Reviewer: Michael Orlitzky

Issue created by migration from https://trac.sagemath.org/ticket/31535

Answer 1 · 2021-03-22T03:02:47.000Z

Branch: u/mkoeppe/new_make_targets__sagelib_sdist____sage_docbuild_sdist_______and__sagelib_tox__________

Answer 2 · 2021-03-22T03:03:30.000Z

Author: Matthias Koeppe

Answer 3 · 2021-03-22T03:03:30.000Z

New commits:

`1eef6d6`	`build/make/Makefile.in: Add targets SPKG-sdist, SPKG-tox, SPKG-tox-% for script packages`

Answer 4 · 2021-03-22T03:03:30.000Z

Commit: 1eef6d6

Answer 5 · 2021-03-22T04:01:18.000Z

Changed commit from 1eef6d6 to 6964933

Answer 6 · 2021-03-22T04:01:18.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`6964933`	`build/make/Makefile.in: fixup`

Answer 7 · 2021-03-22T04:03:29.000Z

Description changed:

--- 
+++ 
@@ -7,5 +7,5 @@
 The new `SPKG-tox-%` targets are particularly useful for invoking them within the portability tests of the top-level `tox`:
 - `$ TARGETS_PRE=config.status tox -e docker-ubuntu-focal-standard -- sage_docbuild-tox` invokes the tox testing of the `sage_docbuild` distribution
 - `$ EXTRA_CONFIGURE_ARGS="--disable-notebook" tox -e docker-ubuntu-focal-standard -- build sagelib-tox-python-sagewheels-nopypi` builds (in a Docker container) the Sage distribution (without Jupyter notebook) and then invokes the tox testing of sagelib (added in #30913) in a venv populated using the wheels
-- `$ EXTRA_CONFIGURE_ARGS="--disable-notebook" TARGETS_PRE=config.status tox -e local-homebrew-macos-maximal -- build-local sagelib-tox-python-sitepackages-nopypi` (added in #30913) builds the non-Python parts of the Sage distribution on top of Homebrew, using a virtual environment that uses homebrew's Python packages
+- `$ EXTRA_CONFIGURE_ARGS="--disable-notebook" TARGETS_PRE=config.status tox -e local-homebrew-macos-maximal -- build-local sagelib-tox-python-sagewheels-sitepackages-nopypi` (added in #30913) builds the non-Python parts of the Sage distribution on top of Homebrew, using a virtual environment that uses homebrew's Python packages

Answer 8 · 2021-03-22T05:18:22.000Z

Description changed:

--- 
+++ 
@@ -7,5 +7,5 @@
 The new `SPKG-tox-%` targets are particularly useful for invoking them within the portability tests of the top-level `tox`:
 - `$ TARGETS_PRE=config.status tox -e docker-ubuntu-focal-standard -- sage_docbuild-tox` invokes the tox testing of the `sage_docbuild` distribution
 - `$ EXTRA_CONFIGURE_ARGS="--disable-notebook" tox -e docker-ubuntu-focal-standard -- build sagelib-tox-python-sagewheels-nopypi` builds (in a Docker container) the Sage distribution (without Jupyter notebook) and then invokes the tox testing of sagelib (added in #30913) in a venv populated using the wheels
-- `$ EXTRA_CONFIGURE_ARGS="--disable-notebook" TARGETS_PRE=config.status tox -e local-homebrew-macos-maximal -- build-local sagelib-tox-python-sagewheels-sitepackages-nopypi` (added in #30913) builds the non-Python parts of the Sage distribution on top of Homebrew, using a virtual environment that uses homebrew's Python packages
-
+- `$ tox -e local-homebrew-macos-maximal -- build-local cysignals networkx sagelib-tox-python-sagewheels-sitepackages-norequirements-nopypi` (added in #30913) builds the non-Python parts of the Sage distribution on top of Homebrew, using a virtual environment that uses homebrew's Python packages
+- `tox -e local-conda-forge-macos-src-environment -- sage_conf sagelib-tox-python-sagewheels-sitepackages-norequirements-nopypi` uses the `src/environment.yml` file to set up a complete environment from conda and then tests sagelib in a venv using tox

Answer 9 · 2021-03-22T05:18:38.000Z

Description changed:

--- 
+++ 
@@ -8,4 +8,4 @@
 - `$ TARGETS_PRE=config.status tox -e docker-ubuntu-focal-standard -- sage_docbuild-tox` invokes the tox testing of the `sage_docbuild` distribution
 - `$ EXTRA_CONFIGURE_ARGS="--disable-notebook" tox -e docker-ubuntu-focal-standard -- build sagelib-tox-python-sagewheels-nopypi` builds (in a Docker container) the Sage distribution (without Jupyter notebook) and then invokes the tox testing of sagelib (added in #30913) in a venv populated using the wheels
 - `$ tox -e local-homebrew-macos-maximal -- build-local cysignals networkx sagelib-tox-python-sagewheels-sitepackages-norequirements-nopypi` (added in #30913) builds the non-Python parts of the Sage distribution on top of Homebrew, using a virtual environment that uses homebrew's Python packages
-- `tox -e local-conda-forge-macos-src-environment -- sage_conf sagelib-tox-python-sagewheels-sitepackages-norequirements-nopypi` uses the `src/environment.yml` file to set up a complete environment from conda and then tests sagelib in a venv using tox
+- `$ tox -e local-conda-forge-macos-src-environment -- sage_conf sagelib-tox-python-sagewheels-sitepackages-norequirements-nopypi` uses the `src/environment.yml` file to set up a complete environment from conda and then tests sagelib in a venv using tox

Answer 10 · 2021-03-22T19:54:12.000Z

Changed commit from 6964933 to 188bf2e

Answer 11 · 2021-03-22T19:54:12.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`9993348`	`build/make/Makefile.in: Run targets SPKG-tox, SPKG-tox-% outside of sage-env, using SAGE_ORIG_PATH - so that tox from conda works`
`4533018`	`tox.ini: Add local-conda-environment-src`
`188bf2e`	`tox.ini, src/doc/bootstrap: Get tox into standard builds`

Answer 12 · 2021-03-22T20:01:40.000Z

Description changed:

--- 
+++ 
@@ -8,4 +8,4 @@
 - `$ TARGETS_PRE=config.status tox -e docker-ubuntu-focal-standard -- sage_docbuild-tox` invokes the tox testing of the `sage_docbuild` distribution
 - `$ EXTRA_CONFIGURE_ARGS="--disable-notebook" tox -e docker-ubuntu-focal-standard -- build sagelib-tox-python-sagewheels-nopypi` builds (in a Docker container) the Sage distribution (without Jupyter notebook) and then invokes the tox testing of sagelib (added in #30913) in a venv populated using the wheels
 - `$ tox -e local-homebrew-macos-maximal -- build-local cysignals networkx sagelib-tox-python-sagewheels-sitepackages-norequirements-nopypi` (added in #30913) builds the non-Python parts of the Sage distribution on top of Homebrew, using a virtual environment that uses homebrew's Python packages
-- `$ tox -e local-conda-forge-macos-src-environment -- sage_conf sagelib-tox-python-sagewheels-sitepackages-norequirements-nopypi` uses the `src/environment.yml` file to set up a complete environment from conda and then tests sagelib in a venv using tox
+- `$ tox -e local-conda-forge-macos-src-environment -- sage_conf sagelib-tox-python-sagewheels-sitepackages-norequirements-nobuildisolation-nopypi` uses the `src/environment.yml` file to set up a complete environment from conda and then tests sagelib in a venv using tox

Answer 13 · 2021-03-22T20:45:59.000Z

Description changed:

--- 
+++ 
@@ -8,4 +8,4 @@
 - `$ TARGETS_PRE=config.status tox -e docker-ubuntu-focal-standard -- sage_docbuild-tox` invokes the tox testing of the `sage_docbuild` distribution
 - `$ EXTRA_CONFIGURE_ARGS="--disable-notebook" tox -e docker-ubuntu-focal-standard -- build sagelib-tox-python-sagewheels-nopypi` builds (in a Docker container) the Sage distribution (without Jupyter notebook) and then invokes the tox testing of sagelib (added in #30913) in a venv populated using the wheels
 - `$ tox -e local-homebrew-macos-maximal -- build-local cysignals networkx sagelib-tox-python-sagewheels-sitepackages-norequirements-nopypi` (added in #30913) builds the non-Python parts of the Sage distribution on top of Homebrew, using a virtual environment that uses homebrew's Python packages
-- `$ tox -e local-conda-forge-macos-src-environment -- sage_conf sagelib-tox-python-sagewheels-sitepackages-norequirements-nobuildisolation-nopypi` uses the `src/environment.yml` file to set up a complete environment from conda and then tests sagelib in a venv using tox
+- `$ tox -e local-conda-forge-macos-src-environment -- sage_conf sagelib-tox-python-sagewheels-sitepackages-norequirements-nobuildisolation-nopypi` (needs #30913, #31384) uses the `src/environment.yml` file to set up a complete environment from conda and then tests sagelib in a venv using tox

Answer 14 · 2021-04-02T20:21:54.000Z

comment:11

Moving this ticket to 9.4, as it seems unlikely that it will be merged in 9.3, which is in the release candidate stage

Answer 15 · 2021-07-19T00:44:56.000Z

comment:12

Setting a new milestone for this ticket based on a cursory review.

Answer 16 · 2021-10-16T20:38:57.000Z

Dependencies: #32442, #32698

Answer 17 · 2021-10-16T20:44:34.000Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

`8af6041`	`Merge tag '9.5.beta2' into t/32442/__configure__make___with_sage_venv_the_default`
`b3f7d34`	`.gitignore: Add /venv`
`fc4b641`	`configure.ac: Remove conveniene symlinks before (re)creating them`
`8bcbf1d`	`Makefile (distclean): Remove convenience symlinks prefix, venv`
`dcb4a08`	`configure.ac: Indicate default in help string for --with-sage-venv`
`e33bc89`	`Merge tag '9.5.beta3' into t/32442/__configure__make___with_sage_venv_the_default`
`2101b8c`	`build/pkgs/python3/spkg-build.in: Make sure that python finds sqlite3 when determining which extension modules to build`
`3bbc5d8`	`build/pkgs/python3/spkg-build.in: Set rpath`
`124b605`	`Merge #32698`
`bca2141`	`pkgs/sagemath-standard/tox.ini: Use SAGE_VENV or venv symlink to find wheels`

Answer 18 · 2021-10-16T20:44:34.000Z

Changed commit from 188bf2e to bca2141

Answer 19 · 2021-10-16T20:57:09.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`bdbab3a`	`build/make/Makefile.in: Remove SPKG-tox, SPKG-sdist targets`

Answer 20 · 2021-10-16T20:57:09.000Z

Changed commit from bca2141 to bdbab3a

Answer 21 · 2021-10-16T20:58:10.000Z

comment:17

Reduced the scope of the ticket

Answer 22 · 2021-10-16T20:58:10.000Z

Description changed:

--- 
+++ 
@@ -1,10 +1,8 @@
-For all script packages `SPKG` that have an embedded source tree -- currently `sagelib`, `sage_docbuild`, `sage_sws2rst` -- we add new targets:
+For all script packages `SPKG` that have an embedded source tree -- currently `sagelib`, `sage_docbuild`, `sage_sws2rst` -- we add a new target:
 
-- `SPKG-sdist` (which makes an sdist via the `spkg-src` script)
-- `SPKG-tox`   (which calls `tox` in the package's source tree)
 - `SPKG-tox-%` (which calls `tox -e %` in the package's source tree)
 
-The new `SPKG-tox-%` targets are particularly useful for invoking them within the portability tests of the top-level `tox`:
+These targets are particularly useful for invoking them within the portability tests of the top-level `tox`:
 - `$ TARGETS_PRE=config.status tox -e docker-ubuntu-focal-standard -- sage_docbuild-tox` invokes the tox testing of the `sage_docbuild` distribution
 - `$ EXTRA_CONFIGURE_ARGS="--disable-notebook" tox -e docker-ubuntu-focal-standard -- build sagelib-tox-python-sagewheels-nopypi` builds (in a Docker container) the Sage distribution (without Jupyter notebook) and then invokes the tox testing of sagelib (added in #30913) in a venv populated using the wheels
 - `$ tox -e local-homebrew-macos-maximal -- build-local cysignals networkx sagelib-tox-python-sagewheels-sitepackages-norequirements-nopypi` (added in #30913) builds the non-Python parts of the Sage distribution on top of Homebrew, using a virtual environment that uses homebrew's Python packages

Answer 23 · 2021-12-01T12:12:07.000Z

comment:18

I've tried to test these locally but I always get some complaint about sage_setup.py. For example,

$ EXTRA_CONFIGURE_ARGS="--without-system-python3" tox -e local-direct -- build sagelib-tox-python-sagewheels

This proceeded to build everything inside of .tox/local-direct, but then...

cd '/home/mjo/src/sage.git/build/pkgs/sagelib/src' && export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/lib/llvm/12/bin:/home/mjo/bin:/home/mjo/bin" && SAGE_SPKG_WHEELS=/home/mjo/src/sage.git/.tox/local-direct/local/var/lib/sage/wheels tox -v -v -v -e python-sagewheels
using tox.ini: /home/mjo/src/sage.git/pkgs/sagemath-standard/tox.ini (pid 12133)
  removing /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/log
using tox-3.24.4 from /usr/lib/python3.9/site-packages/tox/__init__.py (pid 12133)
GLOB start: packaging 
GLOB sdist-make: /home/mjo/src/sage.git/pkgs/sagemath-standard/setup.py
[12140] /home/mjo/src/sage.git/pkgs/sagemath-standard$ /usr/bin/python3.9 setup.py sdist --formats=zip --dist-dir /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/dist >.tox/log/GLOB-0.log
ERROR: invocation failed (exit code 1), logfile: /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/log/GLOB-0.log
================================== log start ===================================
Traceback (most recent call last):
  File "/home/mjo/src/sage.git/pkgs/sagemath-standard/setup.py", line 31, in <module>
    from sage_setup.excepthook import excepthook
ModuleNotFoundError: No module named 'sage_setup'

=================================== log end ====================================
ERROR: FAIL could not package project - v = InvocationError('/usr/bin/python3.9 setup.py sdist --formats=zip --dist-dir /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/dist', 1)
make[1]: *** [Makefile:2798: sagelib-tox-python-sagewheels] Error 2
make[1]: Leaving directory '/home/mjo/src/sage.git/build/make'

real	0m4.988s
user	0m1.223s
sys	0m0.145s
***************************************************************
Error building Sage.

The following package(s) may have failed to build (not necessarily
during this run of 'make sagelib-tox-python-sagewheels'):

It is safe to delete any log files and build directories, but they
contain information that is helpful for debugging build problems.
WARNING: If you now run 'make' again, the build directory of the
same version of the package will, by default, be deleted. Set the
environment variable SAGE_KEEP_BUILT_SPKGS=yes to prevent this.

make: *** [Makefile:39: sagelib-tox-python-sagewheels] Error 1
ERROR: InvocationError for command /bin/bash -c 'export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/lib/llvm/12/bin:/home/mjo/bin:/home/mjo/bin && : &&  case "" in 1|y*|Y*);; *) ./bootstrap ;; esac &&  : &&  case "" in 1|y*|Y*);; *) ./configure --prefix=/home/mjo/src/sage.git/.tox/local-direct/local --enable-experimental-packages --enable-download-from-upstream-url  --with-system-python3=yes  --without-system-python3 ;; esac &&  case "build sagelib-tox-python-sagewheels" in  bash)    bash -i; exit ;;  config*) ;;  *)       make -k V=0 base-toolchain ;;  esac &&  make -k V=0 SAGE_CHECK=warn SAGE_CHECK_PACKAGES="!cython,!r,!python3,!gap,!cysignals,!linbox,!git,!ppl,!cmake,!rpy2,!sage_sws2rst"  build sagelib-tox-python-sagewheels && ( [ -z "" ] || make -k V=0 SAGE_CHECK=warn SAGE_CHECK_PACKAGES="!cython,!r,!python3,!gap,!cysignals,!linbox,!git,!ppl,!cmake,!rpy2,!sage_sws2rst"  || echo "(error ignored)" ) ' (exited with code 2)
__________________________________________________________________________________ summary __________________________________________________________________________________
ERROR:   local-direct: commands failed

Is that expected (did I construct my example wrong?)

Answer 24 · 2021-12-01T17:34:13.000Z

Changed dependencies from #32442, #32698 to #32899

Answer 25 · 2021-12-01T17:35:01.000Z

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`e5ad518`	`pkgs/sagemath-standard/setup.py: Do not depend on sage_setup for sdist or egg_info`
`821d16e`	`src/doc/en/developer/packaging_sage_library.rst: Add bootstrap to testing instructions to avoid the trap of #32868`
`77c957f`	`src/doc/en/developer/packaging_sage_library.rst: Put hierarchy section one level higher`
`9375ef8`	`pkgs/sagemath-standard/tox.ini: Use SAGE_VENV or venv symlink to find wheels`
`c6fc111`	`Prettier diagram`
`7d6a44c`	`Use :mod: as markup for packages/modules`
`f83d424`	`Improve ABC example`
`817a8d0`	`src/doc/en/developer/packaging_sage_library.rst: More :mod: and :class: markup`
`b555735`	`src/doc/en/developer/packaging_sage_library.rst: Link to pypi.org and to documentation of packaging metadata`
`ad12d25`	`Merge #32899`

Answer 26 · 2021-12-01T17:35:01.000Z

Changed commit from bdbab3a to ad12d25

Answer 27 · 2021-12-01T17:36:43.000Z

comment:21

Should be better with this branch

Answer 28 · 2021-12-01T17:36:55.000Z

comment:22

(needs ./bootstrap)

Answer 29 · 2021-12-03T21:30:54.000Z

comment:23

I just noticed these as well, probably from an empty $CC --version in sage_setup. No idea if they're new, I probably just happened to be looking in the right place at exactly the right time:

  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Running command /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/python-s\
agewheels-nopypi/bin/python /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/\
python-sagewheels-nopypi/lib/python3.9/site-packages/pip/_vendor/pep517/in_proc\
ess/_in_process.py prepare_metadata_for_build_wheel /tmp/tmphl6_9kin
    /bin/sh: 1: --version: not found

  Building wheel for sagemath-standard (PEP 517): started
  Running command /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/python-sag\
ewheels-nopypi/bin/python /home/mjo/src/sage.git/pkgs/sagemath-standard/.tox/py\
thon-sagewheels-nopypi/lib/python3.9/site-packages/pip/_vendor/pep517/in_proces\
s/_in_process.py build_wheel /tmp/tmpqovldekw
  /bin/sh: 1: --version: not found

Answer 30 · 2021-12-04T00:08:06.000Z

comment:24

Yes, this is #30876

Answer 31 · 2021-12-04T01:29:16.000Z

Reviewer: Michael Orlitzky

Answer 32 · 2021-12-04T01:29:16.000Z

comment:25

Looking better now; regardless, the Makefile targets themselves are working as intended.

Answer 33 · 2021-12-04T01:39:04.000Z

comment:26

Thanks!

Answer 34 · 2021-12-23T20:26:10.000Z

Changed branch from u/mkoeppe/new_make_targets__sagelib_sdist____sage_docbuild_sdist_______and__sagelib_tox__________ to ad12d25