Meta-ticket: Refactor and improve system package related scripts, tox.ini, build/bin/write_dockerfile.sh

Question

Meta-ticket: Refactor and improve system package related scripts, tox.ini, build/bin/write_dockerfile.sh

mkoeppe opened this issue 6 years ago · 44 comments

mkoeppe commented 6 years ago

Improvements to low-level system packages scripts:

#30861, #30865, #30947, #30951, #30968: Refactor system package code in tox.ini, build/bin/write_dockerfile.sh by extending and using sage_bootstrap
#33873: Extend build/bin/sage-print-system-package-command so that it can provide all distribution-specific commands such as apt-get install for build/bin/write_dockerfile.sh
#29041 sage_bootstrap: Add command "sage -package list --output={install-requires,requirements,pipfile,debian,....}"
#33860 sage-package: Add commands "dependencies", "trees"

Improvements to the system package database:

#29124 Add script packages build/pkgs/_prereq, build/pkgs/_toolchain, build/pkgs/_bootstrap
#36910 define shell quoting in files like debian.txt
handle options correctly when installing packages one by one, fix up passing installation options when IGNORE_MISSING_PACKAGES=yes

Improvements to tox docker:

#29536 Make docker images from GitHub CI workflow and regular Sage Docker images interoperable
optionally create a script for docker run instead of a Dockerfile (this gives more flexibility for multiarch - see #29143)
cache Docker images with system packages installed ... https://github.com/marketplace/actions/build-docker-images-using-cache
move log extraction code from tox.yml to tox.ini (#29530 did some preparation already)
system package installation should retry a few times when installation errors because of network errors occur. In particular for docker because an incomplete step will be cached
Make obtaining image from container of failing builds more robust: Use LABEL commands in dockerfiles, use docker ps --filter label=.... to look up the container (https://docs.docker.com/engine/reference/commandline/ps/)
setenv/passenv SAGE_SERVER
Remove TARGETS_PRE/TARGETS distinction: Instead leave a stamp file and exit from make with an error to trigger the ADD of src/; for example, add sagelib dependency on $(SAGE_SRC)/sage, and add the rule to Makefile:

$(SAGE_SRC)/sage:
	touch .need_src; exit 1

use environment variable MAKE for docker builds as well, instead of asking users to pass the USE_MAKEFLAGS variable
rewrite Dockerfile generation via dev container templates (https://github.com/devcontainers/cli)

Improvements to tox local:

#31216: tox.ini (local): Add environment variables to skip system package installs, mechanism for a local interactive shell
#31064: ci-cygwin*.yml: delegate to tox, add more stages, use more specific SAGE_LOCAL
#21469: Enable VPATH builds (several independent build trees connected to one source tree)
#29416: tox.ini: Add local-copy
the complicated rule for the commands of local consists entirely of workarounds for things that should really be fixed/improved in our Makefiles:
- #30721: build/make/Makefile.in: base-toolchain should be a dependency of every non-base, non-toolchain package
- #32785: tox.ini and build/bin/write-dockerfile.sh use better defaults for SAGE_CHECK_PACKAGES; this should be moved to build/make/Makefile.in (SAGE_CHECK_PACKAGES_DEFAULT_yes).
- #31535 New make targets "sagelib-sdist", "sage_docbuild-sdist", ... and "sagelib-tox-...", ...
  Other improvements to tox.ini:
#32764: Make tox -p auto and tox --listenvs useful again (it does not understand the nested braces)
#31574: tox.ini, build/bin/write-dockerfile.sh: Do not use make -k by default

Improvements to auto-generated parts of the manuals:

#29558 Autogenerated parts of installation guide
#29557 Add script package _recommended and generate "recommended system packages" info

Improvements for messages at the end of a ./configure run:

#29560 sage-guess-package-system: Ignore conda if no environment is activated
Perhaps sage-guess-package-system should report several package systems such as conda ubuntu or homebrew cpan when that makes sense
#29363 - At the end of configure, indicate which optional/experimental packages are configured to be installed
#30624: Improve wording and formatting of configure's recommendation message
#29372 - At the end of configure, show installation hints for disabled optional packages separately
Hints after ./configure: maybe sort and remove duplicates (https://groups.google.com/d/msg/sage-devel/EWTVN-Fmc8w/y9933Zo1AQAJ)
#32060 configure: Change defaults to --with-system-gcc=force, --with-system-python3=force
#29586: Improve configure's recommendations
#30863: refine definition of variable SAGE_NEED_SYSTEM_PACKAGES in m4/sage_spkg_collect.m4

Tickets regarding sage_bootstrap:

#20023 sage_bootstrap: Remove Python 2.6 support
#20104 sage --package: Add commands list :standard:, update-latest (for packages from PyPI), upload
#29919 Restore "huge" package type

See also:

#29060 Meta-ticket: Add Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages

Depends on #29417

CC: @embray @vbraun @dimpase @jhpalmieri @sheerluck @slel @tobiasdiez @seblabbe

Component: build

Branch/Commit: u/mkoeppe/meta_ticket__refactor_and_improve_system_package_related_scripts__tox_ini__build_bin_write_dockerfile_sh @ fd9b928

Issue created by migration from https://trac.sagemath.org/ticket/29146

Answer 1 · 2020-02-05T20:19:11.000Z

Description changed:

--- 
+++ 
@@ -4,3 +4,6 @@
 - helper scripts for extracting lists of packages according to criteria
 - possibly, by extending and using `sage_bootstrap`
 
+Also:
+- distribution-specific commands such as `apt-get install` are in both `build/bin/write_dockerfile.sh` and `build/bin/sage-spkg`. This could be unified as well, or be obtained from another text file etc.
+

Answer 2 · 2020-02-06T20:16:11.000Z

Description changed:

--- 
+++ 
@@ -3,6 +3,7 @@
 - instead of passing `TYPE_PATTERN` to `build/bin/write_dockerfile.sh`, do the work in `tox.ini`.
 - helper scripts for extracting lists of packages according to criteria
 - possibly, by extending and using `sage_bootstrap`
+- optionally create a script for `docker run` instead of a `Dockerfile` (this gives more flexibility for multiarch - see #29143)
 
 Also:
 - distribution-specific commands such as `apt-get install` are in both `build/bin/write_dockerfile.sh` and `build/bin/sage-spkg`. This could be unified as well, or be obtained from another text file etc.

Answer 3 · 2020-02-09T22:40:16.000Z

Description changed:

--- 
+++ 
@@ -6,5 +6,5 @@
 - optionally create a script for `docker run` instead of a `Dockerfile` (this gives more flexibility for multiarch - see #29143)
 
 Also:
-- distribution-specific commands such as `apt-get install` are in both `build/bin/write_dockerfile.sh` and `build/bin/sage-spkg`. This could be unified as well, or be obtained from another text file etc.
+- distribution-specific commands such as `apt-get install` are in both `build/bin/write_dockerfile.sh` and `build/bin/sage-spkg`. This could be unified as well, or be obtained from another text file etc., by using/extending `build/bin/sage-print-system-package-command` from #26964

Answer 4 · 2020-03-09T16:24:08.000Z

Description changed:

--- 
+++ 
@@ -4,6 +4,7 @@
 - helper scripts for extracting lists of packages according to criteria
 - possibly, by extending and using `sage_bootstrap`
 - optionally create a script for `docker run` instead of a `Dockerfile` (this gives more flexibility for multiarch - see #29143)
+- cache Docker images with system packages installed 
 
 Also:
 - distribution-specific commands such as `apt-get install` are in both `build/bin/write_dockerfile.sh` and `build/bin/sage-spkg`. This could be unified as well, or be obtained from another text file etc., by using/extending `build/bin/sage-print-system-package-command` from #26964

Answer 5 · 2020-03-14T21:43:05.000Z

Description changed:

--- 
+++ 
@@ -4,7 +4,7 @@
 - helper scripts for extracting lists of packages according to criteria
 - possibly, by extending and using `sage_bootstrap`
 - optionally create a script for `docker run` instead of a `Dockerfile` (this gives more flexibility for multiarch - see #29143)
-- cache Docker images with system packages installed 
+- cache Docker images with system packages installed ... https://github.com/marketplace/actions/build-docker-images-using-cache
 
 Also:
 - distribution-specific commands such as `apt-get install` are in both `build/bin/write_dockerfile.sh` and `build/bin/sage-spkg`. This could be unified as well, or be obtained from another text file etc., by using/extending `build/bin/sage-print-system-package-command` from #26964

Answer 6 · 2020-03-20T02:57:40.000Z

Description changed:

--- 
+++ 
@@ -6,6 +6,9 @@
 - optionally create a script for `docker run` instead of a `Dockerfile` (this gives more flexibility for multiarch - see #29143)
 - cache Docker images with system packages installed ... https://github.com/marketplace/actions/build-docker-images-using-cache
 
+tox.ini:
+- Add user option for configuring number of threads used 
+
 Also:
 - distribution-specific commands such as `apt-get install` are in both `build/bin/write_dockerfile.sh` and `build/bin/sage-spkg`. This could be unified as well, or be obtained from another text file etc., by using/extending `build/bin/sage-print-system-package-command` from #26964

Answer 7 · 2020-03-22T09:30:54.000Z

Description changed:

--- 
+++ 
@@ -5,6 +5,7 @@
 - possibly, by extending and using `sage_bootstrap`
 - optionally create a script for `docker run` instead of a `Dockerfile` (this gives more flexibility for multiarch - see #29143)
 - cache Docker images with system packages installed ... https://github.com/marketplace/actions/build-docker-images-using-cache
+- define shell quoting in debian .txt, handle options correctly when installing packages one by one 
 
 tox.ini:
 - Add user option for configuring number of threads used

Answer 8 · 2020-03-29T04:07:24.000Z

Description changed:

--- 
+++ 
@@ -5,7 +5,7 @@
 - possibly, by extending and using `sage_bootstrap`
 - optionally create a script for `docker run` instead of a `Dockerfile` (this gives more flexibility for multiarch - see #29143)
 - cache Docker images with system packages installed ... https://github.com/marketplace/actions/build-docker-images-using-cache
-- define shell quoting in debian .txt, handle options correctly when installing packages one by one 
+- define shell quoting in files like `debian.txt`, handle options correctly when installing packages one by one 
 
 tox.ini:
 - Add user option for configuring number of threads used 
@@ -13,3 +13,9 @@
 Also:
 - distribution-specific commands such as `apt-get install` are in both `build/bin/write_dockerfile.sh` and `build/bin/sage-spkg`. This could be unified as well, or be obtained from another text file etc., by using/extending `build/bin/sage-print-system-package-command` from #26964
 
+Related:
+- #29363 - At the end of configure, indicate which optional/experimental packages are configured to be installed
+- #29372 - At the end of configure, show installation hints for disabled optional packages separately
+- #29087 - Add [GitHub](../wiki/GitHub) Actions workflow for testing the build on various Linux distributions via docker; macOS with homebrew; and Windows (cygwin)
+- #29401 - Add documentation of tox and [GitHub](../wiki/GitHub) actions workflow to developer's manual
+

Answer 9 · 2020-03-29T04:08:44.000Z

Description changed:

--- 
+++ 
@@ -6,6 +6,7 @@
 - optionally create a script for `docker run` instead of a `Dockerfile` (this gives more flexibility for multiarch - see #29143)
 - cache Docker images with system packages installed ... https://github.com/marketplace/actions/build-docker-images-using-cache
 - define shell quoting in files like `debian.txt`, handle options correctly when installing packages one by one 
+- Hints after ./configure: maybe sort and remove duplicates (https://groups.google.com/d/msg/sage-devel/EWTVN-Fmc8w/y9933Zo1AQAJ)
 
 tox.ini:
 - Add user option for configuring number of threads used

Answer 10 · 2020-04-09T23:16:51.000Z

Description changed:

--- 
+++ 
@@ -15,6 +15,7 @@
 - distribution-specific commands such as `apt-get install` are in both `build/bin/write_dockerfile.sh` and `build/bin/sage-spkg`. This could be unified as well, or be obtained from another text file etc., by using/extending `build/bin/sage-print-system-package-command` from #26964
 
 Related:
+- #29417 - has various improvements already 
 - #29363 - At the end of configure, indicate which optional/experimental packages are configured to be installed
 - #29372 - At the end of configure, show installation hints for disabled optional packages separately
 - #29087 - Add [GitHub](../wiki/GitHub) Actions workflow for testing the build on various Linux distributions via docker; macOS with homebrew; and Windows (cygwin)

Answer 11 · 2020-04-09T23:16:51.000Z

Dependencies: #29417

Answer 12 · 2020-04-22T00:26:25.000Z

Description changed:

--- 
+++ 
@@ -9,7 +9,7 @@
 - Hints after ./configure: maybe sort and remove duplicates (https://groups.google.com/d/msg/sage-devel/EWTVN-Fmc8w/y9933Zo1AQAJ)
 
 tox.ini:
-- Add user option for configuring number of threads used 
+- Obtain image from container of failing builds. Move&improve code from tox.yml. Use `docker ps --filter label=.... ` to look up the image more reliably (https://docs.docker.com/engine/reference/commandline/ps/)
 
 Also:
 - distribution-specific commands such as `apt-get install` are in both `build/bin/write_dockerfile.sh` and `build/bin/sage-spkg`. This could be unified as well, or be obtained from another text file etc., by using/extending `build/bin/sage-print-system-package-command` from #26964

Answer 13 · 2020-04-25T17:53:27.000Z

Description changed:

--- 
+++ 
@@ -16,8 +16,10 @@
 
 Related:
 - #29417 - has various improvements already 
-- #29363 - At the end of configure, indicate which optional/experimental packages are configured to be installed
-- #29372 - At the end of configure, show installation hints for disabled optional packages separately
 - #29087 - Add [GitHub](../wiki/GitHub) Actions workflow for testing the build on various Linux distributions via docker; macOS with homebrew; and Windows (cygwin)
 - #29401 - Add documentation of tox and [GitHub](../wiki/GitHub) actions workflow to developer's manual
 
+Wishlist items:
+- #29363 - At the end of configure, indicate which optional/experimental packages are configured to be installed
+- #29372 - At the end of configure, show installation hints for disabled optional packages separately
+

Answer 14 · 2020-04-25T17:54:08.000Z

Description changed:

--- 
+++ 
@@ -19,7 +19,7 @@
 - #29087 - Add [GitHub](../wiki/GitHub) Actions workflow for testing the build on various Linux distributions via docker; macOS with homebrew; and Windows (cygwin)
 - #29401 - Add documentation of tox and [GitHub](../wiki/GitHub) actions workflow to developer's manual
 
-Wishlist items:
+Wishlist items for messages at the end of configure:
 - #29363 - At the end of configure, indicate which optional/experimental packages are configured to be installed
 - #29372 - At the end of configure, show installation hints for disabled optional packages separately

Answer 15 · 2020-04-25T18:08:15.000Z

Description changed:

--- 
+++ 
@@ -19,7 +19,11 @@
 - #29087 - Add [GitHub](../wiki/GitHub) Actions workflow for testing the build on various Linux distributions via docker; macOS with homebrew; and Windows (cygwin)
 - #29401 - Add documentation of tox and [GitHub](../wiki/GitHub) actions workflow to developer's manual
 
-Wishlist items for messages at the end of configure:
-- #29363 - At the end of configure, indicate which optional/experimental packages are configured to be installed
-- #29372 - At the end of configure, show installation hints for disabled optional packages separately
+Improvements to auto-generated parts of the manuals:
+- #29558 Autogenerated parts of installation guide
 
+Improvements for messages at the end of a `./configure` run:
+- #29560 `sage-guess-package-system`: Ignore conda if no environment is activated
+- #29363 - At the end of `configure`, indicate which optional/experimental packages are configured to be installed
+- #29372 - At the end of `configure`, show installation hints for disabled optional packages separately
+

Answer 16 · 2020-04-25T18:16:56.000Z

Description changed:

--- 
+++ 
@@ -6,7 +6,6 @@
 - optionally create a script for `docker run` instead of a `Dockerfile` (this gives more flexibility for multiarch - see #29143)
 - cache Docker images with system packages installed ... https://github.com/marketplace/actions/build-docker-images-using-cache
 - define shell quoting in files like `debian.txt`, handle options correctly when installing packages one by one 
-- Hints after ./configure: maybe sort and remove duplicates (https://groups.google.com/d/msg/sage-devel/EWTVN-Fmc8w/y9933Zo1AQAJ)
 
 tox.ini:
 - Obtain image from container of failing builds. Move&improve code from tox.yml. Use `docker ps --filter label=.... ` to look up the image more reliably (https://docs.docker.com/engine/reference/commandline/ps/)
@@ -26,4 +25,5 @@
 - #29560 `sage-guess-package-system`: Ignore conda if no environment is activated
 - #29363 - At the end of `configure`, indicate which optional/experimental packages are configured to be installed
 - #29372 - At the end of `configure`, show installation hints for disabled optional packages separately
+- Hints after ./configure: maybe sort and remove duplicates (https://groups.google.com/d/msg/sage-devel/EWTVN-Fmc8w/y9933Zo1AQAJ)

Answer 17 · 2020-04-25T18:42:32.000Z

Description changed:

--- 
+++ 
@@ -1,25 +1,26 @@
-`build/bin/write_dockerfile.sh`, introduced in #29053, determines lists of system packages in a rather crude way. This should be improved.
-
+Improvements to low-level system packages scripts:
+- `build/bin/write_dockerfile.sh`, introduced in #29053, determines lists of system packages in a rather crude way. This should be improved.
 - instead of passing `TYPE_PATTERN` to `build/bin/write_dockerfile.sh`, do the work in `tox.ini`.
 - helper scripts for extracting lists of packages according to criteria
 - possibly, by extending and using `sage_bootstrap`
+- Extend `build/bin/sage-print-system-package-command` so that it can provide all distribution-specific commands such as `apt-get install` for `build/bin/write_dockerfile.sh` 
+
+Improvements to the system package database:
+- define shell quoting in files like `debian.txt`, handle options correctly when installing packages one by one, fix up passing installation options when `IGNORE_MISSING_PACKAGES=yes`
+- #29124 Add script packages build/pkgs/_prereq, build/pkgs/_toolchain, build/pkgs/_bootstrap
+
+Improvements to `tox docker`:
 - optionally create a script for `docker run` instead of a `Dockerfile` (this gives more flexibility for multiarch - see #29143)
 - cache Docker images with system packages installed ... https://github.com/marketplace/actions/build-docker-images-using-cache
-- define shell quoting in files like `debian.txt`, handle options correctly when installing packages one by one 
+- move log extraction code from tox.yml to tox.ini (#29530 did some preparation already)
+- Make obtaining image from container of failing builds more robust: Use LABEL commands in dockerfiles, use `docker ps --filter label=.... ` to look up the container (https://docs.docker.com/engine/reference/commandline/ps/)
 
-tox.ini:
-- Obtain image from container of failing builds. Move&improve code from tox.yml. Use `docker ps --filter label=.... ` to look up the image more reliably (https://docs.docker.com/engine/reference/commandline/ps/)
-
-Also:
-- distribution-specific commands such as `apt-get install` are in both `build/bin/write_dockerfile.sh` and `build/bin/sage-spkg`. This could be unified as well, or be obtained from another text file etc., by using/extending `build/bin/sage-print-system-package-command` from #26964
-
-Related:
-- #29417 - has various improvements already 
-- #29087 - Add [GitHub](../wiki/GitHub) Actions workflow for testing the build on various Linux distributions via docker; macOS with homebrew; and Windows (cygwin)
-- #29401 - Add documentation of tox and [GitHub](../wiki/GitHub) actions workflow to developer's manual
+Improvements to `tox local`:
+- 
 
 Improvements to auto-generated parts of the manuals:
 - #29558 Autogenerated parts of installation guide
+- #29557 Add script package `_recommended` and generate "recommended system packages" info
 
 Improvements for messages at the end of a `./configure` run:
 - #29560 `sage-guess-package-system`: Ignore conda if no environment is activated
@@ -27,3 +28,6 @@
 - #29372 - At the end of `configure`, show installation hints for disabled optional packages separately
 - Hints after ./configure: maybe sort and remove duplicates (https://groups.google.com/d/msg/sage-devel/EWTVN-Fmc8w/y9933Zo1AQAJ)
 
+See also: 
+- #29060 Meta-ticket: Add Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages
+

Answer 18 · 2020-04-25T18:45:13.000Z

Description changed:

--- 
+++ 
@@ -16,7 +16,11 @@
 - Make obtaining image from container of failing builds more robust: Use LABEL commands in dockerfiles, use `docker ps --filter label=.... ` to look up the container (https://docs.docker.com/engine/reference/commandline/ps/)
 
 Improvements to `tox local`:
-- 
+- #29416: tox.ini: Add local-copy
+- #21469: Enable VPATH builds (several independent build trees connected to one source tree)
+
+Other improvements to `tox.ini`:
+- Make `tox -p auto` useful again (it does not understand the nested braces)
 
 Improvements to auto-generated parts of the manuals:
 - #29558 Autogenerated parts of installation guide

Answer 19 · 2020-04-25T22:51:00.000Z

Description changed:

--- 
+++ 
@@ -28,6 +28,7 @@
 
 Improvements for messages at the end of a `./configure` run:
 - #29560 `sage-guess-package-system`: Ignore conda if no environment is activated
+- Perhaps `sage-guess-package-system` should report several package systems such as `conda ubuntu` or `homebrew cpan` when that makes sense
 - #29363 - At the end of `configure`, indicate which optional/experimental packages are configured to be installed
 - #29372 - At the end of `configure`, show installation hints for disabled optional packages separately
 - Hints after ./configure: maybe sort and remove duplicates (https://groups.google.com/d/msg/sage-devel/EWTVN-Fmc8w/y9933Zo1AQAJ)

Answer 20 · 2020-04-26T03:47:19.000Z

Description changed:

--- 
+++ 
@@ -32,6 +32,7 @@
 - #29363 - At the end of `configure`, indicate which optional/experimental packages are configured to be installed
 - #29372 - At the end of `configure`, show installation hints for disabled optional packages separately
 - Hints after ./configure: maybe sort and remove duplicates (https://groups.google.com/d/msg/sage-devel/EWTVN-Fmc8w/y9933Zo1AQAJ)
+- #29586: Improve configure's recommendations
 
 See also: 
 - #29060 Meta-ticket: Add Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages

Answer 21 · 2020-04-27T05:10:29.000Z

Description changed:

--- 
+++ 
@@ -13,6 +13,7 @@
 - optionally create a script for `docker run` instead of a `Dockerfile` (this gives more flexibility for multiarch - see #29143)
 - cache Docker images with system packages installed ... https://github.com/marketplace/actions/build-docker-images-using-cache
 - move log extraction code from tox.yml to tox.ini (#29530 did some preparation already)
+- system package installation should retry a few times when installation errors because of network errors occur. In particular for docker because an incomplete step will be cached
 - Make obtaining image from container of failing builds more robust: Use LABEL commands in dockerfiles, use `docker ps --filter label=.... ` to look up the container (https://docs.docker.com/engine/reference/commandline/ps/)
 
 Improvements to `tox local`:

Answer 22 · 2020-04-28T00:12:24.000Z

Description changed:

--- 
+++ 
@@ -15,6 +15,7 @@
 - move log extraction code from tox.yml to tox.ini (#29530 did some preparation already)
 - system package installation should retry a few times when installation errors because of network errors occur. In particular for docker because an incomplete step will be cached
 - Make obtaining image from container of failing builds more robust: Use LABEL commands in dockerfiles, use `docker ps --filter label=.... ` to look up the container (https://docs.docker.com/engine/reference/commandline/ps/)
+- setenv/passenv SAGE_SERVER
 
 Improvements to `tox local`:
 - #29416: tox.ini: Add local-copy

Answer 23 · 2020-04-28T17:26:31.000Z

Description changed:

--- 
+++ 
@@ -16,6 +16,7 @@
 - system package installation should retry a few times when installation errors because of network errors occur. In particular for docker because an incomplete step will be cached
 - Make obtaining image from container of failing builds more robust: Use LABEL commands in dockerfiles, use `docker ps --filter label=.... ` to look up the container (https://docs.docker.com/engine/reference/commandline/ps/)
 - setenv/passenv SAGE_SERVER
+- Remove TARGETS_PRE/TARGETS distinction: Instead exit from make with a distinctive error code to trigger the ADD of src/
 
 Improvements to `tox local`:
 - #29416: tox.ini: Add local-copy

Answer 24 · 2020-05-15T19:39:23.000Z

Branch: u/mkoeppe/meta_ticket__refactor_and_improve_system_package_related_scripts__tox_ini__build_bin_write_dockerfile_sh

Answer 25 · 2020-05-15T19:43:54.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`fd9b928`	`build/bin/sage-logger [V=0]: Show build times of successful builds`

Answer 26 · 2020-05-15T19:43:54.000Z

Commit: fd9b928

Answer 27 · 2020-06-20T17:27:39.000Z

Description changed:

--- 
+++ 
@@ -37,6 +37,11 @@
 - Hints after ./configure: maybe sort and remove duplicates (https://groups.google.com/d/msg/sage-devel/EWTVN-Fmc8w/y9933Zo1AQAJ)
 - #29586: Improve configure's recommendations
 
+Tickets regarding `sage_bootstrap`:
+- #29919 Restore "huge" package type
+- #20023 `sage_bootstrap`: Remove Python 2.6 support
+- #20104 `sage --package`: Add commands `list :standard:`, `update-latest` (for packages from PyPI), `upload`
+
 See also: 
 - #29060 Meta-ticket: Add Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages

Answer 28 · 2020-11-03T23:29:50.000Z

Description changed:

--- 
+++ 
@@ -1,8 +1,8 @@
 Improvements to low-level system packages scripts:
-- `build/bin/write_dockerfile.sh`, introduced in #29053, determines lists of system packages in a rather crude way. This should be improved.
-- instead of passing `TYPE_PATTERN` to `build/bin/write_dockerfile.sh`, do the work in `tox.ini`.
-- helper scripts for extracting lists of packages according to criteria
-- possibly, by extending and using `sage_bootstrap`
+- `build/bin/write_dockerfile.sh`, introduced in #29053, determines lists of system packages in a rather crude way. This should be improved. #30861
+   - instead of passing `TYPE_PATTERN` to `build/bin/write_dockerfile.sh`, do the work in `tox.ini`.
+   - helper scripts for extracting lists of packages according to criteria
+   - possibly, by extending and using `sage_bootstrap`
 - Extend `build/bin/sage-print-system-package-command` so that it can provide all distribution-specific commands such as `apt-get install` for `build/bin/write_dockerfile.sh` 
 
 Improvements to the system package database:

Answer 29 · 2020-11-04T20:23:15.000Z

Description changed:

--- 
+++ 
@@ -36,6 +36,7 @@
 - #29372 - At the end of `configure`, show installation hints for disabled optional packages separately
 - Hints after ./configure: maybe sort and remove duplicates (https://groups.google.com/d/msg/sage-devel/EWTVN-Fmc8w/y9933Zo1AQAJ)
 - #29586: Improve configure's recommendations
+- #30863: refine definition of variable SAGE_NEED_SYSTEM_PACKAGES in m4/sage_spkg_collect.m4
 
 Tickets regarding `sage_bootstrap`:
 - #29919 Restore "huge" package type

Answer 30 · 2020-11-04T22:01:37.000Z

Description changed:

--- 
+++ 
@@ -1,5 +1,5 @@
 Improvements to low-level system packages scripts:
-- `build/bin/write_dockerfile.sh`, introduced in #29053, determines lists of system packages in a rather crude way. This should be improved. #30861
+- `build/bin/write_dockerfile.sh`, introduced in #29053, determines lists of system packages in a rather crude way. This should be improved. #30861, #30865
    - instead of passing `TYPE_PATTERN` to `build/bin/write_dockerfile.sh`, do the work in `tox.ini`.
    - helper scripts for extracting lists of packages according to criteria
    - possibly, by extending and using `sage_bootstrap`

Answer 31 · 2020-11-17T02:47:45.000Z

Description changed:

--- 
+++ 
@@ -36,6 +36,7 @@
 - #29372 - At the end of `configure`, show installation hints for disabled optional packages separately
 - Hints after ./configure: maybe sort and remove duplicates (https://groups.google.com/d/msg/sage-devel/EWTVN-Fmc8w/y9933Zo1AQAJ)
 - #29586: Improve configure's recommendations
+- #30624: Improve wording and formatting of configure's recommendation message
 - #30863: refine definition of variable SAGE_NEED_SYSTEM_PACKAGES in m4/sage_spkg_collect.m4
 
 Tickets regarding `sage_bootstrap`:

Answer 32 · 2020-11-29T20:31:05.000Z

Description changed:

--- 
+++ 
@@ -1,5 +1,5 @@
 Improvements to low-level system packages scripts:
-- `build/bin/write_dockerfile.sh`, introduced in #29053, determines lists of system packages in a rather crude way. This should be improved. #30861, #30865
+- `build/bin/write_dockerfile.sh`, introduced in #29053, determines lists of system packages in a rather crude way. This should be improved. #30861, #30865, #30947, #30951, #30968
    - instead of passing `TYPE_PATTERN` to `build/bin/write_dockerfile.sh`, do the work in `tox.ini`.
    - helper scripts for extracting lists of packages according to criteria
    - possibly, by extending and using `sage_bootstrap`

Answer 33 · 2021-01-10T00:19:11.000Z

Description changed:

--- 
+++ 
@@ -21,9 +21,11 @@
 Improvements to `tox local`:
 - #29416: tox.ini: Add local-copy
 - #21469: Enable VPATH builds (several independent build trees connected to one source tree)
+- #31216: tox.ini (local): Add environment variables to skip system package installs, mechanism for a local interactive shell
+- #31064: `ci-cygwin*.yml`: delegate to `tox`, add more stages, use more specific `SAGE_LOCAL`
 
 Other improvements to `tox.ini`:
-- Make `tox -p auto` useful again (it does not understand the nested braces)
+- Make `tox -p auto` and `tox --listenvs` useful again (it does not understand the nested braces)
 
 Improvements to auto-generated parts of the manuals:
 - #29558 Autogenerated parts of installation guide

Answer 34 · 2021-01-10T00:24:56.000Z

Description changed:

--- 
+++ 
@@ -23,6 +23,9 @@
 - #21469: Enable VPATH builds (several independent build trees connected to one source tree)
 - #31216: tox.ini (local): Add environment variables to skip system package installs, mechanism for a local interactive shell
 - #31064: `ci-cygwin*.yml`: delegate to `tox`, add more stages, use more specific `SAGE_LOCAL`
+- the complicated rule for the commands of `local` consists entirely of workarounds for things that should really be fixed/improved in our `Makefile`s:
+   - #30721: `build/make/Makefile.in`: `base-toolchain` should be a dependency of every non-base, non-toolchain package
+   - `tox.ini` and `build/bin/write-dockerfile.sh` use better defaults for `SAGE_CHECK_PACKAGES`; this should be moved to `build/make/Makefile.in` (`SAGE_CHECK_PACKAGES_DEFAULT_yes`).
 
 Other improvements to `tox.ini`:
 - Make `tox -p auto` and `tox --listenvs` useful again (it does not understand the nested braces)

Answer 35 · 2021-03-24T02:04:25.000Z

comment:39

Sage development has entered the release candidate phase for 9.3. Setting a new milestone for this ticket based on a cursory review of ticket status, priority, and last modification date.

Answer 36 · 2021-03-28T05:06:05.000Z

Description changed:

--- 
+++ 
@@ -16,7 +16,12 @@
 - system package installation should retry a few times when installation errors because of network errors occur. In particular for docker because an incomplete step will be cached
 - Make obtaining image from container of failing builds more robust: Use LABEL commands in dockerfiles, use `docker ps --filter label=.... ` to look up the container (https://docs.docker.com/engine/reference/commandline/ps/)
 - setenv/passenv SAGE_SERVER
-- Remove TARGETS_PRE/TARGETS distinction: Instead exit from make with a distinctive error code to trigger the ADD of src/
+- Remove `TARGETS_PRE`/`TARGETS` distinction: Instead exit from make with a distinctive error code to trigger the ADD of src/; for example, add `sagelib` dependency on `$(SAGE_SRC)/sage`, and add the rule to Makefile:
+
+```
+$(SAGE_SRC)/sage:
+	exit 76
+```
 
 Improvements to `tox local`:
 - #29416: tox.ini: Add local-copy

Answer 37 · 2021-03-28T05:09:51.000Z

Description changed:

--- 
+++ 
@@ -16,11 +16,11 @@
 - system package installation should retry a few times when installation errors because of network errors occur. In particular for docker because an incomplete step will be cached
 - Make obtaining image from container of failing builds more robust: Use LABEL commands in dockerfiles, use `docker ps --filter label=.... ` to look up the container (https://docs.docker.com/engine/reference/commandline/ps/)
 - setenv/passenv SAGE_SERVER
-- Remove `TARGETS_PRE`/`TARGETS` distinction: Instead exit from make with a distinctive error code to trigger the ADD of src/; for example, add `sagelib` dependency on `$(SAGE_SRC)/sage`, and add the rule to Makefile:
+- Remove `TARGETS_PRE`/`TARGETS` distinction: Instead leave a stamp file and exit from make with an error to trigger the ADD of src/; for example, add `sagelib` dependency on `$(SAGE_SRC)/sage`, and add the rule to Makefile:
 
 ```
 $(SAGE_SRC)/sage:
-	exit 76
+	touch .need_src; exit 1
 ```
 
 Improvements to `tox local`:

Answer 38 · 2021-03-28T18:38:41.000Z

Description changed:

--- 
+++ 
@@ -1,8 +1,5 @@
 Improvements to low-level system packages scripts:
-- `build/bin/write_dockerfile.sh`, introduced in #29053, determines lists of system packages in a rather crude way. This should be improved. #30861, #30865, #30947, #30951, #30968
-   - instead of passing `TYPE_PATTERN` to `build/bin/write_dockerfile.sh`, do the work in `tox.ini`.
-   - helper scripts for extracting lists of packages according to criteria
-   - possibly, by extending and using `sage_bootstrap`
+- #30861, #30865, #30947, #30951, #30968: Refactor system package code in `tox.ini`, `build/bin/write_dockerfile.sh` by extending and using `sage_bootstrap`
 - Extend `build/bin/sage-print-system-package-command` so that it can provide all distribution-specific commands such as `apt-get install` for `build/bin/write_dockerfile.sh` 
 
 Improvements to the system package database:
@@ -22,6 +19,7 @@
 $(SAGE_SRC)/sage:
 	touch .need_src; exit 1
 ```
+- use environment variable `MAKE` for docker builds as well, instead of asking users to pass the `USE_MAKEFLAGS` variable
 
 Improvements to `tox local`:
 - #29416: tox.ini: Add local-copy
@@ -34,6 +32,8 @@
 
 Other improvements to `tox.ini`:
 - Make `tox -p auto` and `tox --listenvs` useful again (it does not understand the nested braces)
+- #31574: `tox.ini`, `build/bin/write-dockerfile.sh`: Do not use `make -k` by default
+- 
 
 Improvements to auto-generated parts of the manuals:
 - #29558 Autogenerated parts of installation guide

Answer 39 · 2021-03-28T19:51:37.000Z

Description changed:

--- 
+++ 
@@ -3,10 +3,11 @@
 - Extend `build/bin/sage-print-system-package-command` so that it can provide all distribution-specific commands such as `apt-get install` for `build/bin/write_dockerfile.sh` 
 
 Improvements to the system package database:
+- #29124 Add script packages build/pkgs/_prereq, build/pkgs/_toolchain, build/pkgs/_bootstrap
 - define shell quoting in files like `debian.txt`, handle options correctly when installing packages one by one, fix up passing installation options when `IGNORE_MISSING_PACKAGES=yes`
-- #29124 Add script packages build/pkgs/_prereq, build/pkgs/_toolchain, build/pkgs/_bootstrap
 
 Improvements to `tox docker`:
+- #29536 Make docker images from [GitHub](../wiki/GitHub) CI workflow and regular Sage Docker images interoperable
 - optionally create a script for `docker run` instead of a `Dockerfile` (this gives more flexibility for multiarch - see #29143)
 - cache Docker images with system packages installed ... https://github.com/marketplace/actions/build-docker-images-using-cache
 - move log extraction code from tox.yml to tox.ini (#29530 did some preparation already)
@@ -22,18 +23,17 @@
 - use environment variable `MAKE` for docker builds as well, instead of asking users to pass the `USE_MAKEFLAGS` variable
 
 Improvements to `tox local`:
-- #29416: tox.ini: Add local-copy
-- #21469: Enable VPATH builds (several independent build trees connected to one source tree)
 - #31216: tox.ini (local): Add environment variables to skip system package installs, mechanism for a local interactive shell
 - #31064: `ci-cygwin*.yml`: delegate to `tox`, add more stages, use more specific `SAGE_LOCAL`
+- #21469: Enable VPATH builds (several independent build trees connected to one source tree)
+- #29416: tox.ini: Add local-copy
 - the complicated rule for the commands of `local` consists entirely of workarounds for things that should really be fixed/improved in our `Makefile`s:
    - #30721: `build/make/Makefile.in`: `base-toolchain` should be a dependency of every non-base, non-toolchain package
    - `tox.ini` and `build/bin/write-dockerfile.sh` use better defaults for `SAGE_CHECK_PACKAGES`; this should be moved to `build/make/Makefile.in` (`SAGE_CHECK_PACKAGES_DEFAULT_yes`).
-
+   - #31535 New make targets "sagelib-sdist", "sage_docbuild-sdist", ... and "sagelib-tox-...", ...
 Other improvements to `tox.ini`:
 - Make `tox -p auto` and `tox --listenvs` useful again (it does not understand the nested braces)
 - #31574: `tox.ini`, `build/bin/write-dockerfile.sh`: Do not use `make -k` by default
-- 
 
 Improvements to auto-generated parts of the manuals:
 - #29558 Autogenerated parts of installation guide
@@ -43,16 +43,16 @@
 - #29560 `sage-guess-package-system`: Ignore conda if no environment is activated
 - Perhaps `sage-guess-package-system` should report several package systems such as `conda ubuntu` or `homebrew cpan` when that makes sense
 - #29363 - At the end of `configure`, indicate which optional/experimental packages are configured to be installed
+- #30624: Improve wording and formatting of configure's recommendation message
 - #29372 - At the end of `configure`, show installation hints for disabled optional packages separately
 - Hints after ./configure: maybe sort and remove duplicates (https://groups.google.com/d/msg/sage-devel/EWTVN-Fmc8w/y9933Zo1AQAJ)
 - #29586: Improve configure's recommendations
-- #30624: Improve wording and formatting of configure's recommendation message
 - #30863: refine definition of variable SAGE_NEED_SYSTEM_PACKAGES in m4/sage_spkg_collect.m4
 
 Tickets regarding `sage_bootstrap`:
-- #29919 Restore "huge" package type
 - #20023 `sage_bootstrap`: Remove Python 2.6 support
 - #20104 `sage --package`: Add commands `list :standard:`, `update-latest` (for packages from PyPI), `upload`
+- #29919 Restore "huge" package type
 
 See also: 
 - #29060 Meta-ticket: Add Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages

Answer 40 · 2021-06-25T18:32:32.000Z

Description changed:

--- 
+++ 
@@ -46,6 +46,7 @@
 - #30624: Improve wording and formatting of configure's recommendation message
 - #29372 - At the end of `configure`, show installation hints for disabled optional packages separately
 - Hints after ./configure: maybe sort and remove duplicates (https://groups.google.com/d/msg/sage-devel/EWTVN-Fmc8w/y9933Zo1AQAJ)
+- #32060 `configure`: Change defaults to `--with-system-gcc=force`, `--with-system-python3=force`
 - #29586: Improve configure's recommendations
 - #30863: refine definition of variable SAGE_NEED_SYSTEM_PACKAGES in m4/sage_spkg_collect.m4

Answer 41 · 2021-10-25T15:52:53.000Z

Description changed:

--- 
+++ 
@@ -32,7 +32,7 @@
    - `tox.ini` and `build/bin/write-dockerfile.sh` use better defaults for `SAGE_CHECK_PACKAGES`; this should be moved to `build/make/Makefile.in` (`SAGE_CHECK_PACKAGES_DEFAULT_yes`).
    - #31535 New make targets "sagelib-sdist", "sage_docbuild-sdist", ... and "sagelib-tox-...", ...
 Other improvements to `tox.ini`:
-- Make `tox -p auto` and `tox --listenvs` useful again (it does not understand the nested braces)
+- #32764: Make `tox -p auto` and `tox --listenvs` useful again (it does not understand the nested braces)
 - #31574: `tox.ini`, `build/bin/write-dockerfile.sh`: Do not use `make -k` by default
 
 Improvements to auto-generated parts of the manuals:

Answer 42 · 2021-10-28T06:11:46.000Z

Description changed:

--- 
+++ 
@@ -29,7 +29,7 @@
 - #29416: tox.ini: Add local-copy
 - the complicated rule for the commands of `local` consists entirely of workarounds for things that should really be fixed/improved in our `Makefile`s:
    - #30721: `build/make/Makefile.in`: `base-toolchain` should be a dependency of every non-base, non-toolchain package
-   - `tox.ini` and `build/bin/write-dockerfile.sh` use better defaults for `SAGE_CHECK_PACKAGES`; this should be moved to `build/make/Makefile.in` (`SAGE_CHECK_PACKAGES_DEFAULT_yes`).
+   - #32785: `tox.ini` and `build/bin/write-dockerfile.sh` use better defaults for `SAGE_CHECK_PACKAGES`; this should be moved to `build/make/Makefile.in` (`SAGE_CHECK_PACKAGES_DEFAULT_yes`).
    - #31535 New make targets "sagelib-sdist", "sage_docbuild-sdist", ... and "sagelib-tox-...", ...
 Other improvements to `tox.ini`:
 - #32764: Make `tox -p auto` and `tox --listenvs` useful again (it does not understand the nested braces)

Answer 43 · 2022-05-17T05:40:55.000Z

Description changed:

--- 
+++ 
@@ -1,6 +1,8 @@
 Improvements to low-level system packages scripts:
 - #30861, #30865, #30947, #30951, #30968: Refactor system package code in `tox.ini`, `build/bin/write_dockerfile.sh` by extending and using `sage_bootstrap`
 - Extend `build/bin/sage-print-system-package-command` so that it can provide all distribution-specific commands such as `apt-get install` for `build/bin/write_dockerfile.sh` 
+- #29041 sage_bootstrap: Add command "sage -package list --output={install-requires,requirements,pipfile,debian,....}"
+- #33860 sage-package: Add commands "dependencies", "trees"
 
 Improvements to the system package database:
 - #29124 Add script packages build/pkgs/_prereq, build/pkgs/_toolchain, build/pkgs/_bootstrap

Answer 44 · 2022-05-20T03:43:07.000Z

Description changed:

--- 
+++ 
@@ -1,6 +1,6 @@
 Improvements to low-level system packages scripts:
 - #30861, #30865, #30947, #30951, #30968: Refactor system package code in `tox.ini`, `build/bin/write_dockerfile.sh` by extending and using `sage_bootstrap`
-- Extend `build/bin/sage-print-system-package-command` so that it can provide all distribution-specific commands such as `apt-get install` for `build/bin/write_dockerfile.sh` 
+- #33873: Extend `build/bin/sage-print-system-package-command` so that it can provide all distribution-specific commands such as `apt-get install` for `build/bin/write_dockerfile.sh` 
 - #29041 sage_bootstrap: Add command "sage -package list --output={install-requires,requirements,pipfile,debian,....}"
 - #33860 sage-package: Add commands "dependencies", "trees"