Add script packages build/pkgs/_prereq, build/pkgs/_bootstrap
mkoeppe opened this issue · 91 comments
As a follow-up to #29053, we move the build/pkgs/*.txt files to a more systematic location. This will simplify some scripts.
build/pkgs/_prereq/distros/debian.txtetc. - records the minimal requirements (system packages) for installing from source. (we move the currentbuild/pkgs/debian.txtetc. there)build/pkgs/_bootstrap/distros/debian.txtetc. - would record the additional requirements for running./bootstrap(we move the currentbuild/pkgs/debian-bootstrap.txtthere)
Related:
- #28998: Add script-type _spkg_check package that installs tox, mock, pytest etc.
- #29557:
build/pkgs/_recommended- records additional recommended packages such as texlive-xetex (see https://wiki.sagemath.org/prerequisites/Ubuntu)
Possible follow-up:
build/pkgs/_toolchain/dependencies- to replace some special Makefile targets currently inbuild/make/Makefile.in.
Depends on #30940
Depends on #29655
CC: @dimpase @egourgoulhon @jhpalmieri @kiwifb @tobiasdiez @seblabbe @slel @kliem
Component: build
Keywords: sd111
Author: Matthias Koeppe
Branch/Commit: e8da106
Reviewer: Sébastien Labbé, Dima Pasechnik
Issue created by migration from https://trac.sagemath.org/ticket/29124
Description changed:
---
+++
@@ -1,7 +1,8 @@
As a follow-up to #29053.
-- `build/pkgs/PREREQ/distros/debian.txt` etc. - would record the minimal requirements (system packages) for installing from source. (to replace #29053's file build/pkgs/debian.txt)
-- `build/pkgs/BOOTSTRAP/distros/debian.txt` etc. - would record the additional requirements for running `./bootstrap` (autotools and such)
-- `build/pkgs/TOOLCHAIN/deps` - to replace some special Makefile targets currently in `build/make/deps`.
+- `build/pkgs/_prereq/distros/debian.txt` etc. - would record the minimal requirements (system packages) for installing from source. (to replace #29053's file build/pkgs/debian.txt)
+- `build/pkgs/_bootstrap/distros/debian.txt` etc. - would record the additional requirements for running `./bootstrap` (autotools and such)
+- `build/pkgs/_toolchain/deps` - to replace some special Makefile targets currently in `build/make/deps`.
-
+Related:
+- #28998: Add script-type _spkg_check package that installs tox, mock, pytest etc.Description changed:
---
+++
@@ -2,7 +2,9 @@
- `build/pkgs/_prereq/distros/debian.txt` etc. - would record the minimal requirements (system packages) for installing from source. (to replace #29053's file build/pkgs/debian.txt)
- `build/pkgs/_bootstrap/distros/debian.txt` etc. - would record the additional requirements for running `./bootstrap` (autotools and such)
+- `build/pkgs/_prereq_recommended` - records additional recommended packages such as texlive-xetex (see https://wiki.sagemath.org/prerequisitesUbuntu)
- `build/pkgs/_toolchain/deps` - to replace some special Makefile targets currently in `build/make/deps`.
Related:
- #28998: Add script-type _spkg_check package that installs tox, mock, pytest etc.
+for example on fedora-26-standard sagetex testsuite fails:
! LaTeX Error: File `makecmds.sty' not found.
Type X to quit or <RETURN> to proceed,
or enter new name. (Default extension: sty)
Enter file name:
! Emergency stop.
<read *>
l.78 \RequirePackage
{ifpdf}^^M
Description changed:
---
+++
@@ -2,7 +2,7 @@
- `build/pkgs/_prereq/distros/debian.txt` etc. - would record the minimal requirements (system packages) for installing from source. (to replace #29053's file build/pkgs/debian.txt)
- `build/pkgs/_bootstrap/distros/debian.txt` etc. - would record the additional requirements for running `./bootstrap` (autotools and such)
-- `build/pkgs/_prereq_recommended` - records additional recommended packages such as texlive-xetex (see https://wiki.sagemath.org/prerequisitesUbuntu)
+- `build/pkgs/_recommended` - records additional recommended packages such as texlive-xetex (see https://wiki.sagemath.org/prerequisitesUbuntu)
- `build/pkgs/_toolchain/deps` - to replace some special Makefile targets currently in `build/make/deps`.
Related: Installation manual also recommends Tcl/Tk - what are they used for in Sage?
http://doc.sagemath.org/html/en/installation/source.html#tcl-tk
New commits:
bf2dfdf | build/pkgs/_recommended: New |
Author: Matthias Koeppe
Description changed:
---
+++
@@ -2,9 +2,9 @@
- `build/pkgs/_prereq/distros/debian.txt` etc. - would record the minimal requirements (system packages) for installing from source. (to replace #29053's file build/pkgs/debian.txt)
- `build/pkgs/_bootstrap/distros/debian.txt` etc. - would record the additional requirements for running `./bootstrap` (autotools and such)
-- `build/pkgs/_recommended` - records additional recommended packages such as texlive-xetex (see https://wiki.sagemath.org/prerequisitesUbuntu)
- `build/pkgs/_toolchain/deps` - to replace some special Makefile targets currently in `build/make/deps`.
Related:
- #28998: Add script-type _spkg_check package that installs tox, mock, pytest etc.
+- #29557: `build/pkgs/_recommended` - records additional recommended packages such as texlive-xetex (see https://wiki.sagemath.org/prerequisitesUbuntu)
Changed branch from u/mkoeppe/add_script_packages_build_pkgs__prereq__build_pkgs__toolchain__build_pkgs__bootstrap___recommended to none
Description changed:
---
+++
@@ -6,5 +6,5 @@
Related:
- #28998: Add script-type _spkg_check package that installs tox, mock, pytest etc.
-- #29557: `build/pkgs/_recommended` - records additional recommended packages such as texlive-xetex (see https://wiki.sagemath.org/prerequisitesUbuntu)
+- #29557: `build/pkgs/_recommended` - records additional recommended packages such as texlive-xetex (see https://wiki.sagemath.org/prerequisites/Ubuntu)
Description changed:
---
+++
@@ -1,7 +1,7 @@
As a follow-up to #29053.
-- `build/pkgs/_prereq/distros/debian.txt` etc. - would record the minimal requirements (system packages) for installing from source. (to replace #29053's file build/pkgs/debian.txt)
-- `build/pkgs/_bootstrap/distros/debian.txt` etc. - would record the additional requirements for running `./bootstrap` (autotools and such)
+- `build/pkgs/_prereq/distros/debian.txt` etc. - would record the minimal requirements (system packages) for installing from source. (to replace the current `build/pkgs/debian.txt`)
+- `build/pkgs/_bootstrap/distros/debian.txt` etc. - would record the additional requirements for running `./bootstrap` (to replace the current `build/pkgs/debian-bootstrap.txt`)
- `build/pkgs/_toolchain/deps` - to replace some special Makefile targets currently in `build/make/deps`.
Related: Description changed:
---
+++
@@ -1,10 +1,13 @@
-As a follow-up to #29053.
+As a follow-up to #29053, we move the `build/pkgs/*.txt` files to a more systematic location. This will simplify some scripts.
-- `build/pkgs/_prereq/distros/debian.txt` etc. - would record the minimal requirements (system packages) for installing from source. (to replace the current `build/pkgs/debian.txt`)
-- `build/pkgs/_bootstrap/distros/debian.txt` etc. - would record the additional requirements for running `./bootstrap` (to replace the current `build/pkgs/debian-bootstrap.txt`)
-- `build/pkgs/_toolchain/deps` - to replace some special Makefile targets currently in `build/make/deps`.
+- `build/pkgs/_prereq/distros/debian.txt` etc. - records the minimal requirements (system packages) for installing from source. (we move the current `build/pkgs/debian.txt` etc. there)
+- `build/pkgs/_bootstrap/distros/debian.txt` etc. - would record the additional requirements for running `./bootstrap` (we move the current `build/pkgs/debian-bootstrap.txt` there)
Related:
- #28998: Add script-type _spkg_check package that installs tox, mock, pytest etc.
- #29557: `build/pkgs/_recommended` - records additional recommended packages such as texlive-xetex (see https://wiki.sagemath.org/prerequisites/Ubuntu)
+Possible follow-up:
+- `build/pkgs/_toolchain/dependencies` - to replace some special Makefile targets currently in `build/make/Makefile.in`.
+
+Work Issues: Update scripts to new location
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
0281b21 | tox.ini: Simplify, use sage-get-system-packages, sage-package list |
Commit: 0281b21
Branch pushed to git repo; I updated commit sha1. New commits:
3abae24 | tox.ini: Fixup - always include _bootstrap |
Changed work issues from Update scripts to new location to none
I'm not sure about the naming of the local- environments. Why should I expect as a developer that local-sudo installs system packages? What about local-system-packages-ubuntu (for root) and local-sudo-system-packages-ubuntu (for sudo)?
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
0e97683 | tox.ini: Simplify, use sage-get-system-packages, sage-package list |
da3f657 | tox.ini: Fixup - always include _bootstrap |
8fde1dc | build/bin/write-dockerfile.sh: Remove special casing of build/pkgs/$SYSTEM.txt |
8d725b8 | src/doc/en/developer/portability_testing.rst: Update paths in documentation |
I have taken out the addition of the local-root environments to keep this ticket more narrowly focused. Let's return to this in #30944.
Branch pushed to git repo; I updated commit sha1. New commits:
bd7f66c | Makefile (configure): Update dependency on moved files build/pkgs/*.txt |
Branch pushed to git repo; I updated commit sha1. New commits:
5e9366d | Merge tag '9.3.beta2' into t/30947/src_doc_bootstrap__simplify_by_using_new_options_of__sage__package_list_ |
6b9f3a3 | Merge branch 't/30947/src_doc_bootstrap__simplify_by_using_new_options_of__sage__package_list_' into t/29124/script-packages-prereq-toolchain-bootstrap |
Branch pushed to git repo; I updated commit sha1. New commits:
b0459c3 | Merge tag '9.3.beta2' into t/30951/tox_ini__build_bin_write_dockerfile_sh__simplify_by_using_new_options_of__sage__package_list_ |
a8708dd | build/bin/write-dockerfile.sh: Change bash -x back to bash |
8a575d4 | tox.ini (minimal): Use SAGE_PACKAGE_LIST_ARGS=_prereq |
f696cfc | Merge branch 't/30951/tox_ini__build_bin_write_dockerfile_sh__simplify_by_using_new_options_of__sage__package_list_' into t/29124/script-packages-prereq-toolchain-bootstrap |
a5ecf3c | bootstrap: Remove --enable-_recommended etc. |
Branch pushed to git repo; I updated commit sha1. New commits:
0e35938 | m4/sage_spkg_collect.m4: Remove _recommended etc. from the configure package summary |
Am I right to think that only the last two commits of the branch should be reviewed here?
No, there are actually a few more commits earlier that don't come from the merged dependencies
Ok. Then, could the branch be rebased on top of the dependencies, to make the changes made by this ticket consecutive and easier to read (during the review and/or after).
Changed keywords from none to sd111
Hoping we can make progress on this ticket this week - https://wiki.sagemath.org/days111
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
2792314 | Merge branch 't/30947/src_doc_bootstrap__simplify_by_using_new_options_of__sage__package_list_' into t/29124/script-packages-prereq-toolchain-bootstrap |
c901ba6 | Add script packages build/pkgs/_prereq, build/pkgs/_bootstrap |
5b2f4fb | build/pkgs/_prereq/spkg-configure.m4: New |
7ad8286 | src/doc/bootstrap: Remove special casing of build/pkgs/$SYSTEM.txt - _prereq is now just a standard package with spkg-configure.m4 |
ad24294 | tox.ini: Simplify, use sage-get-system-packages, sage-package list |
1a77a6b | build/bin/write-dockerfile.sh: Remove special casing of build/pkgs/$SYSTEM.txt |
35241e7 | src/doc/en/developer/portability_testing.rst: Update paths in documentation |
68baff1 | Makefile (configure): Update dependency on moved files build/pkgs/*.txt |
fae4bd0 | bootstrap: Remove --enable-_recommended etc. |
a9bd145 | m4/sage_spkg_collect.m4: Remove _recommended etc. from the configure package summary |
Thank you. I do have lots of other todos this week, but I hope to be able to review few tickets during the week. I will start with this one and the other ones you put me in cc.
I am not sure if commit a9bd145 works because I still see underscore packages listed in the list of SPKGs which did not find equivalent system packages during the run of ./configure:
notice: the following SPKGs did not find equivalent system packages:
_recommended boost coxeter3 gp2c igraph isl libsemigroups pari_elldata pari_galpol pari_nftables pari_seadata
checking for the package system in use... debian
configure:
hint: installing the following system packages, if not
already present, is recommended and may avoid having to
build them (though some may have to be built anyway):
$ sudo apt-get update
$ sudo apt-get install texlive-latex-extra texlive-xetex latexmk pandoc dvipng default-jdk ffmpeg libavdevice-dev libboost-dev pari-gp2c libigraph-dev libisl-dev
After installation, re-run configure using:
$ ./config.status --recheck && ./config.status
Ok, I see, the goal of commit a9bd145 was to remove _recommended elsewhere in the log:
config.log on 9.3.beta3:
## ----------------------------------------------------------------------- ##
## Build status for each package: ##
## ----------------------------------------------------------------------- ##
4ti2-1.6.7.p0: does not support check for system package; optional, will be installed as an SPKG
_recommended-none: no suitable system package; optional, use "./configure --enable-_recommended" to install
alabaster-0.7.12: does not support check for system package; will be installed as an SPKG
config.log with this branch on top of 9.3.beta3:
## ----------------------------------------------------------------------- ##
## Build status for each package: ##
## ----------------------------------------------------------------------- ##
4ti2-1.6.7.p0: does not support check for system package; optional, will be installed as an SPKG
_bootstrap-none: does not support check for system package; optional, use "./configure --enable-_bootstrap" to install
_prereq-none: does not support check for system package; will be installed as an SPKG
_recommended-none: no suitable system package; optional, use "./configure --enable-_recommended" to install
alabaster-0.7.12: does not support check for system package; will be installed as an SPKG
config.log with this branch on top of 9.3.beta3 (after running make first):
## ----------------------------------------------------------------------- ##
## Build status for each package: ##
## ----------------------------------------------------------------------- ##
4ti2-1.6.7.p0: does not support check for system package; optional, will be installed as an SPKG
alabaster-0.7.12: does not support check for system package; will be installed as an SPKG
appnope-0.1.0.p0: does not support check for system package; will be installed as an SPKG
Still, I would suggest to also remove it from the list notice: the following SPKGs did not find equivalent system packages:.
Reviewer: Sébastien Labbé
Branch pushed to git repo; I updated commit sha1. New commits:
184ae6d | Merge tag '9.3.beta3' into t/30947/src_doc_bootstrap__simplify_by_using_new_options_of__sage__package_list_ |
084fbf6 | src/doc/bootstrap: Use ./sage |
571cc49 | Merge branch 't/30947/src_doc_bootstrap__simplify_by_using_new_options_of__sage__package_list_' into t/29124/script-packages-prereq-toolchain-bootstrap |
Replying to @seblabbe:
Still, I would suggest to also remove it from the list
notice: the following SPKGs did not find equivalent system packages:.
I agree it's not pretty. But all of the SPKGs listed there expand/transform to system packages that are suggested to be installed. Without the _recommended, it will be a mystery where some of the suggested system packages are coming from. So I think it's better to keep it there.
Ok. works for me.
Thank you!
Merge conflict
Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
9c5e8a2 | add references to `bin/sage-build-env` and `bin/sage-build-env-config` |
4b8933b | remove incorrect `$` in `build/bin/sage-build-env` |
071b8cc | Revert "add references to `bin/sage-build-env` and `bin/sage-build-env-config`" |
c51e752 | correct comments to makefile |
aa7ecd6 | fix that we might have changed $FCFLAGS during configure |
94326db | better solution for dealing with -ffree-form |
87dae61 | keep FCFLAGS during gfortran configure |
7577dd4 | added quotes |
3bb3099 | prepare preciouos variables |
c529729 | Merge commit '3bb309944b7e8542b2ac88ed3c9d9a60e68644d7' of git://trac.sagemath.org/sage into t/29124/script-packages-prereq-toolchain-bootstrap |
Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:
b73d6f0 | Merge tag '9.3.beta4' into t/29124/script-packages-prereq-toolchain-bootstrap |
Merged current beta
**********************************************************************
File "src/sage/misc/package.py", line 427, in sage.misc.package.standard_packages
Failed example:
installed[0], installed[-1] # optional - build
Expected:
('alabaster', 'zn_poly')
Got:
('_prereq', 'zn_poly')
**********************************************************************
1 item had failures:
1 of 4 in sage.misc.package.standard_packages
[49 tests, 1 failure, 3.31 s]
----------------------------------------------------------------------
Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
97942e3 | sage.misc.package.pip_installed_packages: Only normalize when optional arg normalization='spkg' |
6706182 | sage.misc.package.list_packages: Return early if SAGE_PKGS is not available |
9258849 | sage.misc.package.list_packages: Streamline handling of exclude_pip |
7f80e3e | sage.misc.package.list_packages: Actually add new items to the list |
78bae1f | Merge tag '9.3.beta3' into t/30940/src_bin_sage_list_packages__make_it_work_if_sage_root_is_not_available |
3cf5ee4 | Merge commit 'a50ddf88975086b14a49895e371477df00fd57b5' of git://trac.sagemath.org/sage into t/30940/src_bin_sage_list_packages__make_it_work_if_sage_root_is_not_available |
b4927e1 | sage.misc.package: Remove/adjust non-robust doctests |
e5fe752 | Merge tag '9.3.beta4' into t/30940/src_bin_sage_list_packages__make_it_work_if_sage_root_is_not_available |
78ff9d5 | src/sage/misc/package.py: Add one more # optional - build |
a44042f | Merge branch 't/30940/src_bin_sage_list_packages__make_it_work_if_sage_root_is_not_available' into t/29124/script-packages-prereq-toolchain-bootstrap |
Branch pushed to git repo; I updated commit sha1. New commits:
64bde5f | Merge tag '9.3.beta5' into t/30940/src_bin_sage_list_packages__make_it_work_if_sage_root_is_not_available |
e9a7572 | src/sage/misc/package.py: Improve source formatting |
c7bcda9 | Merge branch 't/30940/src_bin_sage_list_packages__make_it_work_if_sage_root_is_not_available' into t/29124/script-packages-prereq-toolchain-bootstrap |
9988c5f | ci-cygwin*.yml: Adjust to new script packages _bootstrap, _prereq |
The last commit fixes the issue observed in #29124:
moved the files build/pkgs/*.txt to new locations but forgot to update the CI scripts for Cygwin, which had led to failures like this (https://github.com/mkoeppe/sage/runs/1561097256):
sed: can't read ./build/pkgs/cygwin.txt: No such file or directory
sed: can't read ./build/pkgs/cygwin-bootstrap.txt: No such file or directory
Chocolatey v0.10.15
Package name is required. Please pass at least one package name to install.
Error: Process completed with exit code 1.
Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
6f3b864 | build/pkgs/[s-t]*/SPKG.rst: Reformat title in format 'spkg_name: Short description' |
23558f8 | build/pkgs/[t-z]*/SPKG.rst: Reformat title in format 'spkg_name: Short description' -- completes coverage of all standard packages |
17789df | build/pkgs/[z-z]*/SPKG.rst: Reformat title in format 'spkg_name: Short description' -- completes coverage of all optional packages |
944408d | build/pkgs/[a-z]*/SPKG.rst: Reformat title in format 'spkg_name: Short description' -- completes coverage of all experimental packages |
feafa60 | src/doc/en/installation/standard_packages.rst: Remove |
58c63e1 | src/doc/bootstrap: Add brief explanation of the standard/optional/experimental package types |
54d4fb1 | extra line after version |
17f2f24 | Merge branch 't/29655/improve_build_pkgs___spkg_rst' into t/29124/script-packages-prereq-toolchain-bootstrap |
e7a768b | build/pkgs/{_bootstrap,_prereq}/SPKG.rst: Reformat title |
6ff6545 | Move build/pkgs/opensuse*.txt to build/pkgs/{_bootstrap,_prereq}/distros |
Still needs review
./bootstrap is getting very slow - 3 times slower than ./configure, over 60 sec on a relatively fast machine!
On this branch I'm getting
sage -t --warn-long 53.2 --random-seed=0 src/sage/tests/cmdline.py
**********************************************************************
File "src/sage/tests/cmdline.py", line 219, in sage.tests.cmdline.test_executable
Failed example:
print(out) # optional - build
Expected:
Found local metadata for sqlite-...
SQLite
======
...
SQLite is a software library that implements a self-contained,
serverless, zero-configuration, transactional SQL database engine.
...
Got:
sqlite: An SQL database engine
==============================
<BLANKLINE>
Description
-----------
<BLANKLINE>
SQLite is a software library that implements a self-contained,
serverless, zero-configur
...
and one more similar failure in this file.
Not sure whether it's coming from this ticket, or one of dependencies.
Changed reviewer from Sébastien Labbé to Sébastien Labbé, Dima Pasechnik
Branch pushed to git repo; I updated commit sha1. New commits:
2eac7a1 | src/doc/bootstrap: Use sage-spkg-info directly instead of 'sage --info', much faster now |
3fe8d4c | src/sage/tests/cmdline.py: Adjust doctests |
b57fecc | Merge branch 't/29655/improve_build_pkgs___spkg_rst' into t/29124/script-packages-prereq-toolchain-bootstrap |
hmm, now I see
File "src/sage/tests/cmdline.py", line 219, in sage.tests.cmdline.test_executable
Failed example:
print(out) # optional - build
Expected:
SQLite
======
...
SQLite is a software library that implements a self-contained,
serverless, zero-configuration, transactional SQL database engine.
...
Got:
sqlite: An SQL database engine
==============================
<BLANKLINE>
Description
-----------
<BLANKLINE>
SQLite is a software library that implements a self-contained,
serverless, zero-configuration, transactional SQL database engine.
<BLANKLINE>
...
(and the same in line 232)
So more changes of these test outputs are needed.
The different heading (sqlite: An SQL database engine) comes from src/doc/en/reference/spkg/sqlite.rst, as far as me and grep can tell.
OK, good.
Thanks!
Changed branch from u/mkoeppe/script-packages-prereq-toolchain-bootstrap to e8da106