./bootstrap: Format build/pkgs/SPKG/distros/ information to produce the apt-get/yum command lines shown in the installation manual
Closed this issue · 60 comments
When trying to build Sage on a very bare fedora install (e.g. a Docker container) the packages listed in the build instructions at:
https://doc.sagemath.org/html/en/installation/source.html#linux-prerequisite-installation
are insufficient. At the very least it also needs findutils and which, and currently it also requires python2, though that should be fixed as part of #26953. The need for which comes, at the very least, from MPIR's configure script, which uses which at least on Linux.
With #29053, the necessary distribution packages for an installation are cataloged in build/pkgs/fedora.txt, build/pkgs/SPKG/distros/fedora.txt and similar for other distributions.
In this ticket, we add to bootstrap some code to keep the yum (and apt-get for debian) command-lines in the installation manual up to date. #29053 prepared this by isolating these command lines in separate .txt files in src/doc.
CC: @dimpase @embray @jhpalmieri
Component: documentation
Author: Matthias Koeppe
Branch: 688c68e
Reviewer: John Palmieri, Dima Pasechnik
Issue created by migration from https://trac.sagemath.org/ticket/26964
Retarging tickets optimistically to the next milestone. If you are responsible for this ticket (either its reporter or owner) and don't believe you are likely to complete this ticket before the next release (8.7) please retarget this ticket's milestone to sage-pending or sage-wishlist.
Removing most of the rest of my open tickets out of the 8.7 milestone, which should be closed.
Description changed:
---
+++
@@ -2,6 +2,10 @@
https://doc.sagemath.org/html/en/installation/source.html#linux-prerequisite-installation
-are insufficient. At the very least it also needs `findutils` and `which`, and currently it also requires `python2`, though that should be fixed as part of #26953.
+are insufficient. At the very least it also needs `findutils` and `which`, and currently it also requires `python2`, though that should be fixed as part of #26953. The need for `which` comes, at the very least, from MPIR's `configure` script, which uses `which` at least on Linux.
-The need for `which` comes, at the very least, from MPIR's `configure` script, which uses `which` at least on Linux.
+With #29053, the necessary distribution packages for an installation are cataloged in `build/pkgs/fedora.txt`, `build/pkgs/SPKG/distros/fedora.txt` and similar for other distributions.
+
+In this ticket, we add to `bootstrap` some code to keep the `yum` (and `apt-get` for debian) command-lines in the installation manual up to date. #29053 prepared this by isolating these command lines in separate .txt files in `src/doc`.
+
+I'm not sure how I feel about using the bootstrap script for this purpose. I think I would prefer if it were done by configure instead. Reason: The bootstrap script should really be limited to tasks specific to generating the configure file (whether running autoreconf or downloadng it). Other than that, I think this is a great idea!
No, no, bootstrap is exactly the right phase for this, not configure. Let me explain more.
bootstrap will prepare a source tree so that
make sdistcan be runconfigurecan be run by a user (who does not have autotools)src/setup.py sdistcan be run.src/configurecan be run (#29119)
None of this depends on the configuration determined by configure!
As the installation manual sources are part of the sagelib sdist, the command lines of this ticket should be prepared in the bootstrap phase.
Replying to @mkoeppe:
As the installation manual sources are part of the sagelib sdist, the command lines of this ticket should be prepared in the bootstrap phase.
I want to clarify something about this here: Do you mean the sagelib sdist, or the sage-the-distribution sdist?
I sort of see your point, but I'm still not sure why it needs to go here. Also, as part of #29119 I'm adding a src/bootstrap script for sagelib itself, so maybe something like this would make more sense to go there.
Anyways, setting back to needs_work since no branch is attached. Maybe then I'll see what you're talking about.
Couldn't this also be done as part of make sdist?
Replying to @embray:
Replying to @mkoeppe:
As the installation manual sources are part of the sagelib sdist, the command lines of this ticket should be prepared in the bootstrap phase.
I want to clarify something about this here: Do you mean the sagelib sdist, or the sage-the-distribution sdist?
Both.
Currently, of course, only sage-the-distribution has a working sdist.
sagelib's sdist needs fixing. You might remember ticket #21516 - Fix sagelib sdist (src/setup.py sdist).
I sort of see your point, but I'm still not sure why it needs to go here. Also, as part of #29119 I'm adding a
src/bootstrapscript for sagelib itself, so maybe something like this would make more sense to go there.
In my opinion, there should only be one developer-invoked bootstrap script (the one for the distribution) -- because everything will depend on the distribution's build/pkgs files.
SAGE_ROOT/bootstrap could certainly call a new script src/bootstrap for generating files that end up in src/. If you create that in #29119, fine.
Anyways, setting back to needs_work since no branch is attached. Maybe then I'll see what you're talking about.
Sure. #29041 (src/requirements.txt, src/constraints.txt, src/setup.cfg) has a branch, needs review.
Replying to @embray:
Couldn't this also be done as part of
make sdist?
No, because developers who build sage out of a git worktree do not run make sdist.
Commit: 438bc7f
Branch pushed to git repo; I updated commit sha1. New commits:
438bc7f | ./bootstrap, src/doc/bootstrap: Auto-generate apt-get/yum command lines |
Author: Matthias Koeppe
Branch pushed to git repo; I updated commit sha1. New commits:
3810bc9 | Clean and ignore the generated files |
Branch pushed to git repo; I updated commit sha1. New commits:
bdd69c3 | build/bin/write-dockerfile.sh: Copy src/doc/bootstrap early |
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
8be072b | build/bin/write-dockerfile.sh: Copy src/doc/bootstrap early |
something is not quite right:
(sage-sh) dimpase@clpc171:sage$ ./build/bin/sage-print-system-package-command debian gmp
sudo apt-get gmp
(sage-sh) dimpase@clpc171:sage$ ./build/bin/sage-print-system-package-command conda gmp
# gmp the following packages:
- so
debianworks,condadoes not...
you forgot the verb install in both commands
Branch pushed to git repo; I updated commit sha1. New commits:
eb7a1d9 | build/bin/sage-print-system-package-command: Print usage message if not enough arguments provided |
Branch pushed to git repo; I updated commit sha1. New commits:
146a520 | build/bin/sage-print-system-package-command: Do not print a command if no packages are to be installed |
Replying to @mkoeppe:
Replying to @embray:
Couldn't this also be done as part of
make sdist?No, because developers who build sage out of a git worktree do not run
make sdist.
Sure, but what is required for running make sdist? It seems to me this should be done in two separate cases: One when running make sdist and one when developing out of source. In either case I don't think the bootstrap script is the appropriate place for it.
The only reason you seem to be putting it there is that it would happen to cover both cases, but I think both cases should be handled separately.
I think I might see part of the problem here: It is true that the information you need to generate these documentation pages lives in sage-the-distribution, above the sagelib sources. If you want to package sagelib in a stand-alone fashion you need to get access to this data somehow, but you can't, so the file is generated sort of externally to sagelib, using data from sage-the-distribution.
A similar problem applies to #29119. In the prototype for that I am working on a more general solution to the problem, which is that ./bootstrap makes hard links of files sagelib needs into the sagelib sources under src/. That way, only one copy of the relevant files needs to exist in the existing combined repository, but if sagelib is packaged separately it will retain the necessary files.
I think having that might serve as a solution here too. My feeling is that these documentation pages should be generated by the docbuilder itself. As long as it has access to information about Sage's dependencies it can do this, without the top-level ./bootstrap script having to take any special action.
In other words, what we ultimately want is to be able to build Sage's docs without direct dependency on the sage-the-distribution stuff. As it stands, I fear this PR is introducing too much direct dependency on sage-the-distribution just to be able to build Sage's docs. What we would really want is to be able to run setup.py sdist, the a source tarball just for sagelib and its docs, and later unpack it somewhere else and still be able to build the docs (and sage itself).
How is this supposed to work with an incremental build?
$ make bootstrap
make: `bootstrap' is up to date.
$ make configure
make: `configure' is up to date.
$ make
...
OSError: /Users/jpalmier/Desktop/Sage/git/sage/src/doc/en/installation/source.rst:228: WARNING: Include file '/Users/jpalmier/Desktop/Sage/git/sage/src/doc/en/installation/debian.txt' not found or reading it failed
Replying to @jhpalmieri:
How is this supposed to work with an incremental build?
Thanks for catching this -- I need to add some dependencies
Replying to @embray:
I think I might see part of the problem here: It is true that the information you need to generate these documentation pages lives in sage-the-distribution, above the sagelib sources. If you want to package sagelib in a stand-alone fashion you need to get access to this data somehow, but you can't, so the file is generated sort of externally to sagelib, using data from sage-the-distribution.
A similar problem applies to #29119. [...]
There is a really easy solution:
bootstrapis only allowed in the full distribution source, not in a separate sagelib source. No hardlinks necessary. Only developers bootstrap.- Users don't bootstrap, they start with the configure phase.
- sdist of the distribution and sdist of sagelib generate bootstrapped source archives.
Replying to @embray:
Sure, but what is required for running
make sdist?
Obviously, it requires exactly a bootstrapped source tree!
It seems to me this should be done in two separate cases: One when running
make sdistand one when developing out of source. In either case I don't think the bootstrap script is the appropriate place for it.The only reason you seem to be putting it there is that it would happen to cover both cases, but I think both cases should be handled separately.
bootstrap is the phase before configure, which does not depend on the configuration of the system.
Branch pushed to git repo; I updated commit sha1. New commits:
ce34ce1 | Makefile: Add dependencies on build/pkgs/*.txt build/pkgs/*/distros/*.txt |
Replying to @mkoeppe:
Replying to @jhpalmieri:
How is this supposed to work with an incremental build?
Thanks for catching this -- I need to add some dependencies
Done.
That doesn't help, because the necessary text files are not build/pkgs/*.txt build/pkgs/*/distros/*.txt but rather in src/doc/en/installation. By the way, I'm not sure whether I like the old style
$ sudo apt-get install binutils pixz gcc g++ gfortran make m4 perl tar \
git patch openssl libssl-dev libz-dev bc libbz2-dev liblzma-dev libgmp-dev \
libffi-dev libgf2x-dev libcurl4-openssl-dev libzmq3-dev curl yasm \
pkg-config libntl-dev libmpfr-dev libmpc-dev libflint-dev \
libpcre3-dev libgd-dev libflint-dev libflint-arb-dev \
libsymmetrica2-dev gmp-ecm libecm-dev libisl-dev libgivaro-dev \
libpari-dev pari-gp2c libec-dev liblrcalc-dev \
libm4ri-dev libm4rie-dev liblfunction-dev lcalc \
libopenblas-dev r-base-dev libgsl-dev libcliquer-dev cliquer
or the new style
$ sudo yum install binutils boost-devel bzip2 bzip2-devel curl eclib-devel findutils gcc gcc gcc-c++ gcc-c++ gcc-gfortran gcc-gfortran gfan git givaro-devel gmp gmp-devel gmp-ecm-devel libcurl-devel libmpc-devel lrcalc-devel m4 m4ri-devel m4rie-devel make mpfr-devel nauty ncurses-devel ntl-devel openblas-devel pari-devel pari-elldata pari-galdata pari-galdata pari-galpol pari-gp pari-seadata patch perl perl-ExtUtils-MakeMaker pkg-config python3 readline-devel rw-devel symmetrica-devel tar which xz xz-devel yasm zlib-devel
better.
Is there a ticket which adds homebrew information to the installation/source.rst?
Replying to @jhpalmieri:
That doesn't help, because the necessary text files are not
build/pkgs/*.txt build/pkgs/*/distros/*.txtbut rather insrc/doc/en/installation.
The generated files are in src/doc/en/installation. The source files are in build/pkgs/. Hence the dependency that I added. Changes to the .txt files trigger bootstrapping.
Replying to @jhpalmieri:
Is there a ticket which adds homebrew information to the
installation/source.rst?
#29104 is for homebrew. I haven't added the package information there yet -- any help is very welcome!
Replying to @mkoeppe:
Replying to @jhpalmieri:
That doesn't help, because the necessary text files are not
build/pkgs/*.txt build/pkgs/*/distros/*.txtbut rather insrc/doc/en/installation.The generated files are in
src/doc/en/installation. The source files are inbuild/pkgs/. Hence the dependency that I added. Changes to the .txt files trigger bootstrapping.
But this ticket doesn't change any of those .txt files. In my case I have Sage 9.1.beta3, then I switch to this branch and make fails, even with your latest branch. Is there no way to avoid this?
$ git checkout t/26964/__bootstrap__format_build_pkgs_spkg_distros__information_to_produce_the_apt_get_yum_command_lines_shown_in_the_installation_manual
Switched to branch 't/26964/__bootstrap__format_build_pkgs_spkg_distros__information_to_produce_the_apt_get_yum_command_lines_shown_in_the_installation_manual'
$ make configure
make: `configure' is up to date.
./bootstrap -d generates the correct files, but I have to run it manually.
the idea is that one can make configure is not a good one. configure is "built" by autoconf, not by make.
can we get rid of configure target?
Replying to @jhpalmieri:
That doesn't help, because the necessary text files are not
build/pkgs/*.txt build/pkgs/*/distros/*.txtbut rather insrc/doc/en/installation. By the way, I'm not sure whether I like the old style$ sudo apt-get install binutils pixz gcc g++ gfortran make m4 perl tar \ git patch openssl libssl-dev libz-dev bc libbz2-dev liblzma-dev libgmp-dev \ libffi-dev libgf2x-dev libcurl4-openssl-dev libzmq3-dev curl yasm \ pkg-config libntl-dev libmpfr-dev libmpc-dev libflint-dev \ libpcre3-dev libgd-dev libflint-dev libflint-arb-dev \ libsymmetrica2-dev gmp-ecm libecm-dev libisl-dev libgivaro-dev \ libpari-dev pari-gp2c libec-dev liblrcalc-dev \ libm4ri-dev libm4rie-dev liblfunction-dev lcalc \ libopenblas-dev r-base-dev libgsl-dev libcliquer-dev cliqueror the new style
$ sudo yum install binutils boost-devel bzip2 bzip2-devel curl eclib-devel findutils gcc gcc gcc-c++ gcc-c++ gcc-gfortran gcc-gfortran gfan git givaro-devel gmp gmp-devel gmp-ecm-devel libcurl-devel libmpc-devel lrcalc-devel m4 m4ri-devel m4rie-devel make mpfr-devel nauty ncurses-devel ntl-devel openblas-devel pari-devel pari-elldata pari-galdata pari-galdata pari-galpol pari-gp pari-seadata patch perl perl-ExtUtils-MakeMaker pkg-config python3 readline-devel rw-devel symmetrica-devel tar which xz xz-devel yasm zlib-develbetter.
the new style is built automatically, whereas the old one was manual.
One can surely hack further to add insertion of \ after ~80 chars, but it's certainlty minor...
Branch pushed to git repo; I updated commit sha1. New commits:
688c68e | Makefile [configure]: Add dependency on bootstrap, src/doc/bootstrap |
Replying to @jhpalmieri:
But this ticket doesn't change any of those .txt files. In my case I have Sage 9.1.beta3, then I switch to this branch and
makefails, even with your latest branch. Is there no way to avoid this?
Thanks!
I've added more dependencies to the make configure target that should take care of this case.
Replying to @dimpase:
the idea is that one can
make configureis not a good one.configureis "built" by autoconf, not by make.
I don't think there's anything wrong with these make targets -- also autotools have them (in maintainer mode).
a typical autotools project won't have a Makefile in unbuilt state, only Makefile.am
That is true, but there are Makefile targets that regenerate configure etc. from configure.ac.
That works for me, thanks.
I don't in general ever use make configure explicitly, but I was trying to get the previous versions of this branch to work so I tried it. With the most recent branch, I can just check it out and run make.
is sage-print-system-package-command meant to runnable by the user, or it's a "service" script?
It's an internal script of sage the distribution. That's why it's in build/bin.
Reviewer: John Palmieri, Dima Pasechnik
OK, the only nitpck is that on modern Fedora's it's dnf, not yum.
Thank you!