sagemath/sage

./configure --prefix=SAGE_LOCAL

Closed this issue · 157 comments

I propose to support choosing a location for the SAGE_LOCAL tree, using

./configure --prefix=SAGE_LOCAL

(the default would be, as before, the local subdirectory of SAGE_ROOT - see patch on the ticket).

I am fully aware that Sage's build system does not have a separation between 'make' and 'make install'; see #21495 for that.

Nevertheless, SAGE_LOCAL should be considered the same as the prefix in the sense of the autotools.

This ticket is a step towards #21566.

Release manager configure tarball: http://sage.ugent.be/www/jdemeyer/sage/configure-214.tar.gz

CC: @jdemeyer @embray @kiwifb @nexttime @dimpase

Component: build

Author: Matthias Koeppe, Jeroen Demeyer

Branch: e5926b1

Reviewer: Jeroen Demeyer, Erik Bray, Matthias Koeppe

Issue created by migration from https://trac.sagemath.org/ticket/21479

Description changed:

--- 
+++ 
@@ -3,7 +3,7 @@
 ```
 ./configure --prefix=SAGE_LOCAL
 ```
-(the default would be, as before, the `local` subdirectory of SAGE_ROOT).
+(the default would be, as before, the `local` subdirectory of SAGE_ROOT - see patch on the ticket).
 
 I am fully aware that Sage's build system does not have a separation between 'make' and 'make install' (this would require all packages to support a two-stage install (DESTDIR).) Nevertheless, SAGE_LOCAL should be considered the same as the `prefix` in the sense of the autotools.
 

New commits:

fa16faeUse AC_PREFIX_DEFAULT to default prefix to SAGE_ROOT/local

Commit: fa16fae

Author: Matthias Koeppe

Description changed:

--- 
+++ 
@@ -5,6 +5,8 @@
 ```
 (the default would be, as before, the `local` subdirectory of SAGE_ROOT - see patch on the ticket).
 
-I am fully aware that Sage's build system does not have a separation between 'make' and 'make install' (this would require all packages to support a two-stage install (DESTDIR).) Nevertheless, SAGE_LOCAL should be considered the same as the `prefix` in the sense of the autotools.
+I am fully aware that Sage's build system does not have a separation between 'make' and 'make install'; see #21480 for that. 
+
+Nevertheless, SAGE_LOCAL should be considered the same as the `prefix` in the sense of the autotools.
 
 

Description changed:

--- 
+++ 
@@ -5,7 +5,7 @@
 ```
 (the default would be, as before, the `local` subdirectory of SAGE_ROOT - see patch on the ticket).
 
-I am fully aware that Sage's build system does not have a separation between 'make' and 'make install'; see #21480 for that. 
+I am fully aware that Sage's build system does not have a separation between 'make' and 'make install'; see #21495 for that. 
 
 Nevertheless, SAGE_LOCAL should be considered the same as the `prefix` in the sense of the autotools.
 
comment:6

(i completely agree with this)

a main obstacle is: the build system writes to $prefix during the build...

i suggest to do the transition one-package-at-a-time, and after --disable-$package is in place.

with --disable-all, it will be easiest to avoid writing to $prefix.

comment:7

On this short term ticket, we do allow "make" to write into $prefix; and "make install" will be a no-op.

More ambitious plans are to be discussed on the long term ticket #21495, not on this ticket. Thanks.

comment:8

i agree that there are intermediate steps to take. but i don't yet fully understand this approach.

i think the behaviour of just "make" should not change, regardless of --prefix. it works perfectly well and as intended right now. clearly it should do something different under the hood, but that's not on this ticket.

we do allow "make" to write into $prefix

why would you need/want that? can you please give an example?

my feeling is, that entangling prefix and SAGE_LOCAL further complicates the transition considerably. there will be no way to tell whether "this part still uses SAGE_LOCAL" vs. "this has already been cleaned". newcomers tend to grep a variable from the code, see how it is used and use it the same way. for a looong time to come.

Changed author from Matthias Koeppe to Matthias Koeppe, Jeroen Demeyer

Description changed:

--- 
+++ 
@@ -8,5 +8,3 @@
 I am fully aware that Sage's build system does not have a separation between 'make' and 'make install'; see #21495 for that. 
 
 Nevertheless, SAGE_LOCAL should be considered the same as the `prefix` in the sense of the autotools.
-
-

Changed author from Matthias Koeppe, Jeroen Demeyer to Matthias Koeppe

comment:10

I made a new ticket (#21501) to allow $SAGE_LOCAL to be changed.

Dependencies: #21501

comment:11

i wrote

entangling prefix and SAGE_LOCAL further complicates the transition considerably.

@mkoeppe i've thought about it. it seems, the alternatives are much worse (#21501) or much more ambitious.

please go ahead with prefix==SAGE_LOCAL.

please consider to add a remark in some documentation (better place: configure --help?) on this temporary sort of prefix, such as

"currently, make install is a no-op, that will change (hopefully). you should not expect a fully functional installation in $prefix before make install has finished."

comment:12

Thanks Felix for the discussion.

If you want to help, could you work on rebasing #15105.

Branch pushed to git repo; I updated commit sha1. New commits:

64e697dAllow a custom $SAGE_LOCAL directory
dc81cf6Merge remote-tracking branch 'trac/u/jdemeyer/ticket/21501' into t/21479/__configure___prefix_sage_local
ac12edcMerge tag '7.4.beta5' into t/21479/__configure___prefix_sage_local
924b30bHandle --prefix

Changed commit from fa16fae to 924b30b

Branch pushed to git repo; I updated commit sha1. New commits:

6f8eaacFixup

Changed commit from 924b30b to 6f8eaac

comment:15

Here's a first version of what I have in mind.

Some concerns:

  • bootstrap. It's somewhat unclear what to do here. By the way, is it supposed to use the distribution's Python or ours?
  • sage-location.
comment:16

I don't like the duplication in build/make/install and src/bin/sage-env.

Here is a suggestion: add a new file, say src/bin/sage-config (or whatever you want to call it) to deal with configurable things like SAGE_LOCAL. You can also define the other SAGE_* directories (except SAGE_ROOT) there.

Then you can source this file in both src/bin/sage-env and build/make/install, which can then be independent of configure.

And I don't see the point of changing $SAGE_ROOT/sage because that is additional duplication.

comment:17

Replying to @mkoeppe:

Some concerns:

  • bootstrap. It's somewhat unclear what to do here. By the way, is it supposed to use the distribution's Python or ours?
  • sage-location.

In both cases, I don't get what the problem is.

Branch pushed to git repo; I updated commit sha1. New commits:

c9091f5Have sage-env, sage, install read in generated file sage-env-config
66fc193Fix for locating sage-env-config from sage-env

Changed commit from 6f8eaac to 66fc193

comment:19

Replying to @jdemeyer:

Here is a suggestion: add a new file, say src/bin/sage-config (or whatever you want to call it) to deal with configurable things like SAGE_LOCAL. You can also define the other SAGE_* directories (except SAGE_ROOT) there.

Thanks for the suggestion.
The ticket is indeed simpler that way.
Please take a look.

comment:20

For easier testing, I plan to review this after #21501 and #21480 are in a beta release. You can remind me when that happens.

Branch pushed to git repo; I updated commit sha1. New commits:

e0dc9a3Merge tag '7.4.beta6' into t/21479/__configure___prefix_sage_local

Changed commit from 66fc193 to e0dc9a3

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

751bd0fReword TODO item
3a8cc0eFix typo in comment
0dd9c50Respect environment variable MAKE
17f90d8beautification
e5f9065More comments
7791cd9Remove --buildbase code
74169e7Pass SAGE_SRC to generate_py_source.mk
0394333Add new file to MANIFEST.in
fdedb02Install SAGE_LOCAL/bin/sage instead of SAGE_ROOT/sage as Jupyter kernel
8215254Merge branch 't/21480/keep_src__clean_by_using___build_base_when_building_sagelib' into t/21479/__configure___prefix_sage_local

Changed commit from e0dc9a3 to 8215254

comment:23

Branch is now on top of 7.4.beta6 (which has #21501) + #21480.

Description changed:

--- 
+++ 
@@ -8,3 +8,5 @@
 I am fully aware that Sage's build system does not have a separation between 'make' and 'make install'; see #21495 for that. 
 
 Nevertheless, SAGE_LOCAL should be considered the same as the `prefix` in the sense of the autotools.
+
+This ticket is a step towards #21566.
comment:25

Replying to @mkoeppe:

Here's a first version of what I have in mind.

Some concerns:

  • bootstrap. It's somewhat unclear what to do here. By the way, is it supposed to use the distribution's Python or ours?

Just to answer this question: the bootstrap package is intended to use the system Python (after all, it's needed well before Sage's Python is ever built).

Branch pushed to git repo; I updated commit sha1. New commits:

06fe631Merge tag '7.4.rc1' into t/21479/__configure___prefix_sage_local

Changed commit from 8215254 to 06fe631

comment:27

Branch is now on top of 7.4.rc1.

Jeroen - ready for review.

Changed commit from 06fe631 to 1adc18c

Branch pushed to git repo; I updated commit sha1. New commits:

1adc18cMerge tag '7.4' into t/21479/__configure___prefix_sage_local
comment:29

Now on top of 7.4.

Branch pushed to git repo; I updated commit sha1. New commits:

d82cfbcMerge tag '7.5.beta0' into t/21479/__configure___prefix_sage_local

Changed commit from 1adc18c to d82cfbc

comment:32

What does this comment mean:

## The first test does not actually work (see above).

("see above" can be really confusing because you don't know where to look.

Comments like these don't belong in the sources but in a Trac ticket:

	# TODO: Making directories in $prefix
	# should be done by 'make', not configure.
# TODO: Should the install hierarchy (local) really be deleted by make distclean?
#SAGE_ENV_DIR=`dirname $0`    ### Does not work. $0 is /bin/bash
## Note this is a bashism, not sure if this is allowed in this file.
comment:33

Then a more serious remark: I doubt that you are using BASH_SOURCE correctly. From the bash man page:

       BASH_SOURCE
              An array variable whose members are the source filenames where the corresponding shell function names in the FUNCNAME array variable  are
              defined.  The shell function ${FUNCNAME[$i]} is defined in the file ${BASH_SOURCE[$i]} and called from ${BASH_SOURCE[$i+1]}.

At the very least, I need more explanation of why your code does the right thing.

comment:35

Replying to @jdemeyer:

What does this comment mean:

## The first test does not actually work (see above).

("see above" can be really confusing because you don't know where to look.

It's referring to dirname $0 -- which does not give the location of sage-env but rather /bin/bash. This is because sage-env is sourced (.) rather than executed.

Note this is existing code in sage-env which simply never worked.

Comments like these don't belong in the sources but in a Trac ticket:

I know, these comments are for the reviewer. I'll be happy remove them after we had a discussion about them... So do you have a comment on these?

	# TODO: Making directories in $prefix
	# should be done by 'make', not configure.
# TODO: Should the install hierarchy (local) really be deleted by make distclean?
#SAGE_ENV_DIR=`dirname $0`    ### Does not work. $0 is /bin/bash
## Note this is a bashism, not sure if this is allowed in this file.
comment:36

Replying to @jdemeyer:

Then a more serious remark: I doubt that you are using BASH_SOURCE correctly. From the bash man page:

       BASH_SOURCE
              An array variable whose members are the source filenames where the corresponding shell function names in the FUNCNAME array variable  are
              defined.  The shell function ${FUNCNAME[$i]} is defined in the file ${BASH_SOURCE[$i]} and called from ${BASH_SOURCE[$i+1]}.

At the very least, I need more explanation of why your code does the right thing.

BASH_SOURCE[0] gives the source of the "currently running function". See https://www.gnu.org/software/bash/manual/html_node/Bash-Variables.html (you need to read the entries for FUNCNAME and BASH_SOURCE).

comment:37

Replying to @mkoeppe:

Replying to @jdemeyer:

What does this comment mean:

## The first test does not actually work (see above).

("see above" can be really confusing because you don't know where to look.

It's referring to dirname $0 -- which does not give the location of sage-env but rather /bin/bash. This is because sage-env is sourced (.) rather than executed.

Don't tell me but tell the person reading sage-env (i.e. clarify the comment in the file).

I know, these comments are for the reviewer. I'll be happy remove them after we had a discussion about them... So do you have a comment on these?

Comments for the reviewer can go on Trac.

Branch pushed to git repo; I updated commit sha1. New commits:

c5998eaRemove TODO comment; reflected in #21775
bd8d2c7Remove TODO comment; reflected in #21524
ce48cf8src/bin/sage-env: Improve comments
c347768src/bin/sage-env: Remove broken "dirname $0" code

Changed commit from d82cfbc to c347768

comment:39

OK, done; please take a look

Branch pushed to git repo; I updated commit sha1. New commits:

4275121Installation manual: Document configure --prefix

Changed commit from c347768 to 4275121

Branch pushed to git repo; I updated commit sha1. New commits:

ea994ccMerge tag '7.5.beta1' into t/21479/__configure___prefix_sage_local

Changed commit from 4275121 to ea994cc

comment:42

ping?

comment:43

Running ./configure without arguments:

[...]
config.status: creating build/make/Makefile-auto
config.status: creating src/Makefile
config.status: creating src/bin/sage-env-config
config.status: executing depfiles commands
config.status: executing mkdirs commands
config.status: creating directory /usr/local/src/sage-config/logs/pkgs
config.status: creating directory NONE
config.status: creating directory NONE/bin
config.status: creating directory NONE/etc
config.status: creating directory NONE/include
config.status: creating directory NONE/lib
config.status: creating directory NONE/share
config.status: creating directory NONE/var/lib/sage/installed
config.status: creating symbolic link lib64 -> lib

Changed dependencies from #21501 to none

Reviewer: Jeroen Demeyer

comment:44

This is just because you write SAGE_LOCAL instead of "$SAGE_LOCAL".

When you fix this, I suggest to squash the commits to one commit and rebase to 7.5.beta1.

comment:45

The file sage-env-config needs to be ignored by .gitignore and deleted when running make distclean.

comment:46

This line in sage needs quoting:

. $SAGE_ROOT/src/bin/sage-env-config

And since you're running bash anyway, you might as well use source instead of . which is more visible.

comment:47

And do you really need this:

if [ $? -ne 0 ]; then
    echo >&2 "Error: $SAGE_ROOT/src/bin/sage-env-config could not be read"
fi

The shell will already write an error message, no need for an additional message.

comment:48

I also think that determining $SAGE_LOCAL should be done after

# If this is a freshly-unpacked binary tarball then run the installer
# Note: relocate-once.py deletes itself upon successful completion
if [ -x "$SAGE_ROOT/relocate-once.py" ]; then
    "$SAGE_ROOT/relocate-once.py"
fi
comment:49

Can you explain under which circumstances it is needed to read sage-env-config from sage-env? It seems to me that sage-env-config would already be read, either by the top-level sage script or by build/make/install.

comment:50

It is good style to start all error messages with Error: and write them to stderr. build/make/install doesn't do that.

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

4f832c2Add configure option --prefix=SAGE_LOCAL

Changed commit from ea994cc to 4f832c2

comment:52

Squashed and rebased on 7.5.beta1

Changed commit from 4f832c2 to 5fcda83

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

6308c3bAdd configure option --prefix=SAGE_LOCAL
ff48d5aFixup configure without --prefix
67baa7aSource sage-env-config after calling relocate-once.py
5fcda83top-level sage script: Remove redundant sage-env-config error message

Branch pushed to git repo; I updated commit sha1. New commits:

1d0aa34src/bin/sage-env-config: Prefix error message with 'Error:'
a0f90afmake misc-clenan: Delete src/bin/sage-env-config
e525547Add sage-env-config to .gitignore

Changed commit from 5fcda83 to e525547

Branch pushed to git repo; I updated commit sha1. New commits:

600639asage-env: Quoting fix

Changed commit from e525547 to 600639a

comment:56

Replying to @jdemeyer:

Can you explain under which circumstances it is needed to read sage-env-config from sage-env? It seems to me that sage-env-config would already be read, either by the top-level sage script or by build/make/install.

$SAGE_LOCAL/bin/sage becomes directly executable; one does not have to go through that top-level script.

comment:57

Replying to @jdemeyer:

since you're running bash anyway, you might as well use source instead of . which is more visible.

I'm not sure, I like . more than source.

comment:59

Replying to @mkoeppe:

$SAGE_LOCAL/bin/sage becomes directly executable; one does not have to go through that top-level script.

Then let me turn the question around: why do you need to change $SAGE_ROOT/sage?

comment:60

I'm still not entirely convinced that BASH_SOURCE[0] really does what we want. Why not use $SAGE_ROOT/src/bin/sage-env-config?

Or maybe we should just source both sage-env-config and sage-env from src/bin/sage?

comment:62

Replying to @jdemeyer:

Why not use $SAGE_ROOT/src/bin/sage-env-config?

Things installed in SAGE_LOCAL definitely should not depend on the source directory.

Or maybe we should just source both sage-env-config and sage-env from src/bin/sage?

That would be possible. There might be a few more places though that need to source the extra file.

comment:63

Replying to @jdemeyer:

Replying to @mkoeppe:

$SAGE_LOCAL/bin/sage becomes directly executable; one does not have to go through that top-level script.

Then let me turn the question around: why do you need to change $SAGE_ROOT/sage?

$SAGE_ROOT/sage sources $SAGE_ROOT/src/bin/sage-env-config so that it works when nothing has been installed so far.

comment:64

Replying to @mkoeppe:

$SAGE_ROOT/sage sources $SAGE_ROOT/src/bin/sage-env-config so that it works when nothing has been installed so far.

Sorry, I don't follow. What is the "it" in the sentence above? In other words, what would not work?

Changed commit from 600639a to a3467bf

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

f6c4513Add configure option --prefix=SAGE_LOCAL
7d44f0bFixup configure without --prefix
387ab94Source sage-env-config after calling relocate-once.py
fc66139top-level sage script: Remove redundant sage-env-config error message
fe19d3fsrc/bin/sage-env-config: Prefix error message with 'Error:'
269ad80make misc-clean: Delete src/bin/sage-env-config
727123aAdd sage-env-config to .gitignore
be4c390sage-env: Quoting fix
a3467bftop-level sage script: No need to source sage-env-config
comment:66

Replying to @jdemeyer:

Replying to @mkoeppe:

$SAGE_ROOT/sage sources $SAGE_ROOT/src/bin/sage-env-config so that it works when nothing has been installed so far.

Sorry, I don't follow. What is the "it" in the sentence above? In other words, what would not work?

I retract my comment.

I have removed this code from $SAGE_ROOT/sage.
(Also rebased on latest beta.)

comment:67

Replying to @mkoeppe:

Or maybe we should just source both sage-env-config and sage-env from src/bin/sage?

That would be possible. There might be a few more places though that need to source the extra file.

I think I wouldn't like to change this on this ticket. There are indeed a few places that source sage-env: build/bin/sage-spkg, build/make/deps, src/bin/sage-update-src. Probably not so nice if we have to change all of them.

comment:68

Replying to @mkoeppe:

$SAGE_LOCAL/bin/sage becomes directly executable; one does not have to go through that top-level script.

I realized that this is not entirely true. sage-env contains some code for discovering SAGE_ROOT when it is not set, but that code fails if $SAGE_LOCAL is not a direct subdirectory of $SAGE_ROOT.
I suppose I could add the definition of SAGE_ROOT to sage-env-config.
But that could go on a follow-up ticket as well.

comment:70

ping

Changed reviewer from Jeroen Demeyer to Jeroen Demeyer, Erik Bray

comment:71

Looks good to me at least, but I don't have a complete view of the implications. I also still think #21495 has merit, but that shouldn't hold this up, as it's a step in the right direction I think regardless of what anyone thinks about #21495 or if anything is done about it.

comment:73

Doesn't work if you dont have autotools

19make -j2 build/make/Makefile
20make[1]: Entering directory `/home/buildbot/slave/sage_git/build'
21make[1]: warning: -jN forced in submake: disabling jobserver mode.
22./bootstrap -d
23make[2]: Entering directory `/home/buildbot/slave/sage_git/build'
24rm -rf config configure build/make/Makefile-auto.in
25make[2]: Leaving directory `/home/buildbot/slave/sage_git/build'
26src/bin/sage-env: line 206: /home/buildbot/slave/sage_git/build/src/bin/sage-env-config: No such file or directory
27sage-env: Error: /home/buildbot/slave/sage_git/build/src/bin/sage-env-config could not be read.
28./bootstrap: line 31: aclocal: command not found
29Bootstrap failed, downloading required files instead.
30./bootstrap: line 61: sage-download-file: command not found
31Error: downloading configure-200.tar.gz failed
32make[1]: *** [configure] Error 1
33make[1]: Target `build/make/Makefile' not remade because of errors.
34make[1]: Leaving directory `/home/buildbot/slave/sage_git/build'
35make: *** [start] Error 2
36program finished with exit code 2

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

928f67dAdd configure option --prefix=SAGE_LOCAL
66a8066Fixup configure without --prefix
b5ecc81Source sage-env-config after calling relocate-once.py
c1c36fftop-level sage script: Remove redundant sage-env-config error message
ae6263bsrc/bin/sage-env-config: Prefix error message with 'Error:'
27189bemake misc-clean: Delete src/bin/sage-env-config
eb922b8Add sage-env-config to .gitignore
4cdaa43sage-env: Quoting fix
71f88cctop-level sage script: No need to source sage-env-config
52826b1bootstrap: Don't use sage-env, set PATH directly

Changed commit from a3467bf to 52826b1

comment:76

Replying to @vbraun:

Doesn't work if you dont have autotools

Thanks for catching this.
Fixed. Needs review.

comment:77

Erik, if you could take another look...

comment:78

I'm not really clear about what changed. I don't understand what the issue with autotools was or how this was fixed. I see that you did do something to address that but I don't understand the original issue in the first place--would be better if Volker commented since he pointed it out.

comment:79

Jeroen, could you take another look?

comment:80

This ticket needs review.

comment:81

pong

comment:82

Why does this require changes to bootstrap anyway?