Discover SAGE_SCRIPTS_DIR to make $SAGE_LOCAL/bin/sage work from any directory, in an environment without SAGE_* variables

Question

Discover SAGE_SCRIPTS_DIR to make $SAGE_LOCAL/bin/sage work from any directory, in an environment without SAGE_* variables

mkoeppe opened this issue 7 years ago · 48 comments

This ticket makes $SAGE_LOCAL/bin/sage work without environment variables set, or the current working directory to be something particular.

It does that by

adding SAGE_ROOT to $SAGE_LOCAL/bin/sage-env-config (via src/bin/sage-env-config.in).
adding code to the top of sage-env that determines SAGE_SCRIPTS_DIR from the directory of itself.

Previous discussion in https://groups.google.com/d/msg/sage-devel/oODph9-yjaI/FVeBsCk_CgAJ:

On Wednesday, May 30, 2018 at 9:01:59 AM UTC-7, Nils Bruin wrote:

Currently, if I run the script $SAGE_LOCAL/bin/sage in my normal environment (where SAGE_ROOT is not set) then I get:

Error: You must set the SAGE_ROOT environment variable or run this
script from the SAGE_ROOT or SAGE_ROOT/local/bin/ directory.
Error setting environment variables by sourcing '/usr/local/sage/sage-git/local/bin/sage-env';
possibly contact sage-devel (see http://groups.google.com/group/sage-devel).

If instead I run the script $SAGE_ROOT/sage a value for SAGE_ROOT is figured out and everything works
fine when called from my normal environment.

CC: @jdemeyer @nbruin @embray @kiwifb @EmmanuelCharpentier @tobihan @saraedum @slel @timokau @dimpase

Component: build

Keywords: SAGE_LOCAL, SAGE_ROOT

Author: Matthias Koeppe

Branch: 4415da5

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/25486

Answer 1 · 2018-06-01T21:12:16.000Z

comment:1

Likewise, a naive trial to start the GAP shipped by Sage fails:

$ /path/to/sagedir/local/bin/gap
Set the environment variable SAGE_LOCAL.

and one instead has to run:

$ /path/to/sagedir/sage --gap

or

$ SAGE_LOCAL='/path/to/sagedir' /path/to/sagedir/local/bin/gap

(where /path/to/sagedir should be replaced by the actual path to
the Sage installation directory).

This makes it hard to use
Gap.app, the macOS
interface to GAP by Russ Woodroofe, with the GAP shipped by Sage.

Answer 2 · 2018-06-01T21:12:16.000Z

Changed keywords from none to SAGE_LOCAL, SAGE_ROOT

Answer 3 · 2018-06-06T10:04:14.000Z

comment:2

One comment: the new version of GAP deprecates the gap.sh as the right way to start GAP. (The gap binary detects the directory that it lives in, and uses that as the default library directory.) Relying on the shell script to check for SAGE_LOCAL probably isn't a great idea for GAP 4.9.1 or later. That might make this more pressing.

If you do need to check for the SAGE_LOCAL variable, it seems to me like a better place to check would be in GAP code. (The IO_environ() function from the IO package will read the environment, which you could do on package load.) But I'm coming from outside, and might not be aware of all issues.

Also, I'm the Gap.app author. If it still looks necessary when I next make updates, I'll consider adding logic to set SAGE_LOCAL to a sensible guess on startup. I ran out of time for this release (as I wanted to get something out at the same time as GAP 4.9.1). I'd love for it to no longer be necessary!

Answer 4 · 2018-07-10T14:32:09.000Z

comment:3

I don't think there should be anything in GAP that cares about $SAGE_LOCAL. If anything Sage's bin/gap should just be a wrapper script that sets the necessary environment variables, rather than patching gap.sh which it currently does =_=

Answer 5 · 2020-01-15T23:30:50.000Z

comment:5

A step into this direction: #29022 "Create module src/sage/env_config.py from src/sage/env_config.py.in, defining variables for use in sage.env.

Answer 6 · 2020-01-15T23:30:59.000Z

Dependencies: #29022

Answer 7 · 2020-01-17T20:21:24.000Z

Branch: u/mkoeppe/_sage_local_bin_sage_should_work_in_an_environment_without_sage___variables

Answer 8 · 2020-01-17T20:27:32.000Z

Description changed:

--- 
+++ 
@@ -1,4 +1,11 @@
-From https://groups.google.com/d/msg/sage-devel/oODph9-yjaI/FVeBsCk_CgAJ:
+This ticket makes `$SAGE_LOCAL/bin/sage` work without environment variables set, or the current working directory to be something particular. 
+
+It does that by 
+- adding `SAGE_ROOT` to  `$SAGE_LOCAL/bin/sage-env-config` (via `src/bin/sage-env-config.in`).
+- adding code to the top of `sage-env` that determines `SAGE_SCRIPTS_DIR` from the directory of itself.
+
+---
+Previous discussion in https://groups.google.com/d/msg/sage-devel/oODph9-yjaI/FVeBsCk_CgAJ:
 
 On Wednesday, May 30, 2018 at 9:01:59 AM UTC-7, Nils Bruin wrote:
 
@@ -12,10 +19,6 @@
  If instead I run the script $SAGE_ROOT/sage a value for SAGE_ROOT is figured out and everything works 
  fine when called from my normal environment.
 
-We should make `$SAGE_LOCAL/bin/sage` work without environment variables set.
-We discussed this in #21479 (`./configure --prefix=SAGE_LOCAL`), when `$SAGE_LOCAL/bin/sage-env-config` was introduced; but then a less ambitious solution was implemented for that ticket.
-
-We could add a line setting `$SAGE_ROOT` in `$SAGE_LOCAL/bin/sage-env-config` (via `src/bin/sage-env-config.in`).

Answer 9 · 2020-01-17T20:27:32.000Z

comment:9

This is adapted from an old branch of #21479 (./configure --prefix=SAGE_LOCAL), when $SAGE_LOCAL/bin/sage-env-config was introduced; but then a less ambitious solution was implemented for that ticket.

New commits:

`a53e0df`	`src/bin/sage-env-config.in: Define SAGE_ROOT`
`c7b1090`	`src/bin/sage-env: Obtain SAGE_SCRIPTS_DIR from sage-env invocation`

Answer 10 · 2020-01-17T20:27:32.000Z

Author: Matthias Koeppe

Answer 11 · 2020-01-17T20:27:32.000Z

Commit: c7b1090

Answer 12 · 2020-01-18T19:12:24.000Z

comment:10

Removed a dependency that is not needed.

Answer 13 · 2020-01-18T19:12:24.000Z

Changed dependencies from #29022 to none

Answer 14 · 2020-01-18T19:12:24.000Z

Description changed:

--- 
+++ 
@@ -3,6 +3,8 @@
 It does that by 
 - adding `SAGE_ROOT` to  `$SAGE_LOCAL/bin/sage-env-config` (via `src/bin/sage-env-config.in`).
 - adding code to the top of `sage-env` that determines `SAGE_SCRIPTS_DIR` from the directory of itself.
+
+
 
 ---
 Previous discussion in https://groups.google.com/d/msg/sage-devel/oODph9-yjaI/FVeBsCk_CgAJ:

Answer 15 · 2020-01-22T11:42:15.000Z

comment:12

A few suggestions:

rename SOURCED_SAGE_ENV_CONFIG to SAGE_ENV_CONFIG_SOURCED (we already have a SAGE_ENV_SOURCED) variable as well.

Make sage-env-config itself responsible for setting SAGE_ENV_CONFIG_SOURCED, rather than setting it in sage-env.

+# SAGE_ROOT is the location of the Sage source/build tree.
+export SAGE_ROOT="@abs_top_srcdir@"

Wouldn't this permanently encode the directory in which Sage happens to be built in any Sage installation, including for binary builds? E.g. if Volker does a binary build and publishes it, then it will contain in local/bin/sage-env-config something like SAGE_ROOT=/home/vbraun/src/sage/ and won't work when moved. So I'm not sure this make sense to me...

How do distros currently handle $SAGE_ROOT, a path that doesn't have significant meaning in standard installs? Perhaps what we need is to do away with any dependency on $SAGE_ROOT at runtime.

Answer 16 · 2020-01-22T11:42:57.000Z

comment:13

I think otherwise in principle I'm +1 to this but I don't see how it resolves that last question...

Answer 17 · 2020-01-22T15:57:15.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`37c68fb`	`Rename SOURCED_SAGE_ENV_CONFIG -> SAGE_ENV_CONFIG_SOURCED, set it in src/bin/sage-env-config.in`

Answer 18 · 2020-01-22T15:57:15.000Z

Changed commit from c7b1090 to 37c68fb

Answer 19 · 2020-01-22T15:57:39.000Z

comment:15

Replying to @embray:

A few suggestions:

rename SOURCED_SAGE_ENV_CONFIG to SAGE_ENV_CONFIG_SOURCED (we already have a SAGE_ENV_SOURCED) variable as well.

Make sage-env-config itself responsible for setting SAGE_ENV_CONFIG_SOURCED, rather than setting it in sage-env.

Thanks! Done.

Answer 20 · 2020-01-22T16:15:09.000Z

comment:16

Replying to @embray:

+# SAGE_ROOT is the location of the Sage source/build tree.
+export SAGE_ROOT="@abs_top_srcdir@"
Wouldn't this permanently encode the directory in which Sage happens to be built in any Sage installation,

Yes, that's what it does. Sage installations (SAGE_LOCAL) and non-distclean source trees (SAGE_ROOT) are not relocatable with or without this ticket.

including for binary builds? E.g. if Volker does a binary build and publishes it, then it will contain in local/bin/sage-env-config something like SAGE_ROOT=/home/vbraun/src/sage/ and won't work when moved.

The binary build uses SAGE_ROOT = a long pseudorandom path and then rewrites all paths, including this one, using https://github.com/sagemath/binary-pkg/blob/master/binary_pkg/templates/relocate-once.py the first time that SAGE_ROOT/sage is used.

How do distros currently handle $SAGE_ROOT, a path that doesn't have significant meaning in standard installs?

Distributions tend to replace all of sage-env by their own tooling anyway.

Perhaps what we need is to do away with any dependency on $SAGE_ROOT at runtime.

Yes, in fact, there's very little left (see #25828/#28815). In any case, that's orthogonal to the present ticket.

Answer 21 · 2020-02-01T14:08:47.000Z

Changed commit from 37c68fb to 6382db9

Answer 22 · 2020-02-01T14:08:47.000Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`4184f49`	`src/bin/sage-env-config.in: Define SAGE_ROOT`
`8cbeafa`	`src/bin/sage-env: Obtain SAGE_SCRIPTS_DIR from sage-env invocation`
`6382db9`	`Rename SOURCED_SAGE_ENV_CONFIG -> SAGE_ENV_CONFIG_SOURCED, set it in src/bin/sage-env-config.in`

Answer 23 · 2020-02-01T14:09:02.000Z

comment:19

Rebased on 9.1.beta2, needs review

Answer 24 · 2020-02-16T20:29:22.000Z

Changed commit from 6382db9 to b6c35a7

Answer 25 · 2020-02-16T20:29:22.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`b6c35a7`	`Merge tag '9.1.beta4' into t/25486/_sage_local_bin_sage_should_work_in_an_environment_without_sage___variables`

Answer 26 · 2020-03-02T01:44:10.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`b4277ec`	`Merge tag '9.1.beta6' into t/25486/_sage_local_bin_sage_should_work_in_an_environment_without_sage___variables`

Answer 27 · 2020-03-02T01:44:10.000Z

Changed commit from b6c35a7 to b4277ec

Answer 28 · 2020-03-09T21:26:07.000Z

Changed commit from b4277ec to c6879b5

Answer 29 · 2020-03-09T21:26:07.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`c6879b5`	`Merge tag '9.1.beta7' into t/25486/_sage_local_bin_sage_should_work_in_an_environment_without_sage___variables`

Answer 30 · 2020-03-19T13:46:24.000Z

Changed commit from c6879b5 to 3671eb3

Answer 31 · 2020-03-19T13:46:24.000Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`afd05b3`	`src/bin/sage-env-config.in: Define SAGE_ROOT`
`50d2e09`	`src/bin/sage-env: Obtain SAGE_SCRIPTS_DIR from sage-env invocation`
`3671eb3`	`Rename SOURCED_SAGE_ENV_CONFIG -> SAGE_ENV_CONFIG_SOURCED, set it in src/bin/sage-env-config.in`

Answer 32 · 2020-03-19T13:46:41.000Z

comment:24

Rebased on 9.1.beta8, needs review

Answer 33 · 2020-04-14T06:38:06.000Z

comment:25

pushing these forward to 9.2

Answer 34 · 2020-06-07T21:56:56.000Z

Changed commit from 3671eb3 to 4415da5

Answer 35 · 2020-06-07T21:56:56.000Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`37d79ac`	`src/bin/sage-env-config.in: Define SAGE_ROOT`
`68c1762`	`src/bin/sage-env: Obtain SAGE_SCRIPTS_DIR from sage-env invocation`
`4415da5`	`Rename SOURCED_SAGE_ENV_CONFIG -> SAGE_ENV_CONFIG_SOURCED, set it in src/bin/sage-env-config.in`

Answer 36 · 2020-06-07T21:57:18.000Z

comment:27

Rebased on 9.2.beta0

Answer 37 · 2020-06-11T12:57:41.000Z

comment:28

lgtm

Answer 38 · 2020-06-11T12:57:41.000Z

Reviewer: Dima Pasechnik

Answer 39 · 2020-06-11T13:01:40.000Z

comment:29

oops, sorry, doesn't it need a shebang to make sure it is running in bash?

Answer 40 · 2020-06-12T11:08:39.000Z

comment:30

Well, I guess this ticket conflicts somehow with #29345

Answer 41 · 2020-06-24T05:12:38.000Z

comment:31

Actually it is already documented that sage-env "uses bash features, so it cannot be used with other shells." It is sourced by other scripts, so it does not need a #! line.

Answer 42 · 2020-06-25T23:27:41.000Z

comment:32

lgtm

Answer 43 · 2020-06-26T00:22:35.000Z

comment:33

Thanks!

Answer 44 · 2020-06-27T09:26:50.000Z

Changed branch from u/mkoeppe/_sage_local_bin_sage_should_work_in_an_environment_without_sage___variables to 4415da5

Answer 45 · 2020-07-05T19:47:36.000Z

comment:35

Follow up: #29446

Answer 46 · 2020-07-05T19:47:36.000Z

Changed commit from 4415da5 to none

Answer 47 · 2020-07-13T04:04:52.000Z

comment:36

Replying to @mkoeppe:

Actually it is already documented that sage-env "uses bash features, so it cannot be used with other shells." It is sourced by other scripts, so it does not need a #! line.

I fixed all of the bashisms in #29345, but missed that comment. The sage build now fails again with any other /bin/sh:

cd '/home/mjo/src/sage.git/build/pkgs/sage_conf' && . '/home/mjo/src/sage.git/src/bin/sage-env' && . '/home/mjo/src/sage.git/build/bin/sage-build-env-config' && sage-logger -p '/home/mjo/src/sage.git/build/pkgs/sage_conf/spkg-install' '/home/mjo/src/sage.git/logs/pkgs/sage_conf-none.log'
/bin/sh: 123: /home/mjo/src/sage.git/src/bin/sage-env: Bad substitution
Error: SAGE_SCRIPTS_DIR is set to a bad value:
SAGE_SCRIPTS_DIR=/home/mjo/src/sage.git/build/pkgs/sage_conf
You must correct it or erase it and rerun this script
make[3]: *** [Makefile:2022: /home/mjo/src/sage.git/local/var/lib/sage/installed/sage_conf-none] Error 1

Debian's dash shell is crippled, but would it be possible to symlink /bin/sh to dash on e.g. Arch linux, or maybe to ksh on some other distro that we have CI for? (To prevent regressions in the future?)

Answer 48 · 2020-07-13T04:12:03.000Z

comment:37

Follow-up on #30128