Make docker images from tox/GitHub CI workflow, regular Sage Docker images, gitpod, CoCalc interoperable

Question

Make docker images from tox/GitHub CI workflow, regular Sage Docker images, gitpod, CoCalc interoperable

mkoeppe opened this issue 6 years ago · 29 comments

mkoeppe commented 6 years ago

Adjust directory names to something that works for both cases ...
- docker/Dockerfile currently has:
  - SAGE_ROOT=/home/sage/sage
  - SAGE_LOCAL=$SAGE_ROOT/local
- Docker images built with tox/GH Actions (including Gitpod) and also computop/sage have:
  - SAGE_ROOT=/sage
  - SAGE_LOCAL=/sage/local
- After this ticket:
  - SAGE_ROOT=/sage
  - SAGE_LOCAL=/opt/sage
- CoCalc Docker image has:
  - SAGE_ROOT=/usr/local/sage
  - SAGE_LOCAL=/usr/local/sage/local
#33851: Define a script package representing "useful stuff to have on a Docker image"
Replace all apt-get in docker/Dockerfile by using a Docker image built using build/bin/write-dockerfile as the base image.
By default, docker/Dockerfile would use configure --without-system-... a lot so that CI is done against "the reference distribution"
Move build/bin/write-dockerfile.sh into docker/
Add a mechanism (environment variable) to tox -e docker... to copy SAGE_LOCAL from a given Docker image.

CC: @kliem @dimpase @embray @saraedum @slel @fchapoton @tobiasdiez @williamstein @culler @NathanDunfield

Component: docker

Keywords: ci, ContinuousIntegration, sd109

Issue created by migration from https://trac.sagemath.org/ticket/29536

Answer 1 · 2020-04-21T02:44:56.000Z

Description changed:

--- 
+++ 
@@ -1,3 +1,7 @@
 ... by adjusting directory names to something that works for both cases ...
 
+`docker/Dockerfile`:
+- `SAGE_ROOT=/home/sage/sage`
+- `SAGE_LOCAL=$SAGE_ROOT/local`
 
+

Answer 2 · 2020-04-26T08:35:29.000Z

comment:2

by the way, we don't seem to have in docs an explanation regarding file shared between docker images and regular file system of the host system.

Answer 3 · 2020-05-27T12:03:08.000Z

Changed keywords from ci, ContinuousIntegration to ci, ContinuousIntegration, sd109

Answer 4 · 2020-05-28T00:25:49.000Z

Description changed:

--- 
+++ 
@@ -1,7 +1,12 @@
-... by adjusting directory names to something that works for both cases ...
+1. adjust directory names to something that works for both cases ...
 
-`docker/Dockerfile`:
-- `SAGE_ROOT=/home/sage/sage`
-- `SAGE_LOCAL=$SAGE_ROOT/local`
+   `docker/Dockerfile`:
+   - `SAGE_ROOT=/home/sage/sage`
+   - `SAGE_LOCAL=$SAGE_ROOT/local`
 
+2. Define a script package representing "useful stuff to have on a Docker image" (more than the `_recommended` package from #29557)
 
+3. Replace all `apt-get` in `docker/Dockerfile` by using a Docker image built using `build/bin/write-dockerfile` as the base image.
+
+4. By default, `docker/Dockerfile` would use `configure --without-system-...` a lot so that CI is done against "the reference distribution"
+

Answer 5 · 2020-05-28T14:57:39.000Z

comment:5

I'm confused--should this say "GitHub" in the title or "GitLab"? I don't know of any GitHub CI docker images.

Answer 6 · 2020-05-28T15:26:47.000Z

comment:6

Replying to @embray:

I'm confused--should this say "GitHub" in the title or "GitLab"? I don't know of any GitHub CI docker images.

please see
https://doc.sagemath.org/html/en/developer/index.html#testing-on-multiple-platforms

we now have a builder of docker images for multiple platforms, configurable, and by default hosted on github.

Answer 7 · 2020-05-29T17:48:16.000Z

Description changed:

--- 
+++ 
@@ -1,8 +1,8 @@
 1. adjust directory names to something that works for both cases ...
 
-   `docker/Dockerfile`:
-   - `SAGE_ROOT=/home/sage/sage`
-   - `SAGE_LOCAL=$SAGE_ROOT/local`
+   `docker/Dockerfile` currently has:
+    - `SAGE_ROOT=/home/sage/sage`
+    - `SAGE_LOCAL=$SAGE_ROOT/local`
 
 2. Define a script package representing "useful stuff to have on a Docker image" (more than the `_recommended` package from #29557)

Answer 8 · 2020-09-04T18:01:17.000Z

Description changed:

--- 
+++ 
@@ -10,3 +10,7 @@
 
 4. By default, `docker/Dockerfile` would use `configure --without-system-...` a lot so that CI is done against "the reference distribution"
 
+
+5. Move `build/bin/write-dockerfile` into `docker/`
+
+

Answer 9 · 2020-11-25T20:00:44.000Z

Dependencies: #30960

Answer 10 · 2021-03-24T02:04:25.000Z

comment:14

Sage development has entered the release candidate phase for 9.3. Setting a new milestone for this ticket based on a cursory review of ticket status, priority, and last modification date.

Answer 11 · 2021-03-28T18:57:47.000Z

Description changed:

--- 
+++ 
@@ -13,4 +13,6 @@
 
 5. Move `build/bin/write-dockerfile` into `docker/`
 
+6. Add a mechanism (environment variable) to `tox -e docker...` to copy SAGE_LOCAL from a given Docker image.
 
+

Answer 12 · 2021-10-01T08:06:26.000Z

comment:17

Given that GitLab Docker build is broken for ages, can we arrange for GitHub-built docker images to be pushed to the main Docker repo,
or at least create some sort of advertised Docker repo on GitHub?

Answer 13 · 2021-12-27T21:44:42.000Z

Description changed:

--- 
+++ 
@@ -16,3 +16,5 @@
 6. Add a mechanism (environment variable) to `tox -e docker...` to copy SAGE_LOCAL from a given Docker image.
 
 
+
+Gitpod - see #32749

Answer 14 · 2022-05-11T21:44:06.000Z

comment:22

Bikeshedding time! Let's pick SAGE_ROOT, SAGE_LOCAL. My proposal is in the ticket description.

(The only important constraint for me is: Don't nest SAGE_LOCAL within SAGE_ROOT - for flexibility such as bind-mounting a new version of the source tree over SAGE_ROOT without hiding SAGE_LOCAL.)

Answer 15 · 2022-05-11T21:44:06.000Z

Description changed:

--- 
+++ 
@@ -1,20 +1,24 @@
-1. adjust directory names to something that works for both cases ...
-
-   `docker/Dockerfile` currently has:
-    - `SAGE_ROOT=/home/sage/sage`
-    - `SAGE_LOCAL=$SAGE_ROOT/local`
+1. Adjust directory names to something that works for both cases ...
+   - `docker/Dockerfile` currently has:
+      - `SAGE_ROOT=/home/sage/sage`
+      - `SAGE_LOCAL=$SAGE_ROOT/local`
+   - Docker images built with tox/GH Actions (including Gitpod) have:
+      - `SAGE_ROOT=/sage`
+      - `SAGE_LOCAL=/sage/local`
+   - **After this ticket:**
+      - `SAGE_ROOT=/sage`
+      - `SAGE_LOCAL=/opt/sage`
 
 2. Define a script package representing "useful stuff to have on a Docker image" (more than the `_recommended` package from #29557)
+   - perhaps call it `_devtools`? 
 
 3. Replace all `apt-get` in `docker/Dockerfile` by using a Docker image built using `build/bin/write-dockerfile` as the base image.
 
 4. By default, `docker/Dockerfile` would use `configure --without-system-...` a lot so that CI is done against "the reference distribution"
 
 
-5. Move `build/bin/write-dockerfile` into `docker/`
+5. Move `build/bin/write-dockerfile.sh` into `docker/`
 
 6. Add a mechanism (environment variable) to `tox -e docker...` to copy SAGE_LOCAL from a given Docker image.
 
 
-
-Gitpod - see #32749

Answer 16 · 2022-05-11T21:46:49.000Z

Changed dependencies from #30960 to #33740

Answer 17 · 2022-05-11T21:46:49.000Z

Description changed:

--- 
+++ 
@@ -11,6 +11,7 @@
 
 2. Define a script package representing "useful stuff to have on a Docker image" (more than the `_recommended` package from #29557)
    - perhaps call it `_devtools`? 
+   - include the stuff added in #33740 to `environment-dev.yml`
 
 3. Replace all `apt-get` in `docker/Dockerfile` by using a Docker image built using `build/bin/write-dockerfile` as the base image.

Answer 18 · 2022-05-11T21:49:51.000Z

Description changed:

--- 
+++ 
@@ -12,6 +12,7 @@
 2. Define a script package representing "useful stuff to have on a Docker image" (more than the `_recommended` package from #29557)
    - perhaps call it `_devtools`? 
    - include the stuff added in #33740 to `environment-dev.yml`
+   - solve #27677, #29153
 
 3. Replace all `apt-get` in `docker/Dockerfile` by using a Docker image built using `build/bin/write-dockerfile` as the base image.

Answer 19 · 2022-05-14T17:51:42.000Z

Description changed:

--- 
+++ 
@@ -11,7 +11,7 @@
 
 2. Define a script package representing "useful stuff to have on a Docker image" (more than the `_recommended` package from #29557)
    - perhaps call it `_devtools`? 
-   - include the stuff added in #33740 to `environment-dev.yml`
+   - include the stuff added in #33740 to `environment-dev.yml` and #33671
    - solve #27677, #29153
 
 3. Replace all `apt-get` in `docker/Dockerfile` by using a Docker image built using `build/bin/write-dockerfile` as the base image.

Answer 20 · 2022-05-14T17:53:00.000Z

Description changed:

--- 
+++ 
@@ -10,7 +10,7 @@
       - `SAGE_LOCAL=/opt/sage`
 
 2. Define a script package representing "useful stuff to have on a Docker image" (more than the `_recommended` package from #29557)
-   - perhaps call it `_devtools`? 
+   - perhaps call it `_develop`? 
    - include the stuff added in #33740 to `environment-dev.yml` and #33671
    - solve #27677, #29153

Answer 21 · 2022-05-14T18:05:03.000Z

Description changed:

--- 
+++ 
@@ -9,10 +9,7 @@
       - `SAGE_ROOT=/sage`
       - `SAGE_LOCAL=/opt/sage`
 
-2. Define a script package representing "useful stuff to have on a Docker image" (more than the `_recommended` package from #29557)
-   - perhaps call it `_develop`? 
-   - include the stuff added in #33740 to `environment-dev.yml` and #33671
-   - solve #27677, #29153
+2. #33851: Define a script package representing "useful stuff to have on a Docker image" 
 
 3. Replace all `apt-get` in `docker/Dockerfile` by using a Docker image built using `build/bin/write-dockerfile` as the base image.

Answer 22 · 2022-05-14T18:05:43.000Z

Branch: u/mkoeppe/make_docker_images_from_tox_github_ci_workflow__regular_sage_docker_images__and_gitpod_interoperable

Answer 23 · 2022-05-14T18:20:34.000Z

Commit: 58e112a

Answer 24 · 2022-05-14T18:20:34.000Z

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`10338a7`	`Change ci as well`
`0cf7334`	`Add sage-setup as install-requires dep`
`6d2b149`	`Use relative paths to not install src package`
`1dba3ba`	`Maybe thats the correct shebang?`
`c1b2fbf`	`Revert install_requires change`
`ad56ba3`	`Go back to two pip installs for now`
`b42f5f5`	`Readd how to use specific python version`
`87f360c`	`Readd name arg`
`01eb374`	`Merge #33740`
`58e112a`	`bootstrap-conda: Use script package _develop instead of hardcoded list of dev tools; remove unused RECOMMENDED`

Answer 25 · 2022-05-14T20:19:11.000Z

Changed commit from 58e112a to none

Answer 26 · 2022-05-14T20:19:11.000Z

Changed dependencies from #33740 to none

Answer 27 · 2022-05-14T20:19:11.000Z

Changed branch from u/mkoeppe/make_docker_images_from_tox_github_ci_workflow__regular_sage_docker_images__and_gitpod_interoperable to none

Answer 28 · 2022-05-17T21:44:28.000Z

Description changed:

--- 
+++ 
@@ -8,6 +8,9 @@
    - **After this ticket:**
       - `SAGE_ROOT=/sage`
       - `SAGE_LOCAL=/opt/sage`
+   - CoCalc Docker image has:
+      - `SAGE_ROOT=/usr/local/sage`
+      - `SAGE_LOCAL=/usr/local/sage/local`
 
 2. #33851: Define a script package representing "useful stuff to have on a Docker image"

Answer 29 · 2022-05-17T23:05:30.000Z

Description changed:

--- 
+++ 
@@ -2,7 +2,7 @@
    - `docker/Dockerfile` currently has:
       - `SAGE_ROOT=/home/sage/sage`
       - `SAGE_LOCAL=$SAGE_ROOT/local`
-   - Docker images built with tox/GH Actions (including Gitpod) have:
+   - Docker images built with tox/GH Actions (including Gitpod) and also `computop/sage` have:
       - `SAGE_ROOT=/sage`
       - `SAGE_LOCAL=/sage/local`
    - **After this ticket:**