gitpod integration using Docker images from portability testing workflow
mkoeppe opened this issue · 199 comments
Gitpod allows one to setup a complete dev environment in the cloud. It is free to use for up to 50 hours per month. In this ticket the config necessary for making this work with sagemath is added.
You can try this out by going to https://gitpod.io/#https://github.com/mkoeppe/sage/tree/mkoeppe-9.5.beta9%2Bupdates.
It works by building docker images on github, push them to GH packages (https://github.com/mkoeppe?tab=packages&q=gitpod) and then use these ones as the base for the gitpod workspace.
After this ticket is merged, we should activate the automatic prebuild by installing the github app (https://www.gitpod.io/docs/prebuilds#on-github). With this enabled, every push would trigger a prebuild of the complete environment (including the build of all dependencies and cythonizion) so that one has a up-to-date code env very quickly without the need to build anything.
Follow-up: #33113 gitpod usability improvements
Alternative: #32749, which only builds the docker image on gitpod (which is currently not really feasible as they have a time limit of 1h)
Depends on #30933
Depends on #29285
CC: @tobiasdiez @dimpase @saraedum
Component: docker
Author: Tobias Diez, Matthias Koeppe
Branch/Commit: 078f247
Reviewer: Matthias Koeppe, Dima Pasechnik
Issue created by migration from https://trac.sagemath.org/ticket/33103
Last 10 new commits:
e99a09e | sed -i.bak 's/ubunty/ubuntu/g' .github/workflows/*.yml |
edb4364 | .github/workflows/tox.yml: Replace homebrew-macos-python3_xcode-standard by homebrew-macos-usrlocal-python3_xcode-standard |
e27e161 | Merge #32703 |
96a45d1 | tox.ini, .github/workflows: Remove ubuntu-groovy |
59ec3d0 | Merge #33068 |
cbf03d1 | .github/workflows/tox.yml: Add docker-gitpod |
4a9edfc | build/bin/write-dockerfile.sh: Handle __SUDO |
d697d94 | build/bin/write-dockerfile.sh: Use relative workdir 'sage' |
c7cc5cd | .gitpod.yml, docker/.gitpod.Dockerfile: Reverse SAGE_LOCAL symlink |
bde9f74 | build/bin/write-dockerfile.sh: Handle $__CHOWN |
Author: Tobias Diez, Matthias Koeppe
Description changed:
---
+++
@@ -1 +1,5 @@
Based on #32749.
+
+Docker images are pushed to GH Packages - https://github.com/mkoeppe?tab=packages&q=gitpod - and can be used to initialize gitpod
+
+Branch pushed to git repo; I updated commit sha1. New commits:
7ef6a6e | docker/.gitpod.Dockerfile: Remove, use custom image from GH Packages instead |
Description changed:
---
+++
@@ -2,4 +2,5 @@
Docker images are pushed to GH Packages - https://github.com/mkoeppe?tab=packages&q=gitpod - and can be used to initialize gitpod
-
+To try out (after the image has been built):
+- https://gitpod.io/#https://github.com/mkoeppe/sage/tree/t/33103/gitpod_integration_using_docker_images_from_portability_testing_workflowDescription changed:
---
+++
@@ -2,5 +2,5 @@
Docker images are pushed to GH Packages - https://github.com/mkoeppe?tab=packages&q=gitpod - and can be used to initialize gitpod
-To try out (after the image has been built):
+To try out (after the image has been built, https://github.com/mkoeppe/sage/runs/4680839663?check_suite_focus=true):
- https://gitpod.io/#https://github.com/mkoeppe/sage/tree/t/33103/gitpod_integration_using_docker_images_from_portability_testing_workflowBranch pushed to git repo; I updated commit sha1. New commits:
6b41c4a | .github/workflows/tox.yml: Also push ...-with-targets-pre |
Running into pyenv issues (#29285) here too:
[pytest] installing. Log file: /home/gitpod/sage/logs/pkgs/pytest.log
[pytest] error installing, exit status 1. End of log file:
[pytest] /home/gitpod/sage-local/var/lib/sage/venv-python3.8/bin/python3: No module named pip
[pytest] Full log file: /home/gitpod/sage/logs/pkgs/pytest.log
make[4]: *** [Makefile:2756: pytest-no-deps] Error 1
Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
aeca1a0 | Makefile (pypi-sdists): Add sage_setup |
cd3dce9 | pkgs/sage-conf_pypi/setup.py: Update directory of configured sage_conf.py |
c695921 | pkgs/sage-conf/sage_conf.py.in (SAGE_SPKG_WHEELS): Update so it works if SAGE_VENV != SAGE_LOCAL |
cebcda3 | Merge tag '9.5.beta4' into t/29039/pip_installable_package_sage_bootstrap |
0026892 | Merge tag '9.5.beta5' into t/29039/pip_installable_package_sage_bootstrap |
2a4a323 | Merge tag '9.5.beta7' into t/29039/pip_installable_package_sage_bootstrap |
02234ce | Merge #29039 |
b1413d4 | build/bin/write-dockerfile.sh: Move 'ADD pkgs' earlier |
859be3e | build/bin/write-dockerfile.sh: ADD src/*.m4 files for bootstrapping |
af7c8d3 | Merge #33067 |
Description changed:
---
+++
@@ -2,5 +2,5 @@
Docker images are pushed to GH Packages - https://github.com/mkoeppe?tab=packages&q=gitpod - and can be used to initialize gitpod
-To try out (after the image has been built, https://github.com/mkoeppe/sage/runs/4680839663?check_suite_focus=true):
+To try out (after the image has been built, https://github.com/mkoeppe/sage/runs/4682227249?check_suite_focus=true):
- https://gitpod.io/#https://github.com/mkoeppe/sage/tree/t/33103/gitpod_integration_using_docker_images_from_portability_testing_workflowBranch pushed to git repo; I updated commit sha1. New commits:
fad8942 | build/bin/write-dockerfile.sh: Use relative workdir 'sage' (again) |
Description changed:
---
+++
@@ -3,4 +3,4 @@
Docker images are pushed to GH Packages - https://github.com/mkoeppe?tab=packages&q=gitpod - and can be used to initialize gitpod
To try out (after the image has been built, https://github.com/mkoeppe/sage/runs/4682227249?check_suite_focus=true):
-- https://gitpod.io/#https://github.com/mkoeppe/sage/tree/t/33103/gitpod_integration_using_docker_images_from_portability_testing_workflow
+- https://gitpod.io/#https://github.com/mkoeppe/sage/tree/mkoeppe-9.5.beta9%2BupdatesThis works. To be done:
- Additional tags for the images, for example
latestfor releases, or branch/tag name (${GITHUB_REF_NAME}- https://docs.github.com/en/actions/learn-github-actions/environment-variables#default-environment-variables) - suitable for.gitpod.yml - Could use
docker/.gitpod.Dockerfileagain withCOPY --fromto make the final image smaller (currently /home/gitpod/sage has the old source tree)
Description changed:
---
+++
@@ -2,5 +2,5 @@
Docker images are pushed to GH Packages - https://github.com/mkoeppe?tab=packages&q=gitpod - and can be used to initialize gitpod
-To try out (after the image has been built, https://github.com/mkoeppe/sage/runs/4682227249?check_suite_focus=true):
+To try out:
- https://gitpod.io/#https://github.com/mkoeppe/sage/tree/mkoeppe-9.5.beta9%2BupdatesBranch pushed to git repo; I updated commit sha1. New commits:
fdb49a1 | tox.ini, .github/workflows/tox*.yml: Also tag the Docker images with GITHUB_REF_NAME |
Branch pushed to git repo; I updated commit sha1. New commits:
ab757ba | .github/workflows/tox.yml: Fix passing of EXTRA_DOCKER_TAGS |
Branch pushed to git repo; I updated commit sha1. New commits:
96528eb | .github/workflows/tox*.yml: Tag docker images 'latest' or 'dev' when building a release tag |
Changed reviewer from https://github.com/mkoeppe/sage/runs/4686167146 to https://github.com/mkoeppe/sage/runs/4686610379
Changed reviewer from https://github.com/mkoeppe/sage/runs/4686610379 to https://github.com/mkoeppe/sage/runs/4687073955
Changed reviewer from https://github.com/mkoeppe/sage/runs/4687073955 to https://github.com/mkoeppe/sage/runs/4687310789
Changed reviewer from https://github.com/mkoeppe/sage/runs/4687310789 to https://github.com/mkoeppe/sage/runs/4687310728
Branch pushed to git repo; I updated commit sha1. New commits:
5d1dc0f | .gitpod.yml: Use tag :dev |
Work Issues: Change .gitpod.yml to use sagemath instead of mkoeppe
Also the configuration of VS Code should be improved. It seems to start up using the wrong interpreter (not venv/bin/python3), and also one needs to manually select the right Jupyter kernel.
The scipy folks have some good documentation and configuration (https://scipy.github.io/devdocs/dev/contributor/quickstart_gitpod.html), perhaps we can lift something from there.
Is the prebuild working for this branch?
How can one work with branches that update sage packages? With #32749 one could trigger a rebuild of the docker image (and thus of the system packages) using the #imagebuild context. Can something like this also be implemented here? For example, take the github action build docker as base, and then re-install system or python packages on top of this if they have newer versions?
For the VS integration, #30677 is closed but not yet merged into develop.
Replying to @tobiasdiez:
Is the prebuild working for this branch?
I've just merged #33071 into the branch https://github.com/mkoeppe/sage/tree/mkoeppe-9.5.beta9%2Bupdates to try this out.
How can one work with branches that update sage packages?
Nothing special, make just rebuilds what needs rebuilding.
Replying to @tobiasdiez:
For the VS integration, #30677 is closed but not yet merged into develop.
I forgot to say that the branch https://github.com/mkoeppe/sage/tree/mkoeppe-9.5.beta9%2Bupdates already contains #30677 and other updates
Replying to @mkoeppe:
Replying to @tobiasdiez:
Is the prebuild working for this branch?
I've just merged #33071 into the branch https://github.com/mkoeppe/sage/tree/mkoeppe-9.5.beta9%2Bupdates to try this out.
How can one work with branches that update sage packages?
Nothing special,
makejust rebuilds what needs rebuilding.
So this would then be done as part of the prebuild?
Yes, I think so
Looks like I'll have to add some configuration so that it does prebuilds on all branches
I created a "project" and now it looks like the prebuild triggered by a push is running - https://gitpod.io/projects/sage
Branch pushed to git repo; I updated commit sha1. New commits:
3e28e13 | .gitpod.yml: Set github prebuild options to true |
Before the push I had merged #32968 with the Sphinx update; and the prebuild has just installed the upgraded package
Branch pushed to git repo; I updated commit sha1. New commits:
49575f5 | .gitpod.yml: Switch to image ghcr.io/sagemath/sage/sage-docker-gitpod-standard-with-targets:dev |
Changed work issues from Change .gitpod.yml to use sagemath instead of mkoeppe to none
Description changed:
---
+++
@@ -4,3 +4,6 @@
To try out:
- https://gitpod.io/#https://github.com/mkoeppe/sage/tree/mkoeppe-9.5.beta9%2Bupdates
+
+Follow-up: #33113 gitpod usability improvements
+The prebuild was just killed because of the 35m timeout.
What's the best choice of timeout according to your experiments?
If its really the timeout that killed the prebuild (in which the prebuild would be green and be used for the next workspace, but maybe doesn't contain all packages) then you can go to 50min. Maybe a bit less since the docker image is quite big so this part may take some time. But essentially you have to calculate 60min - bootstrap and config (2-3min) - docker init and copying (?? min).
However, for me the prebuild often timed out after 1h (then its shown red in the project view). The only solution I've found for this is not using symlinks into /home/.
https://gitpod.io/projects/sage/e8780c03-b8b8-4810-a772-d4cc2994b6bf shows
[sagelib-9.5.beta9] installing. Log file: /workspace/sage/logs/pkgs/sagelib-9.5.beta9.log
[pytest] successfully installed.
[sphinx-4.3.1] successfully installed.
make[1]: *** [Makefile:39: all-build] Terminated
make: *** [Makefile:16: build] Terminated
exit
but it is still claims "RUNNING: Prebuild in progress"
However, https://gitpod.io/projects/sage/prebuilds shows READY.
🤙 This task ran as a workspace prebuild
🎉 Well done on saving 37 minutes
A few more points:
- please use the recent version of gitpod.yml from https://github.com/tobiasdiez/sagetrac-mirror/blob/develop/.gitpod.yml (the ssh stuff and num threads = 8)
--prefix=$HOME/sage-localin gitpod.yml should probably be/workspace/sage-localor a few lines above$HOMEshould be used?- probably 8 parallel tasks are fine. gitpod gives 6 (or 7?) vCPUs to users.
Replying to @mkoeppe:
🤙 This task ran as a workspace prebuild 🎉 Well done on saving 37 minutes
Nice, then you can go to 50min indeed.
Branch pushed to git repo; I updated commit sha1. New commits:
e7d4e96 | .gitpod.yml: Increase timeout for make in prebuild, increase parallelism |
Presumably the before script should try to resume the build in case the prebuild timed out.
I agree. But init is called after before during the prebuild, and I couldn't find a way to detect if one is in a prebuild or not...
Branch pushed to git repo; I updated commit sha1. New commits:
fdb332d | .gitpod.yml: Run 'make' again in 'before' task if prebuild 'make' timed out |
Branch pushed to git repo; I updated commit sha1. New commits:
aaa9777 | .gitpod.yml: Use SAGE_NUM_THREADS=8 |
Replying to @tobiasdiez:
- please use the recent version of gitpod.yml from https://github.com/tobiasdiez/sagetrac-mirror/blob/develop/.gitpod.yml (the ssh stuff and num threads = 8)
I've updated it - the ssh stuff was already same
Branch pushed to git repo; I updated commit sha1. New commits:
87900a7 | .gitpod.yml: Refer to /home/gitpod as /Users/mkoeppe |
Branch pushed to git repo; I updated commit sha1. New commits:
0864c6e | .github/workflows/tox*.yml: Push stable releases also to Docker tag 'dev' |
https://gitpod.io/projects/sage/134676a9-2a0e-429a-816d-68ad30441f58
[sphinx-4.3.1] installing. Log file: /workspace/sage/logs/pkgs/sphinx-4.3.1.log
[sagelib-9.5.beta9] installing. Log file: /workspace/sage/logs/pkgs/sagelib-9.5.beta9.log
[pytest] successfully installed.
[sphinx-4.3.1] successfully installed.
[sagelib-9.5.beta9] successfully installed.
make --no-print-directory sage_docbuild-SAGE_VENV-no-deps
[sage_docbuild-9.5.beta9] installing. Log file: /workspace/sage/logs/pkgs/sage_docbuild-9.5.beta9.log
[sage_docbuild-9.5.beta9] successfully installed.
make[2]: Leaving directory '/workspace/sage/build/make'
real 31m8.684s
user 101m44.392s
sys 3m47.093s
SAGE_CHECK=warn, so scanning the log files. This may take a few seconds.
Sage build/upgrade complete!
make[1]: Leaving directory '/workspace/sage'
exit
🤙 This task ran as a workspace prebuild
🎉 Well done on saving 34 minutes
Replying to @sagetrac-git:
Branch pushed to git repo; I updated commit sha1. New commits:
fdb332d | .gitpod.yml: Run 'make' again in 'before' task if prebuild 'make' timed out |
This worked but for some reason the Sphinx package installation was repeated in this step. It's fast but I'll try if refreshing the timestamps of the install records will avoid it.
There are some weird synchronization issues on the server side - it seems that if one creates a workspace right after a prebuild has completed, it can happen that it does not use the prebuild.
Branch pushed to git repo; I updated commit sha1. New commits:
38dafc6 | .gitpod.yml (before): Refer to /workspace/sage/local by full path |
Branch pushed to git repo; I updated commit sha1. New commits:
e6be067 | .gitpod.yml (init): Make sure to remove the directory /home/gitpod/sage-local |
Branch pushed to git repo; I updated commit sha1. New commits:
cdcf2ac | .gitpod.yml (before): establish SAGE_LOCAL here, not in 'init' |
Branch pushed to git repo; I updated commit sha1. New commits:
aef75eb | .gitpod.yml (before): Always remove the source tree used by the Docker build |
OK, now it's in good shape.
Could you check that the SSH stuff works as intended and matches the added documentation?
I see
$ ls -l ~/.ssh
total 24
-rw-r--r-- 1 gitpod gitpod 381 Jan 4 08:53 authorized_keys
-rw------- 1 gitpod gitpod 0 Jan 4 08:53 id_rsa
-rw-r--r-- 1 gitpod gitpod 664 Jan 4 08:53 known_hosts
-rw-r--r-- 1 gitpod gitpod 15544 Jan 4 08:53 supervisor_env
and am not sure what the empty id_rsa is intended for
There you don't need/want the timeout.
I'm not sure if I understand the logic behind this and related commands (maybe add a bit more documentation of what's going on?). But please be aware that the code may be in a different location such as /workspace/sagetrac-mirror if someone starts gitpod from https://github.com/sagemath/sagetrac-mirror (which is probably what's going to happen if you want to check someone elses branch)
Replying to @mkoeppe:
Could you check that the SSH stuff works as intended and matches the added documentation?
and am not sure what the emptyid_rsais intended for
Did you added a PRIVATE_SSH_KEY env variable in gitpod with your trac ssh as described in the docs? It's working for me.
Replying to @mkoeppe:
OK, now it's in good shape.
For me it takes over 5min to start a workspace due to the large workspace image (it displays "fetching build image" or something like this in the loading screen). I think this should be optimized as part of this ticket since otherwise I fear people will not adopt gitpod for e.g. reviewing of tickets.
Also it still loads pyenv: pyenv shell 3.8.12 so https://github.com/tobiasdiez/sagetrac-mirror/blob/20ca179bed4ff605de4111362b02c8e2acca617e/.gitpod.yml#L18-L19 should probably be readded.
Is the somewhat strange indention by design?