Enable gitpod integration
Closed this issue · 194 comments
Gitpod allows one to setup a complete dev environment in the cloud. It is free to use for up to 50 hours per month. In this ticket the config necessary for making this work with sagemath is added.
You can try this out by going to https://gitpod.io/#https://github.com/sagemath/sagetrac-mirror/tree/public/build%2Fgitpod.
It currently takes a bit more than 1 hour until everything is setup. After this ticket is merged, we can enable the automatic prebuild using a github app (https://www.gitpod.io/docs/prebuilds#on-github). With this enabled, every push to the develop branch would trigger a prebuild of the complete environment (including the build of all dependencies and cythonizion) so that one has a up-to-date code env in a matter of a few seconds.
While setting things up, I noticed a few issues. Not sure if they are known problems or even by design. Please let me know if I should open a ticket for them to improve things.
- gitpod has brew installed on linux by default. Sagemath thinks this is the primary package manager and, e.g., configure suggests to run
brew xyzto install new packages. Is brew indeed preferred over apt-get? This is now #32753. - gitpod comes also with pyenv preinstalled. This breaks the build unless one sets
pyenv global system. Otherwise all python packages will be installed in the python env set by pyenv. This is now #29285. - Currently gitpod prebuild has a time limit of 1 hour (with the plan to increase it in the future). To improve our build time I propose to add a new
makeoption that uses prebuilt wheels from pypi instead of compiling everything locally. This is now #32754. - Maxima doesn't work with the homebrew version of ecl? From #29617 I got the impression it should work, but it fails with
;;; Loading "/workspace/sagetrac-mirror/local/var/tmp/sage/build/maxima-5.45.0.p0/src/src/../lisp-utils/defsystem.lisp"
;;; Loading #P"/home/linuxbrew/.linuxbrew/Cellar/ecl/21.2.1_1/lib/ecl-21.2.1/cmp.fas"
#P"/workspace/sagetrac-mirror/local/var/tmp/sage/build/maxima-5.45.0.p0/src/lisp-utils/defsystem.lisp"
>
("../src/" "./" (MAKE::HOME-SUBDIRECTORY "lisp/systems/")
"/usr/local/lisp/Registry/")
>
;;; Loading "/workspace/sagetrac-mirror/local/var/tmp/sage/build/maxima-5.45.0.p0/src/src/maxima.system"
; - Compiling defsystem "maxima"
; - Compiling module "package"
; - Compiling source file
; "/workspace/sagetrac-mirror/local/var/tmp/sage/build/maxima-5.45.0.p0/src/src/maxima-package.lisp"
;;;
;;; Compiling /workspace/sagetrac-mirror/local/var/tmp/sage/build/maxima-5.45.0.p0/src/src/maxima-package.lisp.
;;; OPTIMIZE levels: Safety=2, Space=0, Speed=3, Debug=2
;;;
;;; End of Pass 1.
;;; Internal error:
;;; ** Error code 1 when executing
;;; (EXT:RUN-PROGRAM "gcc-5" ("-I." "-I/home/linuxbrew/.linuxbrew/Cellar/ecl/21.2.1_1/include/" "-I/home/linuxbrew/.linuxbrew/opt/gmp/include" "-I/home/linuxbrew/.linuxbrew/opt/libffi/include" "-I/home/linuxbrew/.linuxbrew/opt/bdw-gc/include" "-D_GNU_SOURCE" "-D_FILE_OFFSET_BITS=64" "-g" "-O2" "-fPIC" "-D_THREAD_SAFE" "-Dlinux" "-O2" "-c" "binary-ecl/maxima-package.c" "-o" "binary-ecl/maxima-package.o")):
;;; exec: No such file or directory
; - Binary file binary-ecl/maxima-package.fas is old or does not exist.
; Compile (and load) source file /workspace/sagetrac-mirror/local/var/tmp/sage/build/maxima-5.45.0.p0/src/src/maxima-package.lisp instead? y
; - Should I bother you if this happens again? y
; - Compiling source file
; "/workspace/sagetrac-mirror/local/var/tmp/sage/build/maxima-5.45.0.p0/src/src/maxima-package.lisp"
;;;
;;; Compiling /workspace/sagetrac-mirror/local/var/tmp/sage/build/maxima-5.45.0.p0/src/src/maxima-package.lisp.
;;; OPTIMIZE levels: Safety=2, Space=0, Speed=3, Debug=2
;;;
;;; End of Pass 1.
;;; Internal error:
;;; ** Error code 1 when executing
;;; (EXT:RUN-PROGRAM "gcc-5" ("-I." "-I/home/linuxbrew/.linuxbrew/Cellar/ecl/21.2.1_1/include/" "-I/home/linuxbrew/.linuxbrew/opt/gmp/include" "-I/home/linuxbrew/.linuxbrew/opt/libffi/include" "-I/home/linuxbrew/.linuxbrew/opt/bdw-gc/include" "-D_GNU_SOURCE" "-D_FILE_OFFSET_BITS=64" "-g" "-O2" "-fPIC" "-D_THREAD_SAFE" "-Dlinux" "-O2" "-c" "binary-ecl/maxima-package.c" "-o" "binary-ecl/maxima-package.o")):
;;; exec: No such file or directory
; - Loading binary file "binary-ecl/maxima-package.fas"
Condition of type: FILE-ERROR
Filesystem error with pathname #P"/workspace/sagetrac-mirror/local/var/tmp/sage/build/maxima-5.45.0.p0/src/src/binary-ecl/maxima-package.fas".
This should be investigated as part of #29159.
Depends on #33068
Depends on #30933
CC: @mkoeppe @saraedum @koffie
Component: build
Branch/Commit: public/build/gitpod @ 974eb4c
Reviewer: Tobias Diez
Issue created by migration from https://trac.sagemath.org/ticket/32749
Description changed:
---
+++
@@ -8,6 +8,6 @@
---
While setting things up, I noticed a few issues. Not sure if they are known problems or even by design. Please let me know if I should open a ticket for them to improve things.
-- gitpod has brew installed on linux by default. Sagemath thinks this is the primary package manager and, e.g., configure suggests to run `brew xyz` to install new packages.
+- gitpod has brew installed on linux by default. Sagemath thinks this is the primary package manager and, e.g., configure suggests to run `brew xyz` to install new packages. Is brew indeed preferred over apt-get?
- gitpod comes also with pyenv preinstalled. This breaks the build unless one sets `pyenv global system`. Otherwise all python packages will be installed in the python env set by pyenv.
- Curretly gitpod prebuild has a time limit of 1 hour (with the plan to increase it in the future). To improve our build time I propose to add a new `make` option that uses prebuilt wheels from pypi instead of compiling everything locally. Description changed:
---
+++
@@ -10,4 +10,47 @@
While setting things up, I noticed a few issues. Not sure if they are known problems or even by design. Please let me know if I should open a ticket for them to improve things.
- gitpod has brew installed on linux by default. Sagemath thinks this is the primary package manager and, e.g., configure suggests to run `brew xyz` to install new packages. Is brew indeed preferred over apt-get?
- gitpod comes also with pyenv preinstalled. This breaks the build unless one sets `pyenv global system`. Otherwise all python packages will be installed in the python env set by pyenv.
-- Curretly gitpod prebuild has a time limit of 1 hour (with the plan to increase it in the future). To improve our build time I propose to add a new `make` option that uses prebuilt wheels from pypi instead of compiling everything locally.
+- Curretly gitpod prebuild has a time limit of 1 hour (with the plan to increase it in the future). To improve our build time I propose to add a new `make` option that uses prebuilt wheels from pypi instead of compiling everything locally.
+- Maxima doesn't work with the homebrew version of ecl? From #29617 I got the impression it should work, but it fails with
+
+```
+;;; Loading "/workspace/sagetrac-mirror/local/var/tmp/sage/build/maxima-5.45.0.p0/src/src/../lisp-utils/defsystem.lisp"
+;;; Loading #P"/home/linuxbrew/.linuxbrew/Cellar/ecl/21.2.1_1/lib/ecl-21.2.1/cmp.fas"
+#P"/workspace/sagetrac-mirror/local/var/tmp/sage/build/maxima-5.45.0.p0/src/lisp-utils/defsystem.lisp"
+>
+("../src/" "./" (MAKE::HOME-SUBDIRECTORY "lisp/systems/")
+ "/usr/local/lisp/Registry/")
+>
+;;; Loading "/workspace/sagetrac-mirror/local/var/tmp/sage/build/maxima-5.45.0.p0/src/src/maxima.system"
+
+; - Compiling defsystem "maxima"
+; - Compiling module "package"
+; - Compiling source file
+; "/workspace/sagetrac-mirror/local/var/tmp/sage/build/maxima-5.45.0.p0/src/src/maxima-package.lisp"
+;;;
+;;; Compiling /workspace/sagetrac-mirror/local/var/tmp/sage/build/maxima-5.45.0.p0/src/src/maxima-package.lisp.
+;;; OPTIMIZE levels: Safety=2, Space=0, Speed=3, Debug=2
+;;;
+;;; End of Pass 1.
+;;; Internal error:
+;;; ** Error code 1 when executing
+;;; (EXT:RUN-PROGRAM "gcc-5" ("-I." "-I/home/linuxbrew/.linuxbrew/Cellar/ecl/21.2.1_1/include/" "-I/home/linuxbrew/.linuxbrew/opt/gmp/include" "-I/home/linuxbrew/.linuxbrew/opt/libffi/include" "-I/home/linuxbrew/.linuxbrew/opt/bdw-gc/include" "-D_GNU_SOURCE" "-D_FILE_OFFSET_BITS=64" "-g" "-O2" "-fPIC" "-D_THREAD_SAFE" "-Dlinux" "-O2" "-c" "binary-ecl/maxima-package.c" "-o" "binary-ecl/maxima-package.o")):
+;;; exec: No such file or directory
+; - Binary file binary-ecl/maxima-package.fas is old or does not exist.
+; Compile (and load) source file /workspace/sagetrac-mirror/local/var/tmp/sage/build/maxima-5.45.0.p0/src/src/maxima-package.lisp instead? y
+; - Should I bother you if this happens again? y
+; - Compiling source file
+; "/workspace/sagetrac-mirror/local/var/tmp/sage/build/maxima-5.45.0.p0/src/src/maxima-package.lisp"
+;;;
+;;; Compiling /workspace/sagetrac-mirror/local/var/tmp/sage/build/maxima-5.45.0.p0/src/src/maxima-package.lisp.
+;;; OPTIMIZE levels: Safety=2, Space=0, Speed=3, Debug=2
+;;;
+;;; End of Pass 1.
+;;; Internal error:
+;;; ** Error code 1 when executing
+;;; (EXT:RUN-PROGRAM "gcc-5" ("-I." "-I/home/linuxbrew/.linuxbrew/Cellar/ecl/21.2.1_1/include/" "-I/home/linuxbrew/.linuxbrew/opt/gmp/include" "-I/home/linuxbrew/.linuxbrew/opt/libffi/include" "-I/home/linuxbrew/.linuxbrew/opt/bdw-gc/include" "-D_GNU_SOURCE" "-D_FILE_OFFSET_BITS=64" "-g" "-O2" "-fPIC" "-D_THREAD_SAFE" "-Dlinux" "-O2" "-c" "binary-ecl/maxima-package.c" "-o" "binary-ecl/maxima-package.o")):
+;;; exec: No such file or directory
+; - Loading binary file "binary-ecl/maxima-package.fas"
+Condition of type: FILE-ERROR
+Filesystem error with pathname #P"/workspace/sagetrac-mirror/local/var/tmp/sage/build/maxima-5.45.0.p0/src/src/binary-ecl/maxima-package.fas".
+```The problem with ecl is clearly due to a broken ecl package in linuxbrew - it tries to use a nonexisting compiler.
Probably build/bin/sage-guess-package-system should be changed so that brew is only tested after the Linux package managers.
Branch pushed to git repo; I updated commit sha1. New commits:
1ae507d | Remove uunnecessary changes to sage-venv |
Changed work issues from Cleanup config, add vs extensions and add docs to none
Thanks for the feedback. I've now removed the linuxbrew install of ecl and opened tickets for the other things. So this is now ready for review.
Description changed:
---
+++
@@ -8,9 +8,9 @@
---
While setting things up, I noticed a few issues. Not sure if they are known problems or even by design. Please let me know if I should open a ticket for them to improve things.
-- gitpod has brew installed on linux by default. Sagemath thinks this is the primary package manager and, e.g., configure suggests to run `brew xyz` to install new packages. Is brew indeed preferred over apt-get?
+- gitpod has brew installed on linux by default. Sagemath thinks this is the primary package manager and, e.g., configure suggests to run `brew xyz` to install new packages. Is brew indeed preferred over apt-get? This is now #32753.
- gitpod comes also with pyenv preinstalled. This breaks the build unless one sets `pyenv global system`. Otherwise all python packages will be installed in the python env set by pyenv.
-- Curretly gitpod prebuild has a time limit of 1 hour (with the plan to increase it in the future). To improve our build time I propose to add a new `make` option that uses prebuilt wheels from pypi instead of compiling everything locally.
+- Currently gitpod prebuild has a time limit of 1 hour (with the plan to increase it in the future). To improve our build time I propose to add a new `make` option that uses prebuilt wheels from pypi instead of compiling everything locally. This is now #32754.
- Maxima doesn't work with the homebrew version of ecl? From #29617 I got the impression it should work, but it fails with
```
@@ -54,3 +54,4 @@
Condition of type: FILE-ERROR
Filesystem error with pathname #P"/workspace/sagetrac-mirror/local/var/tmp/sage/build/maxima-5.45.0.p0/src/src/binary-ecl/maxima-package.fas".
```
+This should be investigated as part of #29159.Does it work?
Consider opening a bug report against the ecl packaging on linuxbrew
Replying to @mkoeppe:
Does it work?
Yes! You can also test this out using the above link. Currently one has to wait for the build to finish. This waiting time can be removed by activating the github integration after this ticket is merged.
Replying to @mkoeppe:
Consider opening a bug report against the ecl packaging on linuxbrew
Will do after this ticket is merged (since I can then reference the gitpod workspace as a reproducible example).
Description changed:
---
+++
@@ -9,7 +9,7 @@
While setting things up, I noticed a few issues. Not sure if they are known problems or even by design. Please let me know if I should open a ticket for them to improve things.
- gitpod has brew installed on linux by default. Sagemath thinks this is the primary package manager and, e.g., configure suggests to run `brew xyz` to install new packages. Is brew indeed preferred over apt-get? This is now #32753.
-- gitpod comes also with pyenv preinstalled. This breaks the build unless one sets `pyenv global system`. Otherwise all python packages will be installed in the python env set by pyenv.
+- gitpod comes also with pyenv preinstalled. This breaks the build unless one sets `pyenv global system`. Otherwise all python packages will be installed in the python env set by pyenv. This is now #29285.
- Currently gitpod prebuild has a time limit of 1 hour (with the plan to increase it in the future). To improve our build time I propose to add a new `make` option that uses prebuilt wheels from pypi instead of compiling everything locally. This is now #32754.
- Maxima doesn't work with the homebrew version of ecl? From #29617 I got the impression it should work, but it fails with
It is asking: "We noticed a new virtual environment has been created. Do you want to select it for the workspace folder?"
Add the correct answer to the documentation?
Maybe you want to use make V=0 so that there's less output during build?
Does this need Sage's Jupyter notebook or does it bring its own?
configure --disable-notebook can be used to avoid building notebook and its many Python deps
Given that the preinstalled linux is old and the linuxbrew has some broken packages, one could wonder whether it would be better to use conda to set up a richer environment with much shorter time for building.
The scipy people seem to be doing that - https://scipy.github.io/devdocs/dev/contributor/quickstart_gitpod.html - but I haven't looked at any details.
The build on Gitpod finished for me, but then the workspace timed out; on opening it again, starting ./sage fails with
ImportError: libzmq.so.5: cannot open shared object file: No such file or directory.
Am I doing something wrong?
Replying to @mkoeppe:
It is asking: "We noticed a new virtual environment has been created. Do you want to select it for the workspace folder?"
Add the correct answer to the documentation?
This will be no longer necessary once we have added the VS config.
Replying to @mkoeppe:
Maybe you want to use
make V=0so that there's less output during build?
Normally the user wouldn't see this, and everything that is printed during the init stage would become the prebuilt log that can be viewed in the admin interface. So I think its fine to have it more detailed, in case something goes wrong.
Replying to @mkoeppe:
Given that the preinstalled linux is old and the linuxbrew has some broken packages, one could wonder whether it would be better to use
condato set up a richer environment with much shorter time for building.
The scipy people seem to be doing that - https://scipy.github.io/devdocs/dev/contributor/quickstart_gitpod.html - but I haven't looked at any details.
That might indeed something to consider. I would keep it for now using plain ubuntu to get some experience how it goes with the prebuilt. We can change it to conda later if necessary.
Replying to @mkoeppe:
The build on Gitpod finished for me, but then the workspace timed out; on opening it again, starting
./sagefails with
ImportError: libzmq.so.5: cannot open shared object file: No such file or directory.
Am I doing something wrong?
I can confirm this.
I also sometimes had other problems with re-started workspaces. For example, when the build was interrupted then next time make tried to install packages (e.g. python) that were using system packages during the first run and thus didn't needed to run. I'm not sure ifs a bug with gitpod (i.e. some important files are not correctly restored) or an issue with sage's build system. Do you have an idea what's going wrong?
Replying to @mkoeppe:
Does this need Sage's Jupyter notebook or does it bring its own?
configure --disable-notebookcan be used to avoid building notebook and its many Python deps
If you want to use VS codes Jupyter notebook integration, jupyter needs to be installed in the venv used.
Branch pushed to git repo; I updated commit sha1. New commits:
a973734 | Move system setup to dockerfile |
Branch pushed to git repo; I updated commit sha1. New commits:
5d3e6da | Disable brew prio |
Replying to @tobiasdiez:
Replying to @mkoeppe:
The build on Gitpod finished for me, but then the workspace timed out; on opening it again, starting
./sagefails with
ImportError: libzmq.so.5: cannot open shared object file: No such file or directory.
Am I doing something wrong?I can confirm this.
I also sometimes had other problems with re-started workspaces. For example, when the build was interrupted then next time
maketried to install packages (e.g. python) that were using system packages during the first run and thus didn't needed to run. I'm not sure ifs a bug with gitpod (i.e. some important files are not correctly restored) or an issue with sage's build system. Do you have an idea what's going wrong?
That was in fact my mistake. Only stuff under the workspace directory is restored by gitpod. Thus, the apt-get installed packages got lost. I've now moved the system setup into a dockerfile which should fix this.
As a follow-up to reduce the build duration, is it possible to build certain libraries (say make ecl), store the result somewhere in the docker image and then on make build reuse the previously built ecl?
Sorry that I haven't had the time to look at the new version in the past weeks. If this is ready for testing by developers, I'd suggest that you post on sage-devel to invite people to try it out.
No worries!
It's working for me and ready for review. However, it currently needs to build everything on each start of the workspace. Thus, that's not very friendly for other devs to try out. But after it is merged and the Gitpod github action is activated, the workspaces are prebuild and thus instantly ready. That's then a better point to advertise it on the developer mailing list.
(Caveat: Currently gitpod only allows prebuilds of up to 1h; I'm not sure if we manage this)
Branch pushed to git repo; I updated commit sha1. New commits:
51c69e3 | No need to intstall git again |
Branch pushed to git repo; I updated commit sha1. New commits:
7d64908 | Merge branch 'develop' of https://github.com/sagemath/sage into public/build/gitpod |
do you know if gitpod allows sharing these prebuilds across accounts?
Normally, the prebuilds are build centrally by a github application that notifies gitpod about changes to certain branches (currently only the default branch). These prebuilds are then used by everyone that starts a new environment based on these branches.
But you do can share a workspace, see https://www.gitpod.io/docs/sharing-and-collaboration. Thus, say you update a dependency in a workspace and this leads to errors, then you can make a snapshot and share it with other developers so that can investigate the issue.
Branch pushed to git repo; I updated commit sha1. New commits:
8cf8b38 | Prebuild non-python packages in docker |
Branch pushed to git repo; I updated commit sha1. New commits:
1aa9500 | Fix log output |
Branch pushed to git repo; I updated commit sha1. New commits:
8c83420 | Reduce image build time |
Branch pushed to git repo; I updated commit sha1. New commits:
ea6c0a3 | Prebuild a few more libs |
Branch pushed to git repo; I updated commit sha1. New commits:
40a07dd | Install pari instead of gap, and set env correctly |
Branch pushed to git repo; I updated commit sha1. New commits:
2ec7921 | Fix syntax |
Branch pushed to git repo; I updated commit sha1. New commits:
00394d2 | Don't need to install sudo |
Branch pushed to git repo; I updated commit sha1. New commits:
e43768f | Try again without pari |
Branch pushed to git repo; I updated commit sha1. New commits:
962ab18 | Try disabling most of the docker build... |
Branch pushed to git repo; I updated commit sha1. New commits:
22d0462 | Reenable env config |
Branch pushed to git repo; I updated commit sha1. New commits:
743e8fc | Reenable everything |
Branch pushed to git repo; I updated commit sha1. New commits:
12cdb6a | More bisecting... |
Branch pushed to git repo; I updated commit sha1. New commits:
23a210c | Another one... |
Branch pushed to git repo; I updated commit sha1. New commits:
5eeae07 | Reactivate everything? |
Branch pushed to git repo; I updated commit sha1. New commits:
8f578ef | Try to ignore workspace folder |
Branch pushed to git repo; I updated commit sha1. New commits:
c0ce19e | Copy only what we need |
I've moved the compilation of most non-python packages to the docker file (as they are relatively constant) to minimize the time one has to wait until the gitpod workspace is completely initialized.
This could be much simplified by just configuring sage-prebuild and the actual build to use the same prefix (configure --prefix=...).
-1 on hardcoding (again) the list of Debian packages. We have infrastructure for that - use it
Overall I think it's too slow to fit well in the "ephemeral workspace" philosophy of gitpod
I'd suggest to integrate with the Docker image stuff described in docker/README.md - a lot of work has been done there to make incremental builds fast.
Replying to @mkoeppe:
This could be much simplified by just configuring
sage-prebuildand the actual build to use the same prefix (configure --prefix=...).
Could you please expand on this idea. Since the current code already use the prefix idea:
RUN ./configure --prefix=/home/gitpod/sage-prebuild
Note also that the final build (in gitpod.yml) should preferably use a normal build (without a special prefix) to be as close as possible to a "normal" dev setup.
Replying to @mkoeppe:
-1 on hardcoding (again) the list of Debian packages. We have infrastructure for that - use it
Okay, but I couldn't figure out how to use this in a Docker file that needs to be static (and not be generated). What do you propose to do?
Replying to @mkoeppe:
I'd suggest to integrate with the Docker image stuff described in
docker/README.md- a lot of work has been done there to make incremental builds fast.
The dockerbuild here is however not meant to be incremental. It's build once and then is reused as a basis for every workspace. It also needs to run below 1h as this is current timeout of gitpod.
Replying to @mkoeppe:
Overall I think it's too slow to fit well in the "ephemeral workspace" philosophy of gitpod
It's only slow currently as the whole build of sage is triggered everytime a workspace is started. Once this ticket is merged, one could activate gitpods 'prebuild' feature that does this in the background automatically, so that starting the workspace is a matter of seconds (which are mainly spent to load the custom docker image + the prebuild files). See the ticket description.
Replying to @tobiasdiez:
Replying to @mkoeppe:
This could be much simplified by just configuring
sage-prebuildand the actual build to use the same prefix (configure --prefix=...).Could you please expand on this idea. Since the current code already use the
prefixidea:RUN ./configure --prefix=/home/gitpod/sage-prebuild
Change this to ./configure --prefix=/home/gitpod/sage-local
and change ./configure -enable-editable in .gitpod.yml to ./configure --prefix=/home/gitpod/sage-local --enable-editable
Note also that the final build (in gitpod.yml) should preferably use a normal build (without a special prefix) to be as close as possible to a "normal" dev setup.
Using --prefix is normal - it's step 4 of our install instructions in README.md
Replying to @tobiasdiez:
Replying to @mkoeppe:
I'd suggest to integrate with the Docker image stuff described in
docker/README.md- a lot of work has been done there to make incremental builds fast.The dockerbuild here is however not meant to be incremental. It's build once and then is reused as a basis for every workspace. It also needs to run below 1h as this is current timeout of gitpod.
Is there a particular reason why you cannot use the existing Docker build https://hub.docker.com/r/sagemath/sagemath/tags
Replying to @mkoeppe:
Replying to @tobiasdiez:
Replying to @mkoeppe:
This could be much simplified by just configuring
sage-prebuildand the actual build to use the same prefix (configure --prefix=...).Could you please expand on this idea. Since the current code already use the
prefixidea:RUN ./configure --prefix=/home/gitpod/sage-prebuild
Change this to
./configure --prefix=/home/gitpod/sage-localand change
./configure -enable-editablein.gitpod.ymlto./configure --prefix=/home/gitpod/sage-local --enable-editable
That's a good suggestion. Sadly gitpod only preserves changes to the workspace folder during prebuilds, so the the make in .gitpod.yml needs to write everything in the local workspace folder and not in some global/system folder. https://www.gitpod.io/docs/prebuilds#workspace-directory-only
Replying to @mkoeppe:
Replying to @tobiasdiez:
Replying to @mkoeppe:
I'd suggest to integrate with the Docker image stuff described in
docker/README.md- a lot of work has been done there to make incremental builds fast.The dockerbuild here is however not meant to be incremental. It's build once and then is reused as a basis for every workspace. It also needs to run below 1h as this is current timeout of gitpod.
Is there a particular reason why you cannot use the existing Docker build https://hub.docker.com/r/sagemath/sagemath/tags
First of all, the end goal is different: the gitpod docker serves as a developer enviroment, and the existing docker config is a means to distribute sage. Concretely, gitpod has a hard timeout of 1h, so everything is optimized to fit into this timeframe (for example, only a handful of non-python packages are built). Also the current docker seems to use a very minimal installation of system packages, while the gitpod image tries to install all relevant system packages; the base image is also different. It seems to be also hard to reuse an existing docker file with gitpod, since one only can specify a dockerfile and not the exact docker build command (in particular not the specific target / build stage). But in the end, there is also not much code duplication between the docker images, so I don't think thats a big issue.
Replying to @tobiasdiez:
Sadly gitpod only preserves changes to the workspace folder during prebuilds, so the the
makein.gitpod.ymlneeds to write everything in the local workspace folder and not in some global/system folder.
Is all of /home/gitpod in the workspace?
Replying to @tobiasdiez:
Replying to @mkoeppe:
Overall I think it's too slow to fit well in the "ephemeral workspace" philosophy of gitpod
It's only slow currently as the whole build of sage is triggered everytime a workspace is started. Once this ticket is merged, one could activate gitpods 'prebuild' feature [...]
Why does this have to wait for the ticket being merged? Can it not be done from an arbitrary branch?
Replying to @mkoeppe:
Replying to @tobiasdiez:
Sadly gitpod only preserves changes to the workspace folder during prebuilds, so the the
makein.gitpod.ymlneeds to write everything in the local workspace folder and not in some global/system folder.Is all of
/home/gitpodin the workspace?
The "workspace" is in /workspace/sagetrac-mirror, and only changes to this folder are preserved in prebuilds. On the other hand, the docker image can change system folders such as /home/gitpod and those changes are accessible during the prebuild. I couldn't find out a way to change something in /workspace using the docker image (gitpod also handles the git checkout for you etc).
Replying to @mkoeppe:
Replying to @tobiasdiez:
Replying to @mkoeppe:
Overall I think it's too slow to fit well in the "ephemeral workspace" philosophy of gitpod
It's only slow currently as the whole build of sage is triggered everytime a workspace is started. Once this ticket is merged, one could activate gitpods 'prebuild' feature [...]
Why does this have to wait for the ticket being merged? Can it not be done from an arbitrary branch?
What needs to be done is to activate the gitpod app for github on the org level. This could be done also before this ticket is merged, but I'm not sure if it would pick up the gitpod config or if that one needs to be present in the default branch.
The prebuild can be configured to run for certain branches, see https://www.gitpod.io/docs/prebuilds#github-specific-configuration. Currently, it uses the default config which enables prebuilds only for the default branch (= develop).
Would https://www.gitpod.io/docs/prebuilds#manual-prebuilds work, or does this also depend on the org-level activation?
Branch pushed to git repo; I updated commit sha1. New commits:
eab7958 | docker/.gitpod.Dockerfile: Use sage-get-system-packages |