tox.ini, GH Actions: Add test for Sage distribution on conda using environment.yml
mkoeppe opened this issue · 38 comments
We create tox environments local-conda-forge-environment[-optional].
In contrast to ...-standard and ...-minimal, they use the files $SAGE_ROOT/environment*.yml (generated by bootstrap) to set up the conda environment.
To test locally:
tox -e local-conda-forge-environment
tox -e local-conda-forge-environment-optional
We also add them to the GH Actions workflow to ensure that the files environment*.yml are generated correctly.
See also:
- #30845, which adds tests for the files
SAGE_ROOT/src/environment*.yml
CC: @tobiasdiez @isuruf @kliem
Component: porting
Author: Matthias Koeppe
Branch/Commit: b998c84
Reviewer: Dima Pasechnik
Issue created by migration from https://trac.sagemath.org/ticket/31099
Can't continue at the moment because miniforge3 on macOS seems broken at the moment
Miniforge3 will now be installed into this location:
/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/local-conda-forge-environment_yml/conda
- Press ENTER to confirm the location
- Press CTRL-C to abort the installation
- Or specify a different location below
[/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/local-conda-forge-environment_yml/conda] >>>
PREFIX=/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/.tox/local-conda-forge-environment_yml/conda
Unpacking payload ...
[y/N]:
No matter what I answer to this promptless question, it aborts
Last 10 new commits:
559dd8e | tox.ini (local-root): Pass --enable-build-as-root to configure |
a85f41c | tox.ini (local): Do not build the toolchain when posargs = config.status or posargs = configure |
1de912a | tox.ini (local-sudo): Run apt-get update with sudo |
3c7e5c4 | tox.ini (local-root, local-sudo): Run output of sage-print-system-package-command through eval |
571cc49 | Merge branch 't/30947/src_doc_bootstrap__simplify_by_using_new_options_of__sage__package_list_' into t/29124/script-packages-prereq-toolchain-bootstrap |
f7ff30c | Merge branch 't/29124/script-packages-prereq-toolchain-bootstrap' into t/30944/tox__improve_local_sudo_ubuntu_standard |
c529729 | Merge commit '3bb309944b7e8542b2ac88ed3c9d9a60e68644d7' of git://trac.sagemath.org/sage into t/29124/script-packages-prereq-toolchain-bootstrap |
b73d6f0 | Merge tag '9.3.beta4' into t/29124/script-packages-prereq-toolchain-bootstrap |
e9ca2c1 | Merge branch 't/29124/script-packages-prereq-toolchain-bootstrap' into t/30944/tox__improve_local_sudo_ubuntu_standard |
d75dc81 | tox.ini (nobootstrap): Do not install system packages for _bootstrap |
Branch pushed to git repo; I updated commit sha1. New commits:
211ad2a | tox.ini (conda-forge-!environment_yml): Do not use condarc.yml, do not install packages other than _bootstrap via conda install |
Branch pushed to git repo; I updated commit sha1. New commits:
4856c09 | tox.ini (local-conda-forge-miniconda): Add variant using the miniconda installer |
Replying to @mkoeppe:
Can't continue at the moment because miniforge3 on macOS seems broken at the moment
[y/N]:No matter what I answer to this promptless question, it aborts
Figured out what was wrong: This was caused by having a .condarc that only defines pkg_dirs but no channels.
Branch pushed to git repo; I updated commit sha1. New commits:
3633e72 | tox.ini (conda-forge-environment_yml): Do use SAGE_ROOT/condarc.yml |
Branch pushed to git repo; I updated commit sha1. New commits:
ecc43b8 | tox.ini (local-conda-environment_yml, local-conda-environment_yml-optional): Finish |
Author: Matthias Koeppe
The title says something about github actions. Do you still plan to do this in this ticket, or is the new local-conda-environment_yml automatically executed?
(by the way, is the yml part of local-conda-environment_yml really necessary? this is hard to remember/write)
So what is the new workflow now with these changes? How do I now create a conda environment with a given name in a given location with the sage dependencies installed, and activate this env later and run say the doctests? Similarly, say I've created a conda env myself and activated it. How can I now install the sage dependencies and run tests/lints using tox in this env?
For the part -n {env:CONDA_SAGE_ENVIRONMENT}, the github action or conda itself (?) issued warnings if the name of the environment was different to what was specified in the environment.yml. In either case, I think the default should be what is specfied in the environment.yml, which I think is sage-build.
Description changed:
---
+++
@@ -1,4 +1,17 @@
-We create a `tox` environment `local-conda-forge-environment_yml`.
-In contrast to `...-standard`, it uses the file `environment*.yml` (generated by `bootstrap`) to set up the conda environment.
+We create `tox` environments `local-conda-forge-environment[-optional]`.
+In contrast to `...-standard` and `...-minimal`, they use the files `$SAGE_ROOT/environment*.yml` (generated by `bootstrap`) to set up the conda environment.
+
+To test locally:
+
+```
+ tox -e local-conda-forge-environment
+ tox -e local-conda-forge-environment-optional
+```
+
+We also add them to the GH Actions workflow to ensure that the files `environment*.yml` are generated correctly.
+See also:
+- #30845, which adds tests for the files `SAGE_ROOT/src/environment*.yml`
+
+Branch pushed to git repo; I updated commit sha1. New commits:
2a7e593 | .github/workflows/tox.yml: Add local-conda-forge-{macos,ubuntu}-environment_yml[-optional] |
a13fd7d | tox.ini: Do not rely on negated factor conditions for EXTRA_SAGE_PACKAGES |
5b945b5 | tox.ini, tox.yml: Rename environment_yml to environment |
1fac20f | tox.ini (local-conda-environment): Use CONDA_SAGE_ENVIRONMENT=sage-build |
Replying to @tobiasdiez:
How do I now create a conda environment with a given name in a given location with the sage dependencies installed, and activate this env later and run say the doctests? Similarly, say I've created a conda env myself and activated it. How can I now install the sage dependencies and run tests/lints using tox in this env?
Great questions. I hope we can add the answers in #28746 (Developer's Guide: Add instructions for sagelib development in a conda environment)
The present ticket does not change anything in this regard. It only tests that environment.yml can be used to set up some environment in which the Sage distribution can be built and tested.
By the way, when you have a chance, could you continue the review of #30940 (or indicate that another reviewer is needed)? It's an indirect prerequisite of this ticket.
Replying to @mkoeppe:
Great questions. I hope we can add the answers in #28746 (Developer's Guide: Add instructions for sagelib development in a conda environment)
I thought the point of using tox in order to manage the conda was to make sure that locally a developer can easily work in a similar way to what is tested on the CI. These question arose because I have trouble understanding how this interplay now works. Ok, I can now run tox -e local-conda-forge-environment but what is the advantage of this over conda env create environment.yml && sage -t ?
Replying to @tobiasdiez:
I thought the point of using tox in order to manage the conda was to make sure that locally a developer can easily work in a similar way to what is tested on the CI.
That's right.
These question arose because I have trouble understanding how this interplay now works. Ok, I can now run
tox -e local-conda-forge-environmentbut what is the advantage of this overconda env create environment.yml && sage -t?
conda env create environment.yml && sage -t does not work -- it's missing the build of Sage between the two commands!
tox -e local-conda-forge-environment includes that -- and it does so in an isolated install tree (SAGE_LOCAL) under .tox/
Branch pushed to git repo; I updated commit sha1. New commits:
4e15dd1 | Merge tag '9.3.beta7' into t/31099/tox_ini__gh_actions__add_test_for_sage_distribution_on_conda_using_environment_yml |
does this need anything from Conda to begin with, or it takes care even of installing Conda?
Oh, I see, it seems to be setting everything up... Thus, also a new way to install Sage, all you need is source tree and tox, no?
Seems that building is done single-thread for some reason. I.e.
make --no-print-directory all-sage
is only using one core. Is it on purpose?
All of the local tox environments pass on the MAKE variable. If you want to build in parallel, use export MAKE="make -j16".
On this ticket, the only change to the existing tox -e local-conda-forge-standard is that using the environment files is tested, instead of using conda install PACKAGES....
lgtm
I tested these locally on Linux, looks good. Further fixes might be added later.
Changed reviewer from https://github.com/mkoeppe/sage/actions/runs/445866532 to Dima Pasechnik
Thanks!
Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:
b998c84 | Merge tag '9.3.beta8' into t/31099/tox_ini__gh_actions__add_test_for_sage_distribution_on_conda_using_environment_yml |