tox.ini: Add `wsl` as a technology, to complement `local` and `docker`
mkoeppe opened this issue · 12 comments
This is for portability testing of the Sage distribution when developing on Windows, as an alternative to using Docker.
wsl would be a "technology", like local or docker -
https://doc.sagemath.org/html/en/developer/portability_testing.html#automatic-docker-based-build-testing-using-tox
The GH workflow for wsl testing from #30216 could then be simplified and generalized by going through tox.
(follow-up from #30216)
References:
Manual install: https://docs.microsoft.com/en-us/windows/wsl/install-manual
Windows Subsystem for Linux Command Reference | Microsoft Docs
https://docs.microsoft.com/en-us/windows/wsl/reference
windows subsystem for linux - How do you install multiple, separate instances of Ubuntu in WSL? - Stack Overflow
https://stackoverflow.com/questions/51584765/how-do-you-install-multiple-separate-instances-of-ubuntu-in-wsl
DDoSolitary/LxRunOffline: A full-featured utility for managing Windows Subsystem for Linux (WSL)
https://github.com/DDoSolitary/LxRunOffline
Manually download Windows Subsystem for Linux (WSL) Distros | Microsoft Docs
https://docs.microsoft.com/en-us/windows/wsl/install-manual
Install Windows Subsystem for Linux (WSL) on Windows 10 | Microsoft Docs
https://docs.microsoft.com/en-us/windows/wsl/install-win10
WSL - Ubuntu Wiki
https://wiki.ubuntu.com/WSL
Depends on #30923
CC: @tobiasdiez @sagetrac-tmonteil
Component: porting
Issue created by migration from https://trac.sagemath.org/ticket/30505
mkoeppe wrote:
In the long run (on some follow up ticket), I think it would actually be good to push wsl provisioning down one level, into tox.ini, by defining environments such as wsl-ubuntu-focal-standard (in analogy to docker-ubuntu-focal-standard). So that on a Windows box, you would be able to just invoke tox to create an isolated wsl container that is only used for testing.
I'm not sure how helpful it is to push it to the tox.ini. For me as a Windows user, WSL is the most convenient way to run sage. But for this I have a central WSL which I setup once and then reuse. Moreover, VS Code can connect to this WSL instance and run all commands trough it as needed. Thus, the workflow is different than say with docker containers.
This ticket is for a different purpose, namely portability testing - where you run Sage against various Linux distributions (and configurations, i.e., what system packages are installed). We do this with Docker, but on a Windows development box it may be more convenient to do it with isolated WSL instances instead.
Description changed:
---
+++
@@ -4,3 +4,27 @@
https://doc.sagemath.org/html/en/developer/portability_testing.html#automatic-docker-based-build-testing-using-tox
(follow-up from #30216)
+
+---
+
+References:
+
+Windows Subsystem for Linux Command Reference | Microsoft Docs
+https://docs.microsoft.com/en-us/windows/wsl/reference
+
+windows subsystem for linux - How do you install multiple, separate instances of Ubuntu in WSL? - Stack Overflow
+https://stackoverflow.com/questions/51584765/how-do-you-install-multiple-separate-instances-of-ubuntu-in-wsl
+
+DDoSolitary/LxRunOffline: A full-featured utility for managing Windows Subsystem for Linux (WSL)
+https://github.com/DDoSolitary/LxRunOffline
+
+Manually download Windows Subsystem for Linux (WSL) Distros | Microsoft Docs
+https://docs.microsoft.com/en-us/windows/wsl/install-manual
+
+Install Windows Subsystem for Linux (WSL) on Windows 10 | Microsoft Docs
+https://docs.microsoft.com/en-us/windows/wsl/install-win10
+
+WSL - Ubuntu Wiki
+https://wiki.ubuntu.com/WSL
+
+Description changed:
---
+++
@@ -2,6 +2,8 @@
`wsl` would be a "technology", like `local` or `docker` -
https://doc.sagemath.org/html/en/developer/portability_testing.html#automatic-docker-based-build-testing-using-tox
+
+The GH workflow for wsl testing from #30216 could then be simplified and generalized by going through tox.
(follow-up from #30216)
Tobias, can we try to get this ticket going? Basically I want to add to the top-level tox.ini a way to create a new (isolated) WSL instance (with several options for the Linux distribution to use).
What exactly do you want to accomplish? Do you want to add a tox command that downloads a given ubuntu distro, installs it as a wsl, installs sage including all dependencies and then run all tests in this wsl?
If that's the case, why do you want to manage the wsl installation from within tox? For integration tests, it would be easier to install wsl with a given linux distro in the github action and then run tox relative to this wsl. For development, it is also easier to let the user setup the initial wsl and reuse it.
(By the way, the same also applies to docker images where you can easily run the github action jobs in a docker container and don't need to manage the container from tox, https://www.petefreitag.com/item/903.cfm. This should also somewhat speedup the workflow as github caches the container.)
Replying to @tobiasdiez:
What exactly do you want to accomplish? Do you want to add a tox command that downloads a given ubuntu distro, installs it as a wsl, installs sage including all dependencies and then run all tests in this wsl?
If that's the case, why do you want to manage the wsl installation from within tox?
For local testing and debugging of portability issues. GH Actions only goes so far, at some point one needs to get an actual local installation.
For development, it is also easier to let the user setup the initial wsl and reuse it.
No, the point is that the tox command will automate this step so that one has a reproducible way to set up a (disposable) wsl copy. In other words, there is not "the" initial wsl but perhaps a dozen wsls with different configurations. By default, a wsl would be reused but the developer would be able to set it up from scratch using tox -r.
It's the same automation win that the existing docker-... tox environments give us: Of course I can set up a Docker container, install packages with apt-get, and then run my tests of the Sage build in it. But with the tox this just becomes one command, tox -e docker-ubuntu-bionic-standard.
One question that I am unsure about: Where would tox come from in a typical Windows developer's system, and what shell would be used to execute the tox commands?
Thanks for your explanation. But I still don't really get why you need/want to manage the wsl installation via tox. If tox is able to configure a normal linux install, then it's only one additional step for the user to create a wsl installation with the said linux.
Moreover, typical WSL installs are of the order of 2 to 5gb (mine is 22gb, which is used exclusively for sage). That's not something you would use in a use-once-then-throw-away spirit, like docker images.
But, of course, you could move the current code in the github wsl workflow to a new tox environment. As a first step towards this, and what I would find actually more important, is to have local-ubuntu environments similar to local-homebrew.
Then you can simply install ubuntu in wsl via the Windows store, and run
wsl
> tox -e local-ubuntu-standard
That's the same amount of characters as tox -e local-wsl-ubuntu-standard ;-)
Replying to @mkoeppe:
One question that I am unsure about: Where would
toxcome from in a typical Windows developer's system, and what shell would be used to execute the tox commands?
You would need to have a python installed in windows, then install tox via pip. Tox would usually be invoked from the "Windows Command line" or powershell console.
Description changed:
---
+++
@@ -10,6 +10,8 @@
---
References:
+
+Manual install: https://docs.microsoft.com/en-us/windows/wsl/install-manual
Windows Subsystem for Linux Command Reference | Microsoft Docs
https://docs.microsoft.com/en-us/windows/wsl/referenceReplying to @tobiasdiez:
of course, you could move the current code in the github wsl workflow to a new tox environment. As a first step towards this, and what I would find actually more important, is to have
local-ubuntuenvironments similar tolocal-homebrew.
Then you can simply install ubuntu in wsl via the Windows store, and runwsl > tox -e local-ubuntu-standardThat's the same amount of characters as
tox -e local-wsl-ubuntu-standard;-)
OK, sounds good, let's do this as the first step. That's now #30923
Setting new milestone based on a cursory review of ticket status, priority, and last modification date.