tox.ini update: Fix breakage caused by src/ext move, add 32-bit linux environments and other architectures supported by docker

Question

tox.ini update: Fix breakage caused by src/ext move, add 32-bit linux environments and other architectures supported by docker

mkoeppe opened this issue 6 years ago · 35 comments

#29053 added tox.ini to support testing the sage distribution in isolated environments via docker.

In 9.1.beta6, the generated docker files do not work due to #21785.

This ticket fixes it and also adds linux environments for various other architectures supported by docker (see https://github.com/docker-library/official-images#architectures-other-than-amd64).

Examples:

$ tox -e docker-debian-buster-standard-i386
$ tox -e docker-ubuntu-bionic-standard-arm64
$ tox -e docker-ubuntu-bionic-minimal-armhf
$ tox -e docker-fedora-31-standard-arm64v8         # Docker for Mac only
$ tox -e docker-ubuntu-latest-standard-s390x       # Docker for Mac only

(The last two work with macOS Docker Desktop because it has binfmt_misc with qemu enabled.)

See also: https://trac.sagemath.org/wiki/ExoticPorts

Resources:

How to Build and Run ARM Docker Containers on x86 Hosts · MatchboxBlog - https://matchboxdorry.gitbooks.io/matchboxblog/content/blogs/build_and_run_arm_images.html
Cross Building and Running Multi-Arch Docker Images – Micheal Waltz – DevOps Engineer - https://www.ecliptik.com/Cross-Building-and-Running-Multi-Arch-Docker-Images/
multiarch/qemu-user-static - Docker Hub - https://hub.docker.com/r/multiarch/qemu-user-static/
multiarch/ubuntu-core - Docker Hub - https://hub.docker.com/r/multiarch/ubuntu-core/

CC: @vbraun @sagetrac-tmonteil @kiwifb @dimpase

Component: porting

Author: Matthias Koeppe

Branch/Commit: 7f122bd

Reviewer: Volker Braun

Issue created by migration from https://trac.sagemath.org/ticket/29143

Answer 1 · 2020-02-02T17:46:40.000Z

Description changed:

--- 
+++ 
@@ -2,3 +2,4 @@
 
 This ticket adds some 32-bit linux environments such as https://hub.docker.com/r/i386/ubuntu/
 
+https://github.com/docker-library/official-images#architectures-other-than-amd64

Answer 2 · 2020-02-02T17:53:50.000Z

Dependencies: #29053, #29104

Answer 3 · 2020-02-02T17:53:50.000Z

comment:2

on top of #29104 to avoid merge conflicts

Answer 4 · 2020-02-02T19:19:29.000Z

Branch: u/mkoeppe/add_to_tox_ini_some_32_bit_linux_environments

Answer 5 · 2020-02-02T19:21:50.000Z

Commit: 839d6fb

Answer 6 · 2020-02-02T19:21:50.000Z

Last 10 new commits:

`98edff6`	`Add build/pkgs/gfortran/distros/fedora.txt`
`843ca76`	`build/pkgs/fedora.txt: Add python3`
`bd6a545`	`tox.ini: Add debian stretch, put sid last`
`23c6334`	`build/pkgs/openblas/distros/arch.txt: Add lapack, cblas`
`fca6654`	`tox.ini: Add toxenvs local-homebrew-macos-minimal, local-homebrew-macos-standard`
`e0e8a4c`	`tox.ini [local-homebrew]: Fixup PATH, PKG_CONFIG_PATH, set LDFLAGS, CPPFLAGS; [local-direct]: New`
`16ca88f`	`tox.ini: Use {envlogdir}; use ln -sf`
`6a1803d`	`tox.ini [local]: First build the base-toolchain`
`4b72987`	`tox.ini [local-homebrew]: Set PATH earlier and set ACLOCAL_PATH for bootstrapping`
`839d6fb`	`tox.ini: Add other archs supported by docker`

Answer 7 · 2020-02-02T19:21:50.000Z

Description changed:

--- 
+++ 
@@ -1,5 +1,13 @@
 #29053 adds `tox.ini` to support testing the sage distribution in isolated environments via docker.
 
-This ticket adds some 32-bit linux environments such as https://hub.docker.com/r/i386/ubuntu/
+This ticket adds linux environments for various other architectures supported by docker (see https://github.com/docker-library/official-images#architectures-other-than-amd64).
 
-https://github.com/docker-library/official-images#architectures-other-than-amd64
+Examples:
+
+```
+$ tox -e docker-fedora-31-standard-arm64v8
+$ tox -e docker-debian-buster-standard-i386
+$ tox -e docker-ubuntu-latest-standard-s390x
+```
+
+}}}

Answer 8 · 2020-02-02T19:23:22.000Z

Author: Matthias Koeppe

Answer 9 · 2020-02-02T21:00:12.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`67e14ce`	`fixup for llocal`

Answer 10 · 2020-02-02T21:00:12.000Z

Changed commit from 839d6fb to 67e14ce

Answer 11 · 2020-02-03T14:07:52.000Z

comment:7

This works on macOS because it is running everything in a virtualbox.

I haven't succeeded in getting it to work on github actions yet. last attempt: https://github.com/mkoeppe/sage/runs/422184737?check_suite_focus=true

Answer 12 · 2020-02-03T14:18:15.000Z

Description changed:

--- 
+++ 
@@ -10,4 +10,3 @@
 $ tox -e docker-ubuntu-latest-standard-s390x
 ```
 
-}}}

Answer 13 · 2020-02-04T22:03:52.000Z

Description changed:

--- 
+++ 
@@ -10,3 +10,9 @@
 $ tox -e docker-ubuntu-latest-standard-s390x
 ```
 
+This works out of the box with macOS Docker Desktop because it has `binfmt_misc` with `qemu` enabled.
+- How to Build and Run ARM Docker Containers on x86 Hosts · MatchboxBlog - https://matchboxdorry.gitbooks.io/matchboxblog/content/blogs/build_and_run_arm_images.html
+- Cross Building and Running Multi-Arch Docker Images – Micheal Waltz – DevOps Engineer - https://www.ecliptik.com/Cross-Building-and-Running-Multi-Arch-Docker-Images/
+- multiarch/qemu-user-static - Docker Hub - https://hub.docker.com/r/multiarch/qemu-user-static/
+- multiarch/ubuntu-core - Docker Hub - https://hub.docker.com/r/multiarch/ubuntu-core/
+

Answer 14 · 2020-02-06T14:39:01.000Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`2141b75`	`Merge tag '9.1.beta3' into t/29104/tox-homebrew`
`2b0a25e`	`tox.ini: Add other archs supported by docker`
`e3f832c`	`fixup for llocal`
`b5361e2`	`tox.ini: Restructure docker image:tag construction, add multiarch for Linux Docker`

Answer 15 · 2020-02-06T14:39:01.000Z

Changed commit from 67e14ce to b5361e2

Answer 16 · 2020-02-06T14:42:13.000Z

Description changed:

--- 
+++ 
@@ -5,9 +5,11 @@
 Examples:
 
 ```
-$ tox -e docker-fedora-31-standard-arm64v8
 $ tox -e docker-debian-buster-standard-i386
-$ tox -e docker-ubuntu-latest-standard-s390x
+$ tox -e docker-ubuntu-bionic-standard-arm64
+$ tox -e docker-ubuntu-bionic-minimal-armhf
+$ tox -e docker-fedora-31-standard-arm64v8         # Docker for Mac only
+$ tox -e docker-ubuntu-latest-standard-s390x       # Docker for Mac only
 ```
 
 This works out of the box with macOS Docker Desktop because it has `binfmt_misc` with `qemu` enabled.

Answer 17 · 2020-02-06T14:46:33.000Z

Changed dependencies from #29053, #29104 to #29104

Answer 18 · 2020-02-06T14:46:33.000Z

Description changed:

--- 
+++ 
@@ -12,7 +12,13 @@
 $ tox -e docker-ubuntu-latest-standard-s390x       # Docker for Mac only
 ```
 
-This works out of the box with macOS Docker Desktop because it has `binfmt_misc` with `qemu` enabled.
+(The last two work with macOS Docker Desktop because it has `binfmt_misc` with `qemu` enabled.)
+
+The branch is on top of #29104 (homebrew).
+
+---
+
+Resources:
 - How to Build and Run ARM Docker Containers on x86 Hosts · MatchboxBlog - https://matchboxdorry.gitbooks.io/matchboxblog/content/blogs/build_and_run_arm_images.html
 - Cross Building and Running Multi-Arch Docker Images – Micheal Waltz – DevOps Engineer - https://www.ecliptik.com/Cross-Building-and-Running-Multi-Arch-Docker-Images/
 - multiarch/qemu-user-static - Docker Hub - https://hub.docker.com/r/multiarch/qemu-user-static/

Answer 19 · 2020-02-06T21:04:38.000Z

comment:13

Tests run at https://github.com/mkoeppe/sage/actions/runs/35630769

Answer 20 · 2020-02-08T23:00:09.000Z

Description changed:

--- 
+++ 
@@ -16,6 +16,8 @@
 
 The branch is on top of #29104 (homebrew).
 
+See also: https://trac.sagemath.org/wiki/ExoticPorts
+
 ---
 
 Resources:

Answer 21 · 2020-02-09T15:47:12.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`4350d0a`	`tox.ini: Add raspbian`

Answer 22 · 2020-02-09T15:47:12.000Z

Changed commit from b5361e2 to 4350d0a

Answer 23 · 2020-02-10T21:04:05.000Z

comment:16

Also this ticket needs review and would facilitate testing on 32bit x86 and ARM

Answer 24 · 2020-02-16T18:05:29.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`59e1d6f`	`Merge tag '9.1.beta4' into t/29143/add_to_tox_ini_some_32_bit_linux_environments`

Answer 25 · 2020-02-16T18:05:29.000Z

Changed commit from 4350d0a to 59e1d6f

Answer 26 · 2020-02-16T20:45:23.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`4eb201b`	`Merge tag '9.1.beta4' into t/29104/tox-homebrew`
`254753d`	`Merge branch 't/29104/tox-homebrew' into t/29143/add_to_tox_ini_some_32_bit_linux_environments`

Answer 27 · 2020-02-16T20:45:23.000Z

Changed commit from 59e1d6f to 254753d

Answer 28 · 2020-03-02T02:36:08.000Z

Changed commit from 254753d to 7f122bd

Answer 29 · 2020-03-02T02:36:08.000Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`749f83e`	`tox.ini: Add other archs supported by docker`
`e941bd5`	`fixup for llocal`
`2441fe9`	`tox.ini: Restructure docker image:tag construction, add multiarch for Linux Docker`
`2f057cb`	`tox.ini: Add raspbian`
`7f122bd`	`build/bin/write-dockerfile.sh: src/ext has moved and is no longer needed for the configure phase`

Answer 30 · 2020-03-02T02:37:36.000Z

Changed dependencies from #29104 to none

Answer 31 · 2020-03-02T02:37:36.000Z

Description changed:

--- 
+++ 
@@ -1,6 +1,8 @@
-#29053 adds `tox.ini` to support testing the sage distribution in isolated environments via docker.
+#29053 added `tox.ini` to support testing the sage distribution in isolated environments via docker.
 
-This ticket adds linux environments for various other architectures supported by docker (see https://github.com/docker-library/official-images#architectures-other-than-amd64).
+In 9.1.beta6, the generated docker files do not work due to #21785.
+
+This ticket fixes it and also adds linux environments for various other architectures supported by docker (see https://github.com/docker-library/official-images#architectures-other-than-amd64).
 
 Examples:
 
@@ -14,8 +16,6 @@
 
 (The last two work with macOS Docker Desktop because it has `binfmt_misc` with `qemu` enabled.)
 
-The branch is on top of #29104 (homebrew).
-
 See also: https://trac.sagemath.org/wiki/ExoticPorts
 
 ---

Answer 32 · 2020-03-02T02:37:36.000Z

comment:20

The branch is no longer on top of #29104 (homebrew).

Needs review.

Answer 33 · 2020-03-02T23:23:44.000Z

Reviewer: Volker Braun

Answer 34 · 2020-03-02T23:33:37.000Z

comment:22

Thanks!

Answer 35 · 2020-03-03T23:21:26.000Z

Changed branch from u/mkoeppe/add_to_tox_ini_some_32_bit_linux_environments to 7f122bd