Add GitHub Actions workflow for testing the build on various Linux distributions via docker; macOS with homebrew; and Windows (cygwin)
mkoeppe opened this issue · 129 comments
This is a follow-up on #29053: Add debian/fedora/arch package information to build/pkgs, generate Dockerfiles and installation help
This ticket adds .github/workflows/tox.yml, which installs a number of system packages and then runs ./bootstrap, ./configure, and make.
Most of the interesting parts are in #29053 - everything can be run "locally" (with tox and docker). This ticket only adds a simple additional file to run this automatically with several jobs on GitHub Actions on a GitHub pull request.
For each of the Linux distributions supported, there are two configurations:
minimal- has enough packages so that the build can succeed;standard- has additionally all standard packages for which we havespkg-configure.m4.
GitHub Actions runs these build jobs on 2-core machines with 7 GB of RAM memory and 14 GB of SSD disk space (https://help.github.com/en/actions/reference/virtual-environments-for-github-hosted-runners#supported-runners-and-hardware-resources), and has a time limit of 6h per job. This is just barely enough for a typical minimal build followed by make ptest to succeed; and plenty of time for a typical standard build to succeed.
Example run: https://github.com/mkoeppe/sage/actions/runs/60588138
Context:
- #29060: Meta-ticket: Add
Dockerfiles and CI scripts for integration testing of source and binary distributions and of downstream packages - #27330: Meta-ticket:
spkg-configure: Try to use as many system packages as possible
Depends on #29367
CC: @dimpase @vbraun @saraedum @isuruf @antonio-rojas @kiwifb
Component: build
Keywords: ContinuousIntegration
Author: Matthias Koeppe
Branch/Commit: cbcd657
Reviewer: Dima Pasechnik
Issue created by migration from https://trac.sagemath.org/ticket/29087
I am all for more CI, but would it not make more sense to add this to the existing GitLab CI?
Last 10 new commits:
80d2f90 | build/pkgs/debian-bootstrap.txt: Add pkg-config |
d31441e | remove nonexisting fedora package |
e4c2bf8 | build/pkgs/arch.txt: Add gcc |
c55071f | build/pkgs/debian.txt: add g++ |
ea04ee7 | build/pkgs/debian.txt: python3-minimal->python3 |
b4e90a0 | build/bin/write-dockerfile.sh: Only install a package if it has spkg-configure.m4 |
0f2b16e | build/bin/sage-spkg: Explain to user what will be done with the system package |
ddcac27 | Squashed commit of the following: |
8c30f07 | Arch package sage-data-elliptic_curves corresponds to Sage's elliptic_curves, not to the optional database_cremona_ellcurve |
6a2bf3d | Add [GitHub](../wiki/GitHub) Actions workflow for testing spkg-configure / build on various Linux distributions |
Description changed:
---
+++
@@ -8,6 +8,9 @@
- `minimal` - has enough packages so that the build can succeed;
- `standard` - has additionally all standard packages for which we have `spkg-configure.m4`.
+Example run: https://github.com/mkoeppe/sage/actions/runs/31716135
+
+
Context:
- #29060: Meta-ticket: Add `Dockerfile`s and CI scripts for integration testing of source and binary distributions and of downstream packages
- #27330: Meta-ticket: `spkg-configure`: Try to use as many system packages as possibleAuthor: Matthias Koeppe
Replying to @saraedum:
I am all for more CI, but would it not make more sense to add this to the existing GitLab CI?
Fine with me too. I have built this on top of tox and docker. It can also run locally. The GitHub-specific part is really minimal - https://github.com/sagemath/sagetrac-mirror/commits/6a2bf3d2140e2d625d4ca96a40485b74fa582e7a
Changed keywords from none to ContinuousIntegration
Personally, I like GitHub. I like GitLab a bit better because it is very easy to provision our own runners (e.g., for obscure platforms.) The problem with GitHub is that some people have quite a few reservations about it (lock-in, Microsoft, …) so I think GitLab is the better choice for such an extensive community.
In any case, glad to see these things happen. Feel free to ping/bug me explicitly if you need reviews on anything docker/CI.
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
3d9565f | fixup conda |
7e59a6f | build/bin/write-dockerfile.sh: Add commands to activate conda env |
2ad9a97 | Merge tag '9.1.beta2' into t/29053/installation_manual__generate__apt_get_install__and__yum_install__lines_from_build_pkgs_at___bootstrap_time |
9fca230 | add doc |
f6cbb8a | add centos-{7,8} |
a9b1ff2 | Add [GitHub](../wiki/GitHub) Actions workflow for testing spkg-configure / build on various Linux distributions |
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
c651058 | on centos, IGNORE_MISSING_SYSTEM_PACKAGES=yes |
98edff6 | Add build/pkgs/gfortran/distros/fedora.txt |
843ca76 | build/pkgs/fedora.txt: Add python3 |
bd6a545 | tox.ini: Add debian stretch, put sid last |
23c6334 | build/pkgs/openblas/distros/arch.txt: Add lapack, cblas |
050d692 | Add [GitHub](../wiki/GitHub) Actions workflow for testing spkg-configure / build on various Linux distributions |
744b04e | .github/workflows/tox.yml: Also put the tox log in the artifact |
Branch pushed to git repo; I updated commit sha1. New commits:
fca6654 | tox.ini: Add toxenvs local-homebrew-macos-minimal, local-homebrew-macos-standard |
e0e8a4c | tox.ini [local-homebrew]: Fixup PATH, PKG_CONFIG_PATH, set LDFLAGS, CPPFLAGS; [local-direct]: New |
16ca88f | tox.ini: Use {envlogdir}; use ln -sf |
6a1803d | tox.ini [local]: First build the base-toolchain |
4b72987 | tox.ini [local-homebrew]: Set PATH earlier and set ACLOCAL_PATH for bootstrapping |
043c2fa | Merge branch 't/29104/tox-homebrew' into t/29087/CI-gh-actions-tox-configure-build |
931cdf8 | spkg-configure.m4 for boost_cropped and boost |
6f04521 | add package names to distros/ |
998287b | Merge branch 'u/dimpase/packages/boostconfig' of git://trac.sagemath.org/sage into t/29087/CI-gh-actions-tox-configure-build |
879d1a4 | .github/workflows/tox.yml: Add tox-local-macos |
Replying to @saraedum:
Personally, I like GitHub. I like GitLab a bit better because it is very easy to provision our own runners (e.g., for obscure platforms.) The problem with GitHub is that some people have quite a few reservations about it (lock-in, Microsoft, …) so I think GitLab is the better choice for such an extensive community.
Yes, I understand these concerns. To avoid lock-in, in this ticket the definition of the tests (for linux) is driven by tox and docker, rather than relying on the specific environments provided by a CI vendor. Likewise for the homebrew tests for macOS on this ticket.
I don't have the time to work on another CI platform but would certainly welcome it if other CI enthusiasts wanted to implement a solution on another platform in a follow-up ticket.
Feel free to ping/bug me explicitly if you need reviews on anything docker/CI.
Thanks. This ticket would be ready for review.
Branch pushed to git repo; I updated commit sha1. New commits:
f311a93 | .github/workflows/tox.yml: Add rpy2, ecl, maxima to TARGETS; also run on pull_request |
Branch pushed to git repo; I updated commit sha1. New commits:
2b9eb79 | Merge tag '9.1.beta3' into t/29087/CI-gh-actions-tox-configure-build |
839d6fb | tox.ini: Add other archs supported by docker |
be0bf8d | Merge branch 't/29143/add_to_tox_ini_some_32_bit_linux_environments' into t/29087/CI-gh-actions-tox-configure-build |
9b8c45f | tox.ini: Add a few more systems |
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
67e14ce | fixup for llocal |
487939d | Merge branch 't/29143/add_to_tox_ini_some_32_bit_linux_environments' into t/29087/CI-gh-actions-tox-configure-build |
e10976b | tox.ini: Add a few more systems |
d611672 | tox.ini: Add a few more systems |
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
e35d3ef | tox.ini: Add a few more systems |
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
2141b75 | Merge tag '9.1.beta3' into t/29104/tox-homebrew |
2b0a25e | tox.ini: Add other archs supported by docker |
e3f832c | fixup for llocal |
b5361e2 | tox.ini: Restructure docker image:tag construction, add multiarch for Linux Docker |
c497ef7 | Add [GitHub](../wiki/GitHub) Actions workflow for testing spkg-configure / build on various Linux distributions |
7a44abf | .github/workflows/tox.yml: Also put the tox log in the artifact |
d52513b | .github/workflows/tox.yml: Add tox-local-macos |
c2973a5 | .github/workflows/tox.yml: Add rpy2, ecl, maxima to TARGETS; also run on pull_request |
dfba1d0 | .github/workflows/tox.yml: add centos, some i386 builds, and arm64 armhf builds |
Branch pushed to git repo; I updated commit sha1. New commits:
2a1dbac | remove centos-8-i386 (does not exist on Docker Hub |
Branch pushed to git repo; I updated commit sha1. New commits:
4350d0a | tox.ini: Add raspbian |
59e1d6f | Merge tag '9.1.beta4' into t/29143/add_to_tox_ini_some_32_bit_linux_environments |
4eb201b | Merge tag '9.1.beta4' into t/29104/tox-homebrew |
254753d | Merge branch 't/29104/tox-homebrew' into t/29143/add_to_tox_ini_some_32_bit_linux_environments |
bd9df11 | Merge branch 't/29143/add_to_tox_ini_some_32_bit_linux_environments' into t/29087/CI-gh-actions-tox-configure-build |
Branch pushed to git repo; I updated commit sha1. New commits:
b654bdb | .github/workflows/tox.yml [homebreew]: Prepare log artifact on error |
Branch pushed to git repo; I updated commit sha1. New commits:
15179a4 | Merge tag '9.1.beta5' into t/29087/CI-gh-actions-tox-configure-build |
Work Issues: rebase, test linbox for 29102
Branch pushed to git repo; I updated commit sha1. New commits:
6bd60d8 | .github/workflows/tox.yml: Add scipy, fpylll, linbox to TARGETS, pass TARGETS to homebrew build |
Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
d576c4c | tox.ini: Add docker-linuxmint-19.3 |
181a8e3 | Merge branch 't/29304/tox_ini__add_docker_linuxmint' into t/29087/CI-gh-actions-tox-configure-build |
cb6f4f3 | build/tox.ini: Add more python versions |
f421bd7 | build/test: Set maxDiff=None to improve debugging of sage_bootstrap |
c02af0d | build/sage_bootstrap: Handle upstream_url field in checksum.ini, add options to 'sage -package' |
5b4a9e8 | build/sage_bootstrap/download: Add option --allow-upstream |
537210d | build/bin/sage-spkg: Add option -o (--allow-upstream) |
e234145 | Merge branch 't/26351/build_pkgs___checksums_ini__add_upstream_tarball_url_field' into t/29087/CI-gh-actions-tox-configure-build |
82b4c09 | tox.ini, build/bin/write-dockerfile.sh: Use sage-spkg -y -o, for packages not on mirrors yet |
3acd913 | .github/workflows/tox.yml: Set TARGETS=build |
Branch pushed to git repo; I updated commit sha1. New commits:
04613b5 | tox.yml: Add linuxmint-19.3 |
Branch pushed to git repo; I updated commit sha1. New commits:
9f486ef | Revert "Move AC_FC_FUNC([dgeqrf]) from openblas/spkg-configure.m4 to gfortran/spkg-configure.m4" |
071e0b3 | Revert "Revert "build/pkgs/openblas/spkg-configure.m4: Do not use AC_FC_FUNC."" |
5c8f4ac | Merge branch 't/29104/tox-homebrew' into t/29087/CI-gh-actions-tox-configure-build |
Branch pushed to git repo; I updated commit sha1. New commits:
31f9250 | tox.ini [fedora-31, fedora-32]: Remove comments that caused the wrong target to be built |
what's the story with AC_FC_FUNC?
Oops, it seems there is no way to call AC_FC_FUNC conditionally, it is still called even if there is no Fortran detected.
Weird, I don't understand why this happens.
Branch pushed to git repo; I updated commit sha1. New commits:
6490d74 | tox.ini: Do not run git testsuite (fails) |
I sent a message to autoconf bugs mailing list, asking whether inabiity to chain
AC_FC_FREEFORM and AC_FC_FUNC is a bug.
New commits:
6490d74 | tox.ini: Do not run git testsuite (fails) |
Branch pushed to git repo; I updated commit sha1. New commits:
46a220e | Implement SAGE_CHECK=warn, sage-spkg -w |
c1f09ec | Merge branch 't/29301/sage_spkg__add_an_option__w_for_warning_only_if_spkg_check_fails' into t/29087/CI-gh-actions-tox-configure-build |
f15ddbb | Use SAGE_CHECK=warn, disable git/ppl testsuites |
Branch pushed to git repo; I updated commit sha1. New commits:
0f70082 | build/pkgs/cygwin.txt: gcc -> gcc-core, gcc-c++ |
47e4395 | build/pkgs/cygwin.txt: Add libcrypt0 |
d95e51a | src/doc/bootstrap: Generate cygwin.txt, cygwin-optional.txt |
36f7b59 | Merge branch 't/29106/ticket-29106' into t/29295/29106+ci |
4941359 | Merge branch 't/29295/29106+ci' into t/29087/CI-gh-actions-tox-configure-build |
Branch pushed to git repo; I updated commit sha1. New commits:
5b7d9a5 | Also store test logs in the artifact |
Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
5776abf | build/bin/sage-guess-package-system: New |
6807e0d | configure: Output a notice regarding recommended system packages |
479c521 | build/bin/sage-guess-package-system: Suppress error output |
cee2894 | m4/sage_spkg_collect.m4: Fixup |
9a8f693 | Merge tag '9.1.beta7' into t/27351/at_end_of_configure__suggest_a_list_of_system_packages_to_install |
35e1e7d | Add missing file build/bin/sage-get-system-packages |
3fc734b | Merge branch 't/27351/at_end_of_configure__suggest_a_list_of_system_packages_to_install' into t/29087/CI-gh-actions-tox-configure-build |
16485f4 | m4/sage_spkg_collect.m4: Adjust style of configgure messages |
d6e8332 | .github/workflows/tox.yml: Mark up configure notice/warning/error |
5627ee1 | .github/workflows/tox.yml: Fix last change: Do not ignore errors from tox |
Could you factor out into a separate ticket the fix for AC_FC_FREEFORM and AC_FC_FUNC thing? Cause it's really a bug fix.
Will do.
Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
96553db | build/pkgs/ntl/spkg-configure.m4: Check for NTL configuration |
ec5cd78 | Merge branch 't/29339/fix_ntl_spkg_configure_m4_so_it_rejects_ntls_built_with_ntl_threads__without_ntl_gmp_lip__without_ntl_gf2x_lib' into t/29104/tox-homebrew |
c3bc092 | Remove tests for NTL_GMP_LIP, NTL_GF2X_LIB |
6d2f6e1 | Merge branch 't/29339/fix_ntl_spkg_configure_m4_so_it_rejects_ntls_built_with_ntl_threads__without_ntl_gmp_lip__without_ntl_gf2x_lib' into t/29104/tox-homebrew |
14e9f12 | build/pkgs/ntl/distros/homebrew.txt: Remove ntl |
70e161b | Remove use of homebrew packages flint, arb |
11f7729 | build/pkgs/patch/distros/homebrew.txt: New |
93a0421 | build/pkgs/boost*/distros/homebrew.txt: New |
4c653c2 | .gitignore: Add prefix |
576a188 | Merge branch 't/29104/tox-homebrew' into t/29087/CI-gh-actions-tox-configure-build |
Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
e8b56ee | .github/workflows/tox.yml: Set TARGETS=build |
4cee692 | tox.yml: Add linuxmint-19.3 |
bcbe627 | tox.ini: Do not run git testsuite (fails) |
d8deee4 | Use SAGE_CHECK=warn, disable git/ppl testsuites |
fe33952 | test 'make build' and ptest |
1311a1d | build/bin/write-dockerfile.sh: Before adding src, make TARGETS_PRE; afterwards, make TARGETS. |
932d46a | Also store test logs in the artifact |
719c4bf | .github/workflows/tox.yml: Mark up configure notice/warning/error |
ad51524 | .github/workflows/tox.yml: Fix last change: Do not ignore errors from tox |
e6a44a3 | .github/workflows/tox.yml: Add slackware-14.2 |
Branch pushed to git repo; I updated commit sha1. New commits:
566533c | github/workflows/tox.yml [tox-local-homebrew-macos]: Set parallelization separately for build and doctests |
So do I understand correctly that merging this into a branch and pushing to github (to a repo with enabled GitHub actions) will trigger the testing. This is not so nice UI. Do I miss something here?
I guess it's just one file that needs to be changed, which one?
I'd prefer to have a command-line way (perhaps using some GitHub tools) to trigger GH Actions on a given branch, say.
When this ticket is merged into Sage, it will suffice to just push to github.
Because the checks run for hours, it would probably be better to run the workflow only on pull requests, rather than on all pushes.
on:
pull_request:
types: [opened, synchronize]