GitHub CI workflow: Add testing of optional packages

Question

GitHub CI workflow: Add testing of optional packages

mkoeppe opened this issue 6 years ago · 47 comments

We add a new GitHub Actions CI workflow tox-optional.yml, which tests most optional packages, using the maximal packages factor.

Depends on #29417
Depends on #29424
Depends on #29426
Depends on #29501
Depends on #29508
Depends on #29510
Depends on #29403

CC: @dimpase @kliem @orlitzky @jhpalmieri

Component: porting

Keywords: ContinuousIntegration

Author: Matthias Koeppe

Branch/Commit: 816477a

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/29341

Answer 1 · 2020-03-15T19:10:48.000Z

Changed keywords from none to ContinuousIntegration

Answer 2 · 2020-03-15T23:38:26.000Z

Branch: u/mkoeppe/github_ci_workflow__add_testing_of_optional_packages

Answer 3 · 2020-03-16T15:28:50.000Z

comment:3

Tests at https://github.com/mkoeppe/sage/actions/runs/56355410

polymake - fails on many platforms (#28423)
e-antic - fails on ubuntu-eoan-standard (#27952)
lrslib - fails on archlinux-latest-standard (#29185)

Last 10 new commits:

`14f2967`	`Actually use libcrypt-devel`
`68314dc`	`Merge branch 't/29106/ticket-29106' into t/29295/29106+ci`
`b263ac3`	`Run ci-cygwin.yml also on pull request`
`8972699`	`Merge branch 't/29295/29106+ci' into t/29087/CI-gh-actions-tox-configure-build`
`8802016`	`test 'make build' and ptest`
`16f979d`	`build/bin/write-dockerfile.sh: Before adding src, make TARGETS_PRE; afterwards, make TARGETS.`
`5b7d9a5`	`Also store test logs in the artifact`
`d1eb116`	`tox.ini: Move homebrew environment settings to script .homebrew-build-env`
`967b49d`	`Merge branch 't/29104/tox-homebrew' into t/29087/CI-gh-actions-tox-configure-build`
`69737b0`	`.github/workflows/tox.yml: Test some optional packages`

Answer 4 · 2020-03-16T15:28:50.000Z

Commit: 69737b0

Answer 5 · 2020-04-12T19:00:32.000Z

Changed dependencies from #29087 to #29417

Answer 6 · 2020-04-15T19:52:06.000Z

comment:5

An obstacle when testing platforms with -maximal and IGNORE_MISSING_SYSTEM_PACKAGES=yes, for example docker-fedora-26-maximal (https://github.com/mkoeppe/sage/runs/589959336)

docker build gives

max depth exceeded

(docker/for-linux#414)

Basically, a sequence of too many RUN commands. Need to revise write-dockerfile

Answer 7 · 2020-04-15T23:57:55.000Z

Changed commit from 69737b0 to ee21154

Answer 8 · 2020-04-15T23:57:55.000Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

`c08176e`	`Merge branch 't/29417/tox-ci-minor-fixes' into t/29403/ci_cygwin_yml__multi_stage_build`
`cd1550e`	`build/bin/sage-logger [V=0]: Show build times of successful builds`
`2c647fc`	`build/pkgs/*/spkg-check.in: Do not print misleading error messages: avoid sdh_make`
`fc914f3`	`Merge branch 't/29501/spkg_check_in__do_not_use_sdh_make_to_do__make_check_' into all-test-9.1.rc0`
`faf0ed6`	`tox.ini: Another quoting fix`
`b08b0b9`	`Merge branch 't/29417/tox-ci-minor-fixes' into all-test-9.1.rc0`
`ebb07d1`	`Optional/experimental packages: Remove order-only dependencies on all standard packages`
`d1110b9`	`Merge branch 't/29426/make_dependencies_of_optional_packages_more_detailed__no_longer_depend_on_all_standard_packages' into optional-test-9.1.rc0`
`dcfade3`	`Set targets`
`ee21154`	`tox.yml: do not test on conda-forge-ubuntu and fedora-32; use maximal`

Answer 9 · 2020-04-16T00:11:07.000Z

Changed commit from ee21154 to df56ac8

Answer 10 · 2020-04-16T00:11:07.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`aed3fb8`	`src/doc/en/developer/portability_testing.rst: Explain how to read [GitHub](../wiki/GitHub) Actions page`
`d39d627`	`tox.yml [python3_xcode]: Install the package`
`ba0c266`	`fixup`
`9060d23`	`fixup`
`bd3e826`	`Merge branch 't/29417/tox-ci-minor-fixes' into t/29341/optional-test-9.1.rc0`
`df56ac8`	`build/bin/write-dockerfile.sh [debian, fedora, slackware, arch]: Define EXISTS commands`

Answer 11 · 2020-04-16T00:44:52.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`3fce626`	`fixup`

Answer 12 · 2020-04-16T00:44:52.000Z

Changed commit from df56ac8 to 3fce626

Answer 13 · 2020-04-18T18:30:56.000Z

Changed dependencies from #29417 to #29417, #29426

Answer 14 · 2020-04-18T18:44:05.000Z

Changed commit from 3fce626 to 288d760

Answer 15 · 2020-04-18T18:44:05.000Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

`af1a4c7`	`build/pkgs/pandoc: New script package`
`a7f230b`	`build/pkgs/pandoc: Add distros/`
`fe346da`	`Merge branch 't/29508/add_system_package_lists_for_pandoc__prereq_of_rst2ipynb_' into t/29341/optional-test-9.1.rc0`
`67256c2`	`Use 'failures', not 'error' when reporting testsuite failures`
`2b68119`	`Merge branch 't/29501/spkg_check_in__do_not_use_sdh_make_to_do__make_check_' into t/29341/optional-test-9.1.rc0`
`ab0120e`	`build/pkgs/slackware*.txt: Move pkg-config, gettext-tools from bootstrap to prereq, add gettext dependency libcroco`
`fafd554`	`build/pkgs/libpng/distros/slackware.txt: New`
`eb9f1c1`	`build/pkgs/slackware.txt: Add libxml2`
`f7493d9`	`build/pkgs/libgd/distros/slackware.txt: Add dependencies`
`288d760`	`Merge branch 't/29424/slackware__system_packages_fix' into t/29341/optional-test-9.1.rc0`

Answer 16 · 2020-04-18T18:48:18.000Z

Changed dependencies from #29417, #29426 to #29417, #29424, #29426, #29501, #29508

Answer 17 · 2020-04-18T18:55:28.000Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`251ab47`	`trac 29492: remove outdated boilerplate from spkg-install.in,`
`1383a59`	`Merge branch 'u/jhpalmieri/remove-boilerplate' of git://trac.sagemath.org/sage into t/29510/spkg_check_in__remove_messages_that_duplicate_sage_spkg_s_messages`
`2533a67`	`build/pkgs/*/spkg-check.in: Remove messages that duplicate sage-spkg messages`
`9b213a4`	`build/bin/sage-spkg: Add message when test suite passed`
`ca36a87`	`build/pkgs/*/spkg-check.in: Do not print misleading error messages: avoid sdh_make`
`c53ddc7`	`Use 'failures', not 'error' when reporting testsuite failures`
`6140a99`	`Merge branch 't/29501/spkg_check_in__do_not_use_sdh_make_to_do__make_check_' into t/29341/optional-test-9.1.rc0`
`5fe6538`	`Merge branch 't/29424/slackware__system_packages_fix' into t/29341/optional-test-9.1.rc0`

Answer 18 · 2020-04-18T18:55:28.000Z

Changed commit from 288d760 to 5fe6538

Answer 19 · 2020-04-18T18:56:05.000Z

Changed dependencies from #29417, #29424, #29426, #29501, #29508 to #29417, #29424, #29426, #29501, #29508, #29510

Answer 20 · 2020-04-18T18:57:40.000Z

Description changed:

--- 
+++ 
@@ -1 +1,2 @@
+We add a new [GitHub](../wiki/GitHub) Actions CI workflow `tox-optional.yml`, which tests most optional packages, using the `maximal` packages factor.

Answer 21 · 2020-04-18T19:11:39.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`84edb00`	`.github/workflows/tox-optional.yml: Rename`

Answer 22 · 2020-04-18T19:11:39.000Z

Changed commit from 5fe6538 to 84edb00

Answer 23 · 2020-04-18T19:12:50.000Z

comment:17

Tests running at https://github.com/mkoeppe/sage/actions/runs/81669565

Answer 24 · 2020-04-18T19:21:58.000Z

Author: Matthias Koeppe

Answer 25 · 2020-04-19T03:17:48.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`7b3446f`	`.github/workflows/tox-optional.yml: Test packages factor 'maximal'`

Answer 26 · 2020-04-19T03:17:48.000Z

Changed commit from 84edb00 to 7b3446f

Answer 27 · 2020-04-19T05:21:03.000Z

Changed commit from 7b3446f to 1dda317

Answer 28 · 2020-04-19T05:21:03.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`1dda317`	`Add cmake to packages not tested`

Answer 29 · 2020-04-19T18:13:21.000Z

comment:22

On Fedora, this breaks installation of python3 because yum info python3 does not work even though yum install python3 works

(https://github.com/mkoeppe/sage/runs/599575888)

Answer 30 · 2020-04-19T18:17:47.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`bd8a2f9`	`build/bin/write-dockerfile.sh [fedora]: Use 'yum install -y --download-only' for checking existence of packages`

Answer 31 · 2020-04-19T18:17:47.000Z

Changed commit from 1dda317 to bd8a2f9

Answer 32 · 2020-04-20T07:24:41.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`807c2bd`	`Fix spelling of --downloadonly`

Answer 33 · 2020-04-20T07:24:41.000Z

Changed commit from bd8a2f9 to 807c2bd

Answer 34 · 2020-04-20T23:43:38.000Z

Changed dependencies from #29417, #29424, #29426, #29501, #29508, #29510 to #29417, #29424, #29426, #29501, #29508, #29510, #29403

Answer 35 · 2020-04-20T23:44:15.000Z

Changed commit from 807c2bd to 784abe8

Answer 36 · 2020-04-20T23:44:15.000Z

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`fc61002`	`ci-cygwin.yml: Package up SAGE_LOCAL/var/tmp/sage/build to avoid error with actions/upload-artifact`
`59ec460`	`./configure --enable-experimental-packages --enable-download-from-upstream-url`
`c9cf3cc`	`Merge branch 't/29392/__configure___enable_experimental_packages___enable_download_from_upstream_url' into t/29403/ci_cygwin_yml__multi_stage_build`
`037f508`	`tox.yml: Pass CONFIGURE_ARGS: --enable-experimental-packages --enable-download-from-upstream-url`
`998f02d`	`ci-cygwin.yml: Fixup for: Package up SAGE_LOCAL/var/tmp/sage/build`
`d6bf4b1`	`ci-cygwin.yml: Install standard system packages also for 'stndard-python2'`
`2779e24`	`ci-cygwin.yml: Another fixup for: Package up SAGE_LOCAL/var/tmp/sage/build`
`27ddea3`	`build/bin/sage-logger: Back out an experimental change`
`bd687c9`	`Merge branch 't/29403/ci_cygwin_yml__multi_stage_build' into t/29341/optional-test-9.1.rc0`
`784abe8`	`ci-cygwin.yml: SAGE_FAT_BINARY=yes`

Answer 37 · 2020-04-20T23:49:05.000Z

Changed commit from 784abe8 to eeadb9d

Answer 38 · 2020-04-20T23:49:05.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`eeadb9d`	`.github/workflows/ci-cygwin.yml: Test more optional packages`

Answer 39 · 2020-04-21T00:03:42.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`316c5f6`	`Fixup`

Answer 40 · 2020-04-21T00:03:42.000Z

Changed commit from eeadb9d to 316c5f6

Answer 41 · 2020-04-22T15:36:47.000Z

Reviewer: Dima Pasechnik

Answer 42 · 2020-04-22T15:36:47.000Z

comment:30

OK, this appears to work, by and large. Surely there are various platforms/packages combos not working, but this is another story.

Answer 43 · 2020-04-22T23:25:04.000Z

comment:31

Thanks!

Answer 44 · 2020-04-23T03:04:12.000Z

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

`816477a`	`Merge tag '9.1.rc1' into t/29341/optional-test-9.1.rc0`

Answer 45 · 2020-04-23T03:04:12.000Z

Changed commit from 316c5f6 to 816477a

Answer 46 · 2020-04-23T03:04:40.000Z

comment:33

Merged in new rc

Answer 47 · 2020-04-24T22:28:14.000Z

Changed branch from u/mkoeppe/github_ci_workflow__add_testing_of_optional_packages to 816477a