GH Actions: Create daily integration branch

Question

GH Actions: Create daily integration branch

mkoeppe opened this issue 4 years ago · 110 comments

We create a GH Actions workflow, run daily (intended for running on sagetrac-mirror) as a cron job (
https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#schedule).

It can also be triggered manually via workflow_dispatch.

Example run at
https://github.com/mkoeppe/sage/runs/5060070979

Merge mergeable Trac tickets from the current milestone, ordered by priority/deps, using git releasemgr merge-all --milestone=current
Run using tox/docker on a stable platform (ubuntu-focal-recommended). (The new package factor recommended is like standard, but with texlive etc. added, as needed for doc-pdf). We set SAGE_DOCTEST_RANDOM_SEED=0 to reduce random failures.
Only testsuite failures that do not already fail in the baseline are considered failures.
Docker images are pushed to ghcr.io, see https://github.com/mkoeppe?tab=packages&q=recommended

Related issues and PRs:

Refinements:

Provide it with Trac credentials so it can set tickets back to needs_work on merge failures (using sagemath/git-trac-command#57)

--> Trac account requested
Run git log --first-parent on files with merge conflict
Provide it with GitHub credentials so it can push a branch $RELEASE_TAG+integration-$DATE, if it passes build + doctests + doc-pdf.
Perhaps git bisect --first-parent for passing; set first failing ticket back to needs_work
Another workflow for integration testing of all tickets in the current milestone with --patchbot-status=Spkg and --status=needs_review positive_review with the full portability CI.

Depends on #33233
Depends on #31529
Depends on #33103
Depends on #33277

CC: @vbraun @dimpase

Component: distribution

Author: Matthias Koeppe

Branch/Commit: u/mkoeppe/gh_actions__create_daily_integration_branch @ 37c76a6

Issue created by migration from https://trac.sagemath.org/ticket/33222

Answer 1 · 2022-01-23T20:05:16.000Z

comment:1

Volker, would you be willing to share your existing scripts that probably already implement something like step 2 above? It does not have to be pretty.

Answer 2 · 2022-01-23T20:16:14.000Z

comment:2

Scripts are part of https://github.com/sagemath/git-trac-command:

$ git releasemgr merge-all -h
usage: git-releasemgr merge-all [-h] [--limit LIMIT]

optional arguments:
  -h, --help     show this help message and exit
  --limit LIMIT  Merge this many tickets

Answer 3 · 2022-01-23T20:25:45.000Z

comment:3

Thanks for the pointer

Answer 4 · 2022-01-23T21:31:57.000Z

Description changed:

--- 
+++ 
@@ -6,6 +6,8 @@
 2. Pick mergeable Trac tickets from the current milestone, order by priority/deps
 
 3.  Merge one by one and push to branch "$RELEASE_TAG+integration-$DATE", if it passes doctests + doc-pdf.
-   - Report failures as comment / status change on the Trac ticket.
+   - Report failures as comment / status change on the Trac ticket. Link to the GH Actions run: `$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID` (from https://docs.github.com/en/actions/learn-github-actions/environment-variables)
 
-
+Related PRs:
+- https://github.com/sagemath/git-trac-command/pull/53
+- https://github.com/sagemath/git-trac-command/pull/54

Answer 5 · 2022-01-23T23:28:37.000Z

Description changed:

--- 
+++ 
@@ -11,3 +11,7 @@
 Related PRs:
 - https://github.com/sagemath/git-trac-command/pull/53
 - https://github.com/sagemath/git-trac-command/pull/54
+- https://github.com/sagemath/git-trac-command/pull/55
+- https://github.com/sagemath/git-trac-command/pull/56
+- https://github.com/sagemath/git-trac-command/pull/57
+

Answer 6 · 2022-01-23T23:34:35.000Z

Description changed:

--- 
+++ 
@@ -3,7 +3,7 @@
 
 1. Start from current release tag, use the corresponding Docker image on gchr.io from a stable platform such as `ubuntu-focal-standard` 
 
-2. Pick mergeable Trac tickets from the current milestone, order by priority/deps
+2. Merge mergeable Trac tickets from the current milestone, ordered by priority/deps, using `git releasemgr merge-all --milestone=MILESTONE`
 
 3.  Merge one by one and push to branch "$RELEASE_TAG+integration-$DATE", if it passes doctests + doc-pdf.
    - Report failures as comment / status change on the Trac ticket. Link to the GH Actions run: `$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID` (from https://docs.github.com/en/actions/learn-github-actions/environment-variables)

Answer 7 · 2022-01-24T01:21:41.000Z

Dependencies: #30933

Answer 8 · 2022-01-24T02:41:10.000Z

Description changed:

--- 
+++ 
@@ -14,4 +14,5 @@
 - https://github.com/sagemath/git-trac-command/pull/55
 - https://github.com/sagemath/git-trac-command/pull/56
 - https://github.com/sagemath/git-trac-command/pull/57
+- https://github.com/sagemath/sage-patchbot/pull/150

Answer 9 · 2022-01-24T02:50:46.000Z

Description changed:

--- 
+++ 
@@ -8,7 +8,9 @@
 3.  Merge one by one and push to branch "$RELEASE_TAG+integration-$DATE", if it passes doctests + doc-pdf.
    - Report failures as comment / status change on the Trac ticket. Link to the GH Actions run: `$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID` (from https://docs.github.com/en/actions/learn-github-actions/environment-variables)
 
-Related PRs:
+Related issues and PRs:
+- https://github.com/sagemath/sage-patchbot/issues/148
+- https://github.com/sagemath/sage-patchbot/issues/64
 - https://github.com/sagemath/git-trac-command/pull/53
 - https://github.com/sagemath/git-trac-command/pull/54
 - https://github.com/sagemath/git-trac-command/pull/55

Answer 10 · 2022-01-24T03:47:10.000Z

Description changed:

--- 
+++ 
@@ -17,4 +17,5 @@
 - https://github.com/sagemath/git-trac-command/pull/56
 - https://github.com/sagemath/git-trac-command/pull/57
 - https://github.com/sagemath/sage-patchbot/pull/150
+- https://github.com/sagemath/git-trac-command/pull/58

Answer 11 · 2022-01-24T23:50:59.000Z

Description changed:

--- 
+++ 
@@ -18,4 +18,5 @@
 - https://github.com/sagemath/git-trac-command/pull/57
 - https://github.com/sagemath/sage-patchbot/pull/150
 - https://github.com/sagemath/git-trac-command/pull/58
+- https://github.com/sagemath/sage-patchbot/pull/153

Answer 12 · 2022-01-25T07:44:49.000Z

Branch: u/mkoeppe/gh_actions__create_daily_integration_branch

Answer 13 · 2022-01-25T17:38:28.000Z

Commit: e803539

Answer 14 · 2022-01-25T17:38:28.000Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`e803539`	`.github/workflows/integration.yml: New`

Answer 15 · 2022-01-25T17:55:27.000Z

Description changed:

--- 
+++ 
@@ -1,12 +1,14 @@
-We create a GH Actions workflow, run daily on `sagetrac-mirror` (
-https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#schedule):
+We create a GH Actions workflow, run daily (intended for running on `sagetrac-mirror`) as a cron job (
+https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#schedule).
 
-1. Start from current release tag, use the corresponding Docker image on gchr.io from a stable platform such as `ubuntu-focal-standard` 
+It can also be triggered manually via `workflow_dispatch`.
 
-2. Merge mergeable Trac tickets from the current milestone, ordered by priority/deps, using `git releasemgr merge-all --milestone=MILESTONE`
+Example run at 
+https://github.com/mkoeppe/sage/runs/4940850295
 
-3.  Merge one by one and push to branch "$RELEASE_TAG+integration-$DATE", if it passes doctests + doc-pdf.
-   - Report failures as comment / status change on the Trac ticket. Link to the GH Actions run: `$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID` (from https://docs.github.com/en/actions/learn-github-actions/environment-variables)
+1. Merge mergeable Trac tickets from the current milestone, ordered by priority/deps, using `git releasemgr merge-all --milestone=current`
+
+2. Run using tox/docker on a stable platform (`ubuntu-focal-standard`)
 
 Related issues and PRs:
 - https://github.com/sagemath/sage-patchbot/issues/148
@@ -20,3 +22,15 @@
 - https://github.com/sagemath/git-trac-command/pull/58
 - https://github.com/sagemath/sage-patchbot/pull/153
 
+Refinements:
+- Provide it with Trac credentials so it can set tickets back to `needs_work` on merge failures (using https://github.com/sagemath/git-trac-command/pull/57)
+
+- Link to the GH Actions run: `$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID` (from https://docs.github.com/en/actions/learn-github-actions/environment-variables)
+
+- Provide it with [GitHub](../wiki/GitHub) credentials so it can push a branch `$RELEASE_TAG+integration-$DATE`, if it passes build + doctests + doc-pdf.
+
+- Perhaps `git bisect --first-parent` for passing; set first failing ticket back to `needs_review`
+
+- Warmstart from a Docker image on gchr.io (#30933) 
+
+

Answer 16 · 2022-01-25T17:55:27.000Z

Author: Matthias Koeppe

Answer 17 · 2022-01-26T05:24:44.000Z

Description changed:

--- 
+++ 
@@ -29,7 +29,7 @@
 
 - Provide it with [GitHub](../wiki/GitHub) credentials so it can push a branch `$RELEASE_TAG+integration-$DATE`, if it passes build + doctests + doc-pdf.
 
-- Perhaps `git bisect --first-parent` for passing; set first failing ticket back to `needs_review`
+- Perhaps `git bisect --first-parent` for passing; set first failing ticket back to `needs_work`
 
 - Warmstart from a Docker image on gchr.io (#30933)

Answer 18 · 2022-01-26T17:19:44.000Z

Description changed:

--- 
+++ 
@@ -8,7 +8,7 @@
 
 1. Merge mergeable Trac tickets from the current milestone, ordered by priority/deps, using `git releasemgr merge-all --milestone=current`
 
-2. Run using tox/docker on a stable platform (`ubuntu-focal-standard`)
+2. Run using tox/docker on a stable platform (`ubuntu-focal-standard`). We set `SAGE_DOCTEST_RANDOM_SEED=0` to reduce random failures.
 
 Related issues and PRs:
 - https://github.com/sagemath/sage-patchbot/issues/148

Answer 19 · 2022-01-26T19:46:42.000Z

Changed dependencies from #30933 to #30933, #33233

Answer 20 · 2022-01-26T20:08:16.000Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`7db353e`	`.github/workflows/integration.yml: New`
`6d30ef9`	`src/sage/tests/cmdline.py: Add diagnostics to a test that often fails`
`73450a8`	`src/bin/sage-runtests: New option --baseline-stats-path`
`3551e19`	`src/sage/doctest/control.py: Load base line stats`
`09ce9e0`	`src/sage/doctest/reporting.py: No error status for failures already seen in baseline`
`3eecc40`	`Merge #33233`
`edecb85`	`.github/workflows/integration.yml: Check out from current repo, use sagetrac-mirror remote for develop and tags`
`c452317`	`.github/workflows/integration.yml: Check out from current repo, use sagetrac-mirror remote for develop and tags`

Answer 21 · 2022-01-26T20:08:16.000Z

Changed commit from e803539 to c452317

Answer 22 · 2022-01-27T05:21:46.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`1903e03`	`.github/workflows/integration.yml: First build baseline, then test integration head`
`45af154`	`.github/workflows/integration.yml: Make git available in the container`
`e4c3890`	`.github/workflows/integration.yml: Add .git`
`7513533`	`.github/workflows/integration.yml: Actually build baseline`

Answer 23 · 2022-01-27T05:21:46.000Z

Changed commit from c452317 to 7513533

Answer 24 · 2022-01-27T06:25:41.000Z

Changed dependencies from #30933, #33233 to #30933, #33233, #31529

Answer 25 · 2022-01-27T06:27:00.000Z

Changed commit from 7513533 to ec1f7c1

Answer 26 · 2022-01-27T06:27:00.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`96771be`	`.github/workflows/integration.yml: Run 'git reset --hard' in the container so that non-ADDed files are not shown as deleted`
`9ddbcd1`	`.github/workflows/integration.yml: Do not fail when baseline ptest has failures`
`5d35b97`	`WIP`
`9dceb34`	`fix M4 bugs`
`65d67d1`	`build/pkgs/texlive/distros: Add *.txt`
`b09d0c7`	`Merge #31529`
`ec1f7c1`	`.github/workflows/integration.yml: Install texlive`

Answer 27 · 2022-01-27T17:15:21.000Z

Changed commit from ec1f7c1 to 51e47b3

Answer 28 · 2022-01-27T17:15:21.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`5733f4e`	`build/pkgs/texlive/distros/debian.txt: Move texlive system packages here from build/pkgs/_recommended/distros/debian.txt`
`79852ce`	`src/doc/bootstrap: Add texlive to RECOMMENDED_SYSTEM_PACKAGES`
`51e47b3`	`Merge #31529`

Answer 29 · 2022-01-28T01:41:30.000Z

Changed dependencies from #30933, #33233, #31529 to #30933, #33233, #31529, #30933

Answer 30 · 2022-01-28T05:36:46.000Z

Changed commit from 51e47b3 to 1c4a7ea

Answer 31 · 2022-01-28T05:36:46.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`b1c705b`	`build/pkgs/texlive/distros/debian.txt: Add more`
`a2a014c`	`Merge #31529`
`21afe51`	`.github/workflows/tox*.yml, tox.ini: Pass ddocker.pkg.github.com credentials via DOCKER_CONFIG_FILE`
`0f2a49a`	`Merge #30933`
`87b93f7`	`.github/workflows/integration.yml: Publish Docker images`
`d17b995`	`.github/workflows/integration.yml: Put git-trac-command outside of Sage tree`
`f7c4f2b`	`.github/workflows/integration.yml, tox.ini: Use new packages factor 'recommended'`
`bd0a101`	`.github/workflows/integration.yml: Also push images for stages 'configured' and 'with-targets-optional'`
`f4fca0e`	`.github/workflows/integration.yml: Fixup package name`
`1c4a7ea`	`.github/workflows/integration.yml: Include action URL in Trac comments via git releasemgr --trac-context`

Answer 32 · 2022-01-28T07:02:24.000Z

Description changed:

--- 
+++ 
@@ -8,7 +8,9 @@
 
 1. Merge mergeable Trac tickets from the current milestone, ordered by priority/deps, using `git releasemgr merge-all --milestone=current`
 
-2. Run using tox/docker on a stable platform (`ubuntu-focal-standard`). We set `SAGE_DOCTEST_RANDOM_SEED=0` to reduce random failures.
+2. Run using tox/docker on a stable platform (`ubuntu-focal-recommended`). (The new package factor `recommended` is like `standard`, but with texlive etc. added, as needed for `doc-pdf`). We set `SAGE_DOCTEST_RANDOM_SEED=0` to reduce random failures.
+
+3. Docker images are pushed to `gchr.io` 
 
 Related issues and PRs:
 - https://github.com/sagemath/sage-patchbot/issues/148

Answer 33 · 2022-01-28T07:02:46.000Z

Description changed:

--- 
+++ 
@@ -10,7 +10,7 @@
 
 2. Run using tox/docker on a stable platform (`ubuntu-focal-recommended`). (The new package factor `recommended` is like `standard`, but with texlive etc. added, as needed for `doc-pdf`). We set `SAGE_DOCTEST_RANDOM_SEED=0` to reduce random failures.
 
-3. Docker images are pushed to `gchr.io` 
+3. Docker images are pushed to `ghcr.io` 
 
 Related issues and PRs:
 - https://github.com/sagemath/sage-patchbot/issues/148

Answer 34 · 2022-01-28T07:33:13.000Z

Description changed:

--- 
+++ 
@@ -10,7 +10,8 @@
 
 2. Run using tox/docker on a stable platform (`ubuntu-focal-recommended`). (The new package factor `recommended` is like `standard`, but with texlive etc. added, as needed for `doc-pdf`). We set `SAGE_DOCTEST_RANDOM_SEED=0` to reduce random failures.
 
-3. Docker images are pushed to `ghcr.io` 
+3. Docker images are pushed to `ghcr.io`, see https://github.com/mkoeppe?tab=packages&q=recommended
+
 
 Related issues and PRs:
 - https://github.com/sagemath/sage-patchbot/issues/148

Answer 35 · 2022-01-28T18:34:54.000Z

Changed commit from 1c4a7ea to 7447909

Answer 36 · 2022-01-28T18:34:54.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`7e317b2`	`build/pkgs/texlive/distros/debian.txt: Add texlive-fonts-recommended`
`37d569d`	`build/pkgs/texlive/distros/debian.txt: Add texlive-lang-*`
`7447909`	`Merge #31529`

Answer 37 · 2022-01-28T20:53:10.000Z

Changed dependencies from #30933, #33233, #31529, #30933 to #30933, #33233, #31529, #30933, #33243

Answer 38 · 2022-01-28T20:56:56.000Z

Changed commit from 7447909 to d2bda36

Answer 39 · 2022-01-28T20:56:56.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`22feeb5`	`build/make/Makefile.in (PLUS): Make it work in GNU make 4.x`
`d2bda36`	`Merge #33243`

Answer 40 · 2022-01-29T01:06:56.000Z

Description changed:

--- 
+++ 
@@ -24,6 +24,8 @@
 - https://github.com/sagemath/sage-patchbot/pull/150
 - https://github.com/sagemath/git-trac-command/pull/58
 - https://github.com/sagemath/sage-patchbot/pull/153
+- https://github.com/sagemath/git-trac-command/pull/59
+- https://github.com/sagemath/git-trac-command/pull/60
 
 Refinements:
 - Provide it with Trac credentials so it can set tickets back to `needs_work` on merge failures (using https://github.com/sagemath/git-trac-command/pull/57)

Answer 41 · 2022-01-29T01:09:37.000Z

Changed commit from d2bda36 to 3ae35d6

Answer 42 · 2022-01-29T01:09:37.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`3ae35d6`	`.github/workflows/integration.yml: Create integration branch`

Answer 43 · 2022-01-29T01:10:20.000Z

Description changed:

--- 
+++ 
@@ -36,6 +36,6 @@
 
 - Perhaps `git bisect --first-parent` for passing; set first failing ticket back to `needs_work`
 
-- Warmstart from a Docker image on gchr.io (#30933) 
+- Warmstart from a Docker image on ghcr.io (#30933)

Answer 44 · 2022-01-29T22:14:25.000Z

Changed commit from 3ae35d6 to ca95c24

Answer 45 · 2022-01-29T22:14:25.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`ca95c24`	`.github/workflows/integration.yml: First run a job 'baseline_build', use cache`

Answer 46 · 2022-01-29T22:15:16.000Z

comment:35

https://github.com/mkoeppe/sage/runs/4993397402?check_suite_focus=true

#5 importing cache manifest from docker.pkg.github.com/mkoeppe/sage/sage-docker-ubuntu-focal-recommended-with-targets
#5 sha256:c3f8749b6884f89e52d82a7beb83dd3938671c5e8c91d13592abcfa40146f0ad
#5 ERROR: unexpected status code [manifests latest]: 401 Unauthorized

This may be the same as docker/buildx#780

Answer 47 · 2022-01-29T22:28:57.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`26d9b8d`	`.github/workflows/integration.yml: Switch from docker.pkg.github.com to ghcr.io`

Answer 48 · 2022-01-29T22:28:57.000Z

Changed commit from ca95c24 to 26d9b8d

Answer 49 · 2022-01-29T22:46:00.000Z

Changed dependencies from #30933, #33233, #31529, #30933, #33243 to #30933, #33233, #31529, #30933, #33243, #33103

Answer 50 · 2022-01-30T02:44:52.000Z

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`6285cd9`	`.gitpod.Dockerfile: Save space by removing apt lists`
`7e74607`	`.gitpod.Dockerfile: Remove built Sage documentation`
`f71c8b6`	`docker/.gitpod.Dockerfile: Clean more`
`f953228`	`Merge tag '9.5.rc0' into t/33103/gitpod_integration_using_docker_images_from_portability_testing_workflow`
`2d0902a`	`src/doc/en/developer/workspace.rst: Fix markup`
`078f247`	`.github/workflows/tox*.yml: Transform GITHUB_REF_NAME to a valid docker tag`
`b095b47`	`Merge #33103`
`036df6d`	`tox.ini: Use --cache-from if DOCKER_PUSH_REPOSITORY is given`
`de2a455`	`.github/workflows/integration.yml: Pass BUILDKIT_INLINE_CACHE=1 as a build-arg`
`898acf5`	`.github/workflows/integration.yml: Reactivate full baseline build`

Answer 51 · 2022-01-30T02:44:52.000Z

Changed commit from 26d9b8d to 898acf5

Answer 52 · 2022-01-30T04:05:00.000Z

Description changed:

--- 
+++ 
@@ -30,6 +30,8 @@
 Refinements:
 - Provide it with Trac credentials so it can set tickets back to `needs_work` on merge failures (using https://github.com/sagemath/git-trac-command/pull/57)
 
+  - Trac account requested
+
 - Link to the GH Actions run: `$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID` (from https://docs.github.com/en/actions/learn-github-actions/environment-variables)
 
 - Provide it with [GitHub](../wiki/GitHub) credentials so it can push a branch `$RELEASE_TAG+integration-$DATE`, if it passes build + doctests + doc-pdf.

Answer 53 · 2022-01-30T04:16:23.000Z

Description changed:

--- 
+++ 
@@ -30,7 +30,7 @@
 Refinements:
 - Provide it with Trac credentials so it can set tickets back to `needs_work` on merge failures (using https://github.com/sagemath/git-trac-command/pull/57)
 
-  - Trac account requested
+  --> Trac account requested
 
 - Link to the GH Actions run: `$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID` (from https://docs.github.com/en/actions/learn-github-actions/environment-variables)
 
@@ -38,6 +38,6 @@
 
 - Perhaps `git bisect --first-parent` for passing; set first failing ticket back to `needs_work`
 
-- Warmstart from a Docker image on ghcr.io (#30933) 
+- Use several `--cache-from`, including current `develop`

Answer 54 · 2022-01-30T07:44:55.000Z

Changed commit from 898acf5 to a545c4e

Answer 55 · 2022-01-30T07:44:55.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`a545c4e`	`.github/workflows/integration.yml: Fixup`

Answer 56 · 2022-01-30T17:18:11.000Z

Changed commit from a545c4e to 627c65c

Answer 57 · 2022-01-30T17:18:11.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`521b12f`	`.github/workflows/integration.yml: Actually run ptest`
`627c65c`	`.github/workflows/integration.yml: Use ptestlong`

Answer 58 · 2022-01-30T17:18:36.000Z

Changed commit from 627c65c to 72eff23

Answer 59 · 2022-01-30T17:18:36.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`72eff23`	`Merge tag '9.5' into t/33222/gh_actions__create_daily_integration_branch`

Answer 60 · 2022-01-30T18:07:26.000Z

Changed dependencies from #30933, #33233, #31529, #30933, #33243, #33103 to #33233, #31529, #33103

Answer 61 · 2022-01-30T18:07:46.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`10259f2`	`.github/workflows, src/doc/en/developer/portability_testing.rst: Switch from docker.pkg.github.com to ghcr.io`
`47c4723`	`.github/workflows/integration.yml: Set EXTRA_DOCKER_TAGS`
`b143b95`	`tox.ini: Also --cache-from ...:dev`

Answer 62 · 2022-01-30T18:07:46.000Z

Changed commit from 72eff23 to b143b95

Answer 63 · 2022-01-30T19:32:37.000Z

Description changed:

--- 
+++ 
@@ -22,10 +22,12 @@
 - https://github.com/sagemath/git-trac-command/pull/56
 - https://github.com/sagemath/git-trac-command/pull/57
 - https://github.com/sagemath/sage-patchbot/pull/150
-- https://github.com/sagemath/git-trac-command/pull/58
 - https://github.com/sagemath/sage-patchbot/pull/153
 - https://github.com/sagemath/git-trac-command/pull/59
 - https://github.com/sagemath/git-trac-command/pull/60
+- https://github.com/sagemath/git-trac-command/pull/61
+- https://github.com/sagemath/git-trac-command/pull/62
+- https://github.com/sagemath/git-trac-command/pull/58
 
 Refinements:
 - Provide it with Trac credentials so it can set tickets back to `needs_work` on merge failures (using https://github.com/sagemath/git-trac-command/pull/57)

Answer 64 · 2022-01-30T19:33:15.000Z

Description changed:

--- 
+++ 
@@ -40,6 +40,4 @@
 
 - Perhaps `git bisect --first-parent` for passing; set first failing ticket back to `needs_work`
 
-- Use several `--cache-from`, including current `develop`
 
-

Answer 65 · 2022-01-30T21:59:55.000Z

Description changed:

--- 
+++ 
@@ -4,11 +4,12 @@
 It can also be triggered manually via `workflow_dispatch`.
 
 Example run at 
-https://github.com/mkoeppe/sage/runs/4940850295
+https://github.com/mkoeppe/sage/runs/4999004937
 
 1. Merge mergeable Trac tickets from the current milestone, ordered by priority/deps, using `git releasemgr merge-all --milestone=current`
 
 2. Run using tox/docker on a stable platform (`ubuntu-focal-recommended`). (The new package factor `recommended` is like `standard`, but with texlive etc. added, as needed for `doc-pdf`). We set `SAGE_DOCTEST_RANDOM_SEED=0` to reduce random failures.
+Only testsuite failures that do not already fail in the baseline are considered failures. 
 
 3. Docker images are pushed to `ghcr.io`, see https://github.com/mkoeppe?tab=packages&q=recommended
 
@@ -34,7 +35,7 @@
 
   --> Trac account requested
 
-- Link to the GH Actions run: `$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID` (from https://docs.github.com/en/actions/learn-github-actions/environment-variables)
+- Run `git log --first-parent` on files with merge conflict
 
 - Provide it with [GitHub](../wiki/GitHub) credentials so it can push a branch `$RELEASE_TAG+integration-$DATE`, if it passes build + doctests + doc-pdf.

Answer 66 · 2022-01-31T00:57:17.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`1dc511c`	`Add Arch package info`
`a505487`	`build/pkgs/texlive/distros/fedora.txt: Add texlive-collection-latexextra`
`7daedd9`	`build/pkgs/texlive/distros/fedora.txt: Add texlive-collection-lang*`
`6af8abb`	`Merge #31529`

Answer 67 · 2022-01-31T00:57:17.000Z

Changed commit from b143b95 to 6af8abb

Answer 68 · 2022-01-31T06:25:55.000Z

Changed commit from 6af8abb to ff8a8d6

Answer 69 · 2022-01-31T06:25:55.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`ff8a8d6`	`build/pkgs/ffmpeg/distros/fedora.txt: Remove ffmpeg - it is not in the standard repo`

Answer 70 · 2022-01-31T18:04:31.000Z

comment:51

With ptestlong it exceeds the 6 hours, so I'll have to split out another job

Answer 71 · 2022-01-31T19:48:03.000Z

Description changed:

--- 
+++ 
@@ -41,4 +41,5 @@
 
 - Perhaps `git bisect --first-parent` for passing; set first failing ticket back to `needs_work`
 
+- Another workflow for integration testing of all tickets in the current milestone with `--patchbot-status=Spkg` and `--status=needs_review positive_review` with the full portability CI.

Answer 72 · 2022-02-01T00:48:59.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`5117e92`	`.github/workflows/integration.yml: Separate job baseline_test`

Answer 73 · 2022-02-01T00:48:59.000Z

Changed commit from ff8a8d6 to 5117e92

Answer 74 · 2022-02-01T16:13:15.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`9647521`	`.github/workflows/integration.yml: Fixup`
`921ea0f`	`.github/workflows/integration.yml: Fixup`

Answer 75 · 2022-02-01T16:13:15.000Z

Changed commit from 5117e92 to 921ea0f

Answer 76 · 2022-02-02T19:10:53.000Z

Changed dependencies from #33233, #31529, #33103 to #33233, #31529, #33103, #33277

Answer 77 · 2022-02-02T19:24:27.000Z

Changed commit from 921ea0f to 5de1724

Answer 78 · 2022-02-02T19:24:27.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`6952de4`	`build/pkgs/texlive/spkg-configure.m4: Check for pdflatex, latexmk, dvipng`
`b85356f`	`build/pkgs/texlive/spkg-configure.m4: Check for tgtermes.sty`
`62fbc45`	`build/pkgs/sagemath_doc_pdf/dependencies: Add texlive`
`e63b9f2`	`Merge tag '9.5' into t/31529/packages/texlive/add_spkg_configure_m4_and_system_package_information__remove_install_script`
`8b2490e`	`build/pkgs/texlive/spkg-configure.m4: ALso check for fncychap.sty`
`7b97dd4`	`update list of gentoo packages`
`016ff89`	`Merge #31529`
`855314d`	`src/doc/bootstrap: Get recommended spkg from build/pkgs/_recommended/dependencies`
`0097d0e`	`Merge #33277`
`5de1724`	`tox.ini (recommended): Use build/pkgs/_recommended/dependencies`

Answer 79 · 2022-02-03T17:46:52.000Z

Branch pushed to git repo; I updated commit sha1. New commits:

`79a3905`	`.github/workflows/integration.yml (baseline_test): Also fetch tags so that we refer to the right image tag`
`37c76a6`	`tox.ini (docker): --cache-from all docker_targets, EXTRA_CACHE_FROM_REPOSITORIES`

Answer 80 · 2022-02-03T17:46:52.000Z

Changed commit from 5de1724 to 37c76a6

Answer 81 · 2022-02-04T04:45:01.000Z

Description changed:

--- 
+++ 
@@ -4,7 +4,7 @@
 It can also be triggered manually via `workflow_dispatch`.
 
 Example run at 
-https://github.com/mkoeppe/sage/runs/4999004937
+https://github.com/mkoeppe/sage/runs/5060070979
 
 1. Merge mergeable Trac tickets from the current milestone, ordered by priority/deps, using `git releasemgr merge-all --milestone=current`

Answer 82 · 2022-02-04T22:55:44.000Z

comment:59

Uff that's quite complicated. Why do you first need to create a new baseline image for the integration tests and not reuse the docker image build during the latest push to the develop branch?

Instead of manually handling the docker pull (using cli commands) I would propose to use the container tag for the workflow (or job). Makes things easier and I assume github also takes care of the permission handling.

Also the integration job is currently timing out, and if I read the logs correctly then it is actually not merging the tickets for the new milestone (but tries to merge tickets for the already finished milestone 9.5): No tickets for milestone sage-9.5 are ready to be merged.

Finally a general remark: please use the new reusable actions instead of copy&pasting stuff around, like the "free space" or "login to ghcr" parts (which shouldn't be necessary in the final integration job because that's only reading, right?).

Answer 83 · 2022-02-04T23:23:32.000Z

comment:60

Replying to @tobiasdiez:

Finally a general remark: please use the new reusable actions instead of copy&pasting stuff around

Refactoring will not happen in this ticket

Answer 84 · 2022-02-04T23:31:23.000Z

comment:61

Replying to @tobiasdiez:

Why do you first need to create a new baseline image for the integration tests and not reuse the docker image build during the latest push to the develop branch?

I need an image with more packages, which is not built by the CI. This is to run make doc-pdf, an essential step that currently no CI is testing.

Instead of manually handling the docker pull (using cli commands) I would propose to use the container tag for the workflow (or job).

I am deliberately setting this up as an image build, not just a container run, because I want to create a Docker image that can be pulled for local inspection.

Answer 85 · 2022-02-04T23:33:43.000Z

comment:62

Replying to @tobiasdiez:

Also the integration job is currently timing out

Yes, there's still something going wrong with the caching, so it's rebuilding something that it should have taken from the previous job

Answer 86 · 2022-02-04T23:35:01.000Z

comment:63

Replying to @tobiasdiez:

"login to ghcr" parts [...] shouldn't be necessary in the final integration job because that's only reading, right?

Actually one does need to login even just for reading packages.

Answer 87 · 2022-02-04T23:36:35.000Z

comment:64

(And the last step does push something - the image with the included repository.)

Answer 88 · 2022-02-04T23:45:32.000Z

comment:65

Replying to @mkoeppe:

Replying to @tobiasdiez:

Finally a general remark: please use the new reusable actions instead of copy&pasting stuff around

Refactoring will not happen in this ticket

I didn't meant you should refactor the existing workflows. But if you want to reuse some steps in the newly created workflow than they should be extracted to a new action (instead of copy&paste them). Refactoring the other workflows to use this new action can be done later.
(You wouldn't also let python code pass that copy&pastes some methods around, right?)

Answer 89 · 2022-02-04T23:47:21.000Z

comment:66

Replying to @mkoeppe:

Replying to @tobiasdiez:

Why do you first need to create a new baseline image for the integration tests and not reuse the docker image build during the latest push to the develop branch?

I need an image with more packages, which is not built by the CI. This is to run make doc-pdf, an essential step that currently no CI is testing.

If it's so important, why not add it as a new target to the existing CI?

Answer 90 · 2022-02-04T23:57:51.000Z

comment:67

Replying to @mkoeppe:

I am deliberately setting this up as an image build, not just a container run, because I want to create a Docker image that can be pulled for local inspection.

Can you expand on why this should be helpful, i.e. where are the advantages over just checking out the branch in your local git (and maybe running it in the baseline-integration docker image if you like). Also you can just execute the github workflow locally using act if you like.

Also, are you aware that there are special actions for the login and publish, which make the whole process way easier (and we don't need to maintain them)? https://docs.github.com/en/actions/publishing-packages/publishing-docker-images#publishing-images-to-github-packages

Answer 91 · 2022-02-04T23:59:06.000Z

comment:68

Replying to @tobiasdiez:

Replying to @mkoeppe:

This is to run make doc-pdf, an essential step that currently no CI is testing.

If it's so important, why not add it as a new target to the existing CI?

Yes, changing the existing CI to build the -recommended instead of -standard workflow may be something that we want to do at some point in the future. But this will need #33062 (split the workflow, with caching of Docker layers). That's exactly what the changes to tox.ini here on the ticket are implementing!

Can't change everything at the same time.

Answer 92 · 2022-02-05T00:01:04.000Z

comment:69

Replying to @tobiasdiez:

Replying to @mkoeppe:

I am deliberately setting this up as an image build, not just a container run, because I want to create a Docker image that can be pulled for local inspection.

Can you expand on why this should be helpful, i.e. where are the advantages over just checking out the branch in your local git (and maybe running it in the baseline-integration docker image if you like).

The build & test can take long. If it is prebuilt on GH Actions and I can download it for inspection, that's a benefit.

Also you can just execute the github workflow locally using act if you like.

I don't like

Answer 93 · 2022-02-05T00:05:53.000Z

comment:70

Replying to @tobiasdiez:

Also, are you aware that there are special actions for the login and publish

Yes, but they don't help much. The login action could replace these 5 lines:

            TOKEN="${{ secrets.DOCKER_PKG_GITHUB_TOKEN }}"
            if [ -z "$TOKEN" ]; then
              TOKEN="${{ secrets.GITHUB_TOKEN }}"
            fi
            if echo "$TOKEN" | docker login ghcr.io -u ${{ github.actor }} --password-stdin; then

And the other one is not useful

Answer 94 · 2022-02-05T00:06:28.000Z

comment:71

Replying to @mkoeppe:

Replying to @tobiasdiez:

Replying to @mkoeppe:

This is to run make doc-pdf, an essential step that currently no CI is testing.

If it's so important, why not add it as a new target to the existing CI?

Yes, changing the existing CI to build the -recommended instead of -standard workflow may be something that we want to do at some point in the future. But this will need #33062 (split the workflow, with caching of Docker layers). That's exactly what the changes to tox.ini here on the ticket are implementing!

Can't change everything at the same time.

Okay, makes sense. Thanks for the explanation.

Answer 95 · 2022-02-05T00:09:23.000Z

comment:72

Replying to @mkoeppe:

Replying to @tobiasdiez:

Replying to @mkoeppe:

I am deliberately setting this up as an image build, not just a container run, because I want to create a Docker image that can be pulled for local inspection.

Can you expand on why this should be helpful, i.e. where are the advantages over just checking out the branch in your local git (and maybe running it in the baseline-integration docker image if you like).

The build & test can take long. If it is prebuilt on GH Actions and I can download it for inspection, that's a benefit.

But the push happens before the final "Build & test integration head", so it neither contains the test results nor the build, or I'm missing something?

Answer 96 · 2022-02-05T00:15:39.000Z

comment:73

Yes, that's true as it is in the current branch - but that final step can be extended to also build; just not done yet

Answer 97 · 2022-02-05T00:17:16.000Z

comment:74

Replying to @mkoeppe:

Replying to @tobiasdiez:

Also, are you aware that there are special actions for the login and publish

Yes, but they don't help much. The login action could replace these 5 lines:
            TOKEN="${{ secrets.DOCKER_PKG_GITHUB_TOKEN }}"
            if [ -z "$TOKEN" ]; then
              TOKEN="${{ secrets.GITHUB_TOKEN }}"
            fi
            if echo "$TOKEN" | docker login ghcr.io -u ${{ github.actor }} --password-stdin; then

Yes, the docker/login-action is not doing much more: https://github.com/docker/login-action/blob/17f28ab24d0d2832d5ff23a1409bbfc373ebcb96/src/docker.ts#L30-L50 Except a bit of proper error handling and logging you out again. Anyway, why not reuse something official so that you don't have to worry about it?

Answer 98 · 2022-02-05T00:24:20.000Z

comment:75

Because of the lines that follow if.

Answer 99 · 2022-02-05T00:26:52.000Z

comment:76

I guess it's getting late here for me, but your messages get more and more cryptic. There are a lot of ifs here...

Answer 100 · 2022-02-05T00:54:55.000Z

comment:77

Sorry, I meant the last if but it also applies to the first if ;)