/reproducing-actions

Insight into the reproducibility of GitHub Actions

Primary LanguageShellOtherNOASSERTION

Reproducing Actions

The Reproducing Actions project aims to provide insight into the reproducibility of GitHub Actions. At the end of this README you will find an about section with more information about the project and statuses.

Statuses

Action Version Status
actions/checkout@v4 v4
actions/setup-java@v4 v4
actions/setup-node@v4 v4
gitleaks/gitleaks-action@v2 v2
actions/stale@v9 v9
actions/github-script@v7 v7
actions/upload-artifact@v4 v4
actions/setup-python@v5 v5
actions/setup-dotnet@v4 v4
actions/download-artifact@v4 v4
actions/setup-go@v5 v5
actions/labeler@v5 v5
actions/dependency-review-action@v4 v4
actions/cache@v4 v4
JamesIves/github-pages-deploy-action@v4 v4
step-security/harden-runner@v2 v2
peter-evans/create-pull-request@v6 v6
ncipollo/release-action@v1 v1
docker/login-action@v3 v3
docker/build-push-action@v5 v5
github/codeql-action@v3 v3
dorny/paths-filter@v3 v3
actions/attest-build-provenance@v1 v1

About

Monitoring

To monitor the reproducibility of GitHub Actions we use GitHub Actions jobs that attempt to rebuild a given Action from its source code and see if that changes the build output. To see if a change occurred, SHA512 checksums are computed on the build output files stored in the repository and compared to SHA512 checksums computed after rebuilding the Action. If these don't match the Action is considered not reproducible.

We only monitor major versions of Actions. This avoids the need of having to commit changes whenever a new Action version is released, but risks incorrect results when build details change (more on this in the "Incorrect Results" section).

Understanding Statuses

The statuses table shows the reproducing state for all monitored Actions. If a status is passing it indicates that Action is currently reproducible and if a status is failing it indicates that Action is not currently reproducible. A failing status (notwithstanding false negatives) may occur when

  • the build is randomized in some way, or
  • the build output has not been updated with respect to the latest source code changes.

Actions are monitored on a daily basis, so there is a slight lag in the status of any given Action.

Incorrect Results

A passing status could be a false positive if:

  • The build process was changed but this has not yet been reflected in this repository (e.g. the files checked for reproducibility are reproducible but new build files are not).

A failing status could be a false negative if:

  • The build process has changed but this has not yet been reflected in this repository (e.g. the build output expected by this project is no longer used).
  • The CI job failed for some other reason (e.g. there was an unexpected error in the job setup).

Action Coverage

This project is geared towards monitoring any JavaScript-based Action with a build script at any supported version. Adding a monitor is a manual effort so not all available Actions are monitored. If you want to add a new monitor, open a new issue or follow the instructions from the Contributing Guidelines.

Any non-JavaScript Action or any JavaScript-based Action without a build step is currently considered out of scope. If you have a need for monitoring of such Actions please check out and comment on issue #1.

Reproducibility

An Action is considered reproducible if the build output files declared in the respective job in this project are byte-for-byte reproducible using the target project's dependencies and build command modulo Unix vs. Windows-style line endings.

Related Work