make reference accessible in reusable workflow

Question

make reference accessible in reusable workflow

Opened this issue 2 years ago · 32 comments

Describe the enhancement

A reusable workflow should be able to access the reference that it was called for.

Context

A reusable workflow in public repos may be called by appending a reference which can be a SHA, a release tag, or a branch name, as for example:
{owner}/{repo}/.github/workflows/{filename}@{ref}

Githubs documentation states:

When a reusable workflow is triggered by a caller workflow, the github context is always associated with the caller workflow.

The problem

Since the github context is always associated with the caller workflow, the reusable workflow cannot access the reference, for example the tag v1.0.0. However, knowing the reference is important when the reusable workflow needs to checkout the repository in order to make use of composite actions.

Code snippet

Assume that the caller workflow is being executed from within the main branch and calls the ref v1.0.0. of a reusable workflow:

name: Caller workflow
on:
  workflow_dispatch:

jobs:
  caller:
    uses: owner/public-repo/.github/workflows/reusable-workflow.yml@v1.0.0

Here is the reusable workflow that uses a composite action:

name: reusable workflows
on:
  workflow_call:

jobs:
  first-job:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3.1.0
        with:
          repository: owner/public-repo
          ref: ${{ github.ref_name }}

      - name: composite action
        uses: ./actions/my-composite-action

In the above code snippet, ${{ github.ref_name }} is main instead of v1.0.0 because github context is always associated with the caller workflow. Therefore, the composite actions code is based on main and not on v1.0.0.

Proposal

Introduce a new github context variable caller_ref which reflects the reference indicated by the caller.
The reusable workflow could then use it as follows:

name: reusable workflows
on:
  workflow_call:

jobs:
  first-job:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3.1.0
        with:
          repository: owner/public-repo
          ref: ${{ github.caller_ref }}

      - name: composite action
        uses: ./actions/my-composite-action

Or even shorter without the need to checkout:

name: reusable workflows
on:
  workflow_call:

jobs:
  first-job:
    runs-on: ubuntu-latest
    steps:
      - name: composite action
        uses: ./actions/my-composite-action@${{ github.caller_ref }}

Now, the composite action would be using v1.0.0. as indicated by the caller.

Related to this FR:

Answer 1 · 2023-01-27T15:05:30.000Z

I have the same issue, and would love some help!

Answer 2 · 2023-02-09T18:42:04.000Z

~~Ran into the same issue, here's a workaround: https://github.com/canonical/data-platform-workflows/blob/main/.github/workflows/_get_workflow_version.yaml~~

To solve this, we're now using this composite action (that implements @maxbergs's solution)

https://github.com/canonical/get-workflow-version-action

Example usage:

# Reusable workflow (e.g. build_charm.yaml)
on:
  workflow_call:

jobs:
  foo:
    runs-on: ubuntu-latest
    steps:
      - name: Get workflow version
        id: workflow-version
        uses: canonical/get-workflow-version-action@v1
        with:
          repository-name: canonical/data-platform-workflows
          file-name: build_charm.yaml
      # Use the version. For example:
      - name: Install Python CLI
        run: pipx install git+https://github.com/canonical/data-platform-workflows@'${{ steps.workflow-version.outputs.sha }}'#subdirectory=python/cli

Answer 3 · 2023-02-24T17:39:50.000Z

I am also interested in a solution for this. My use case is I have a repository within my org where I put shared workflows as well as composite actions. I often want to use composite actions within my reusable workflow in the same repository.

I see that you can do this with reusable workflows, but not yet composite actions: https://github.blog/changelog/2022-01-25-github-actions-reusable-workflows-can-be-referenced-locally/

mjhipp/shared-actions
├── actions
│   └── my-composite-action
│       └── action.yml
└── workflows
    └── my-reusable-workflow.yml

name: My Reusable Workflow
on:
  workflow_call:
jobs:
  my-reusable-workflow:
    runs-on: ubuntu-latest:
    steps:
      - uses: mjhipp/shared-actions/actions/my-composite-action@{???}

I would like to use whatever version I am using for the shared workflow for the called composite action, from the same repo as the called workflow.
I could use 'main', but I like to pin version tags in my actions for safety.

Answer 4 · 2023-05-02T19:18:12.000Z

I'd love this new reference to be created as well. I'm facing the same problem.

Answer 5 · 2023-05-11T22:16:08.000Z

Presumably the github.job_workflow_sha which is "For jobs using a reusable workflow, the commit SHA for the reusable workflow file" should solve this but it appears that this variable isn't ever populated: actions/runner#2417

Answer 6 · 2023-05-19T14:33:41.000Z

I am also interested in a solution for this. My use case is I have a repository within my org where I put shared workflows as well as composite actions. I often want to use composite actions within my reusable workflow in the same repository.

I see that you can do this with reusable workflows, but not yet composite actions: https://github.blog/changelog/2022-01-25-github-actions-reusable-workflows-can-be-referenced-locally/
mjhipp/shared-actions
├── actions
│   └── my-composite-action
│       └── action.yml
└── workflows
    └── my-reusable-workflow.yml
name: My Reusable Workflow
on:
  workflow_call:
jobs:
  my-reusable-workflow:
    runs-on: ubuntu-latest:
    steps:
      - uses: mjhipp/shared-actions/actions/my-composite-action@{???}
I would like to use whatever version I am using for the shared workflow for the called composite action, from the same repo as the called workflow. I could use 'main', but I like to pin version tags in my actions for safety.

I have the same use case and having this implemented would be great for pinning version between the reusable and the composite action

Answer 7 · 2023-07-17T12:29:54.000Z

Also having this need in my enterprise

Answer 8 · 2023-07-18T12:18:36.000Z

This is a basic feature need for using reusable workflows. We need a way to keep reusable workflows and the rest of the associated scripting in the same repository so we can version or do branch development. Please, provide a solution to access some context of the reused workflow, ideally access to local files and current branch/tag.

Answer 9 · 2023-08-02T02:28:58.000Z

Piling on. My org really needs this.

Answer 10 · 2023-08-28T14:33:43.000Z

I'm also interested in a solution for this.

My org has a central repo with our Reusable Workflows and Shared Action, and it would be essential for us to call the Shared Action using the same revision of the Reusable Workflow.

Answer 11 · 2023-09-14T08:15:57.000Z

+1 for this
For now I came up with workaround I'm sharing below.

So I have repository org/workflows with structure:

.github/workflows/shared.yml
actions/action1
actions/action2

.github/workflows/shared.yml is using action/action1 and action/action2 actions inside via e.g. uses: ./action/action.

Repository app calls shared workflow via uses: org/workflows/.github/workflows/shared.yml@v1.0.0.
So we need to know value v1.0.0 in shared workflow to properly checkout reference with local actions.

Here is how it is done in .github/workflows/shared.yml:

jobs:

  action1:
    name: Extract metadata
    runs-on: ubuntu-latest
    steps:

      - name: Checkout calling repository
        uses: actions/checkout@v4
        with:
          fetch-depth: 1

      - name: Get workflow reference
        id: workflows-ref
        run: |
          workflow_ref=${{ github.workflow_ref }}
          repository="${{ github.repository }}/"
          repo_ref="@${{ github.ref }}"
          workflow_path=${workflow_ref#"$repository"}
          workflow_path=${workflow_path%"$repo_ref"}

          ref=$(cat ${workflow_path} | grep pull-request-java-app | cut -d"@" -f2)
          echo "ref=${ref}" >> ${GITHUB_OUTPUT}

      - name: Checkout workflows
        uses: actions/checkout@v4
        with:
          ref: ${{ steps.workflows-ref.outputs.ref }} # use ref
          repository: org/workflows
          token: ${{ secrets.CIJOBTOKENWRITE }}
          path: workflows # checkout into separate folder

      - name: action1
        id: action1
        uses: ./workflows/actions/action1

Answer 12 · 2023-10-19T12:52:36.000Z

The workaround to this problem that we found was the most clean and easy was to simply do a curl request to get the SHA from the workflow run and use that in ref (https://docs.github.com/en/rest/actions/workflow-runs?apiVersion=2022-11-28#get-a-workflow-run).

Example (I don't think I missed anything):

jobs:
  Metadata:
    name: Extract metadata
    runs-on: ubuntu-latest
    permissions:
      actions: read    
    outputs:
      caller-sha: ${{ steps.workflows-ref.outputs.caller-sha }}
    steps:
      - name: Get workflow reference
        id: workflows-ref
        run: |
          sha=$(curl -L -H "Accept: application/vnd.github+json" -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" -H "X-GitHub-Api-Version: 2022-11-28" https://api.github.com/repos/<owner>/<repo>/actions/runs/${{ github.run_id }} | jq -r '.referenced_workflows[0] | .sha')
          echo "caller-sha=$sha" >> $GITHUB_OUTPUT
  Test:
    needs: [Metadata]
    uses: ./.github/workflows/test.yml
    secrets: inherit
    with:
      version: ${{ needs.Metadata.outputs.caller-sha }}

And then in ./.github/workflows/test.yml:

on:
  workflow_call:
    inputs:
      version:
        required: true
        type: string
jobs:
  Test:
    runs-on: ubuntu-latest
    steps:
      - name: Check out actions
        uses: actions/checkout@v4
        with:
          repository: <repo> 
          ref: ${{ input.version }}
          token: <secrets.TOKEN> ##you need a PAT-token with read rights to the repo

Answer 13 · 2024-01-22T22:10:41.000Z

I'm facing this problem too.

Our reusable workflow needs to checkout its own repository in order to execute a Python script. Allowing access to the called workflow repo's reference would enable versioning its behaviour.

Any developments on this? Thanks!

Answer 14 · 2024-03-07T22:26:01.000Z

The workaround to this problem that we found was the most clean and easy was to simply do a curl request to get the SHA from the workflow run and use that in ref (https://docs.github.com/en/rest/actions/workflow-runs?apiVersion=2022-11-28#get-a-workflow-run).

Example (I don't think I missed anything):

jobs:
  Metadata:
    name: Extract metadata
    runs-on: ubuntu-latest
    permissions:
      actions: read    
    outputs:
      caller-sha: ${{ steps.workflows-ref.outputs.caller-sha }}
    steps:
      - name: Get workflow reference
        id: workflows-ref
        run: |
          sha=$(curl -L -H "Accept: application/vnd.github+json" -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" -H "X-GitHub-Api-Version: 2022-11-28" https://api.github.com/repos/<owner>/<repo>/actions/runs/${{ github.run_id }} | jq -r '.referenced_workflows[0] | .sha')
          echo "caller-sha=$sha" >> $GITHUB_OUTPUT
  Test:
    needs: [Metadata]
    uses: ./.github/workflows/test.yml
    secrets: inherit
    with:
      version: ${{ needs.Metadata.outputs.caller-sha }}

And then in ./.github/workflows/test.yml:

on:
  workflow_call:
    inputs:
      version:
        required: true
        type: string
jobs:
  Test:
    runs-on: ubuntu-latest
    steps:
      - name: Check out actions
        uses: actions/checkout@v4
        with:
          repository: <repo> 
          ref: ${{ input.version }}
          token: <secrets.TOKEN> ##you need a PAT-token with read rights to the repo

This was a life-saver for me today. Thank you!

Answer 15 · 2024-03-10T17:31:50.000Z

I don't understand how such obvious requirement is missing... this is one of the first things I immediately needed when extracting workflows and composite actions to the same repo.

It seems like there is not enough dogfeeding happening at Github. :/

Answer 16 · 2024-04-17T18:26:07.000Z

This is definitely a thing that still matters. We're in the process of building out shared workflows that are composed of, well, composite actions in the same repository, and having to manually specify a branch as we modify those composite actions in the workflow file is tedious and error prone. Would be much better to be able to say:

  - uses: {path-to-workflow}/@${{ github.referenced_workflows[0].sha }}

Answer 17 · 2024-05-06T12:37:57.000Z

In case it helps anyone, we created a composite action that implements @maxbergs's solution

https://github.com/canonical/get-workflow-version-action

Example usage:

# Reusable workflow (e.g. build_charm.yaml)
on:
  workflow_call:

jobs:
  foo:
    runs-on: ubuntu-latest
    steps:
      - name: Get workflow version
        id: workflow-version
        uses: canonical/get-workflow-version-action@v1
        with:
          repository-name: canonical/data-platform-workflows
          file-name: build_charm.yaml
      # Use the version. For example:
      - name: Install Python CLI
        run: pipx install git+https://github.com/canonical/data-platform-workflows@'${{ steps.workflow-version.outputs.sha }}'#subdirectory=python/cli

Answer 18 · 2024-07-07T06:34:23.000Z

We have reusable workflows to build docker images and deploy microservices. The docker image workflow is triggered when a version is created and we use ${GITHUB_REF#refs/*/} to extract it.
From what people say above, we must write a composite action to handle this before calling the reusable workflow. This is ok when you have few repos but it becomes daunting when you have 40+ microservices.

We currently copy & paste the workflows because there is not much benefit to using the reusable workflows.
Is there is no other way to do this more efficiently, other than composite actions?

Answer 19 · 2024-07-08T07:06:00.000Z

@ckmoga I don't understand exactly what you're trying to accomplish, but it doesn't sound like you're affected by this issue

GITHUB_REF (or ${{ github.ref }}) is available in a reusable workflow

This issue is about accessing the ref that a reusable workflow was called with—i.e. the version of the reusable workflow—not about accessing the ref of the caller repository that triggered the workflow run

Answer 20 · 2024-07-08T07:29:04.000Z

@carlcsaposs-canonical it is possible that I am missing something but I cannot get my release number using ${GITHUB_REF#refs/*/}
It is empty but in ordinary workflow it is correct

Answer 21 · 2024-07-08T07:35:47.000Z

@ckmoga In that case, I think you're encountering a different issue

fwiw, I was able to access GITHUB_REF in a reusable workflow: https://github.com/carlcsaposs-canonical/test-reusable-workflow-githubref/actions/runs/9835427768/job/27149037144
https://github.com/carlcsaposs-canonical/test-reusable-workflow-githubref/tree/main/.github/workflows

Answer 22 · 2024-07-08T09:53:13.000Z

OK @carlcsaposs-canonical I got it working with $GITHUB_REF_NAME. ${{ github.ref }} throws unknown reference somehow. I think my issue was the curly brackets: ${GITHUB_REF}

Answer 23 · 2024-08-28T09:43:55.000Z

In case it helps anyone, we created a composite action that implements @maxbergs's solution

It's not obvious to me what the advantage of this solution is over directly passing the called workflow repo and ref as workflow inputs. Especially since for the action you need the repo and file name to get the ref.

The repo and file name of the called workflow might change, hence should not be hard-coded in the called workflow file.

You already know the repo, ref, and file name in the caller. So you could pass the repo and file name to the called workflow to execute the action, or just pass repo and ref and skip the action step.

Answer 24 · 2024-08-28T09:53:21.000Z

It's not obvious to me what the advantage of this solution is over directly passing the called workflow repo and ref as workflow inputs. Especially since for the action you need the repo and file name to get the ref.

For us, it's helpful because the repository name and workflow name are determined by the reusable workflow, but the ref is determined by the caller. (So it's easy to hard code repository name & workflow name in the reusable workflow)

We prefer having one source of truth instead of duplicating it as an input (which could lead to mistakes), and because we use tools like Renovate to update our refs (which are not easily capable of also updating that input)

But that's just what works for us—if passing the ref as an input works for you, there's no need to use an action that calls the GitHub API

Answer 25 · 2024-09-06T12:58:30.000Z

Hi guys, following this discussion and looking for a solution I created the following basic action to facilitate what @maxbergs provided us: dariocurr/checkout-called.

I hope it will be useful. The only difficult part, IMHO, might be creating the token with the correct set of permissions

Answer 26 · 2024-09-10T18:06:21.000Z

@dariocurr Thank you so much for creating this action! Worked perfectly.

Leaving this comment so others know to use it.

Answer 27 · 2024-11-14T13:51:53.000Z

@dariocurr , @carlcsaposs-canonical both your solutions are very nice, however, it may not work on self-hosted runner where either pipx or jq is not available :(

Answer 28 · 2024-11-14T14:24:23.000Z

@dariocurr , @carlcsaposs-canonical both your solutions are very nice, however, it may not work on self-hosted runner where either pipx or jq is not available :(

hi @nvincent-vossloh! here's what we use on self-hosted runners to install pipx:
https://github.com/canonical/data-platform-workflows/blob/60f088b7f0f967a8e35d45339f5123a6e74786f7/.github/workflows/integration_test_charm.yaml#L228-L245

Answer 29 · 2024-11-22T14:29:58.000Z

Would be great to see an update from the repository owners here as to whether they will take this up as a new feature...

Answer 30 · 2024-11-24T17:26:37.000Z

I recently encountered this issue and I found a workaround that's relatively simple. I don't know if this will work for everyone but it works in my use case. It gets an oidc JWT to get the job_workflow_ref field, so you need the id-token: write permission when calling the workflow.

My use case: I have a reusable workflow and in this workflow I create a PR comment that links to the workflow docs that is simply a markdown file in the same repo as the reusable workflow. I need the tag to link to the correct workflow doc version.

Here's an example reusable workflow that gets the workflow ref with which it was called and only relies on basic utilities (like curl, cut, base64, tr, etc.) so might be more self-hosted runner friendly in most cases (I use a lot of self-hosted runners). Full example in duboisf/get-reusable-workflow-reference#3:

name: Example workflow that can get it's own reference
on:
  push:
  pull_request:
  workflow_call:

jobs:
  Example:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      # Needed to get the called workflow reference
      id-token: write
    steps:
      - name: Get called workflow reference
        run: |
          JWT=$(curl -s -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" "$ACTIONS_ID_TOKEN_REQUEST_URL")
          PAYLOAD=$(echo "$JWT" | cut -d '.' -f 2 | base64 -d 2>/dev/null || true)

          WORKFLOW_FULL_REF=$(
            echo "$PAYLOAD" \
              | grep -o '"job_workflow_ref":"[^"]*"' \
              | cut -d':' -f2 \
              | cut -d'@' -f2 \
              | tr -d '"'
          )

          echo "WORKFLOW_FULL_REF=$WORKFLOW_FULL_REF"

          case $WORKFLOW_FULL_REF in
            refs/heads/*)
              WORKFLOW_REF=${WORKFLOW_FULL_REF#refs/heads/}
              ;;
            refs/tags/*)
              WORKFLOW_REF=${WORKFLOW_FULL_REF#refs/tags/}
              ;;
            refs/pull/*/merge)
              # exception: we got triggered by a pull request, the ref is the base branch
              WORKFLOW_REF=$GITHUB_HEAD_REF
              ;;
          esac

          echo "::notice ::WORKFLOW_FULL_REF=$WORKFLOW_FULL_REF"
          echo "::notice ::WORKFLOW_REF=$WORKFLOW_REF"

          echo "WORKFLOW_REF=$WORKFLOW_REF" | tee -a $GITHUB_ENV

      - uses: actions/checkout@v4
        with:
          repository: duboisf/get-reusable-workflow-reference
          ref: ${{ env.WORKFLOW_REF }}

      - name: Run script from called workflow
        run: |
          ./scripts/example.sh

And here's how I call it:

name: Call workflow example

on:
  pull_request:

jobs:

  CallWorkflow:
    permissions:
      contents: read
      id-token: write
    uses: duboisf/get-reusable-workflow-reference/.github/workflows/example.yml@v1

Output:

Answer 31 · 2024-11-25T07:55:48.000Z

I recently encountered this issue and I found a workaround that's relatively simple. I don't know if this will work for everyone but it works in my use case. It gets an oidc JWT to get the job_workflow_ref field, so you need the id-token: write permission when calling the workflow.

My use case: I have a reusable workflow and in this workflow I create a PR comment that links to the workflow docs that is simply a markdown file in the same repo as the reusable workflow. I need the tag to link to the correct workflow doc version.

Here's an example reusable workflow that gets the workflow ref with which it was called and only relies on basic utilities (like curl, cut, base64, tr, etc.) so might be more self-hosted runner friendly in most cases (I use a lot of self-hosted runners). Full example in duboisf/get-reusable-workflow-reference#3:

name: Example workflow that can get it's own reference
on:
  push:
  pull_request:
  workflow_call:

jobs:
  Example:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      # Needed to get the called workflow reference
      id-token: write
    steps:
      - name: Get called workflow reference
        run: |
          JWT=$(curl -s -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" "$ACTIONS_ID_TOKEN_REQUEST_URL")
          PAYLOAD=$(echo "$JWT" | cut -d '.' -f 2 | base64 -d 2>/dev/null || true)

          WORKFLOW_FULL_REF=$(
            echo "$PAYLOAD" \
              | grep -o '"job_workflow_ref":"[^"]*"' \
              | cut -d':' -f2 \
              | cut -d'@' -f2 \
              | tr -d '"'
          )

          echo "WORKFLOW_FULL_REF=$WORKFLOW_FULL_REF"

          case $WORKFLOW_FULL_REF in
            refs/heads/*)
              WORKFLOW_REF=${WORKFLOW_FULL_REF#refs/heads/}
              ;;
            refs/tags/*)
              WORKFLOW_REF=${WORKFLOW_FULL_REF#refs/tags/}
              ;;
            refs/pull/*/merge)
              # exception: we got triggered by a pull request, the ref is the base branch
              WORKFLOW_REF=$GITHUB_HEAD_REF
              ;;
          esac

          echo "::notice ::WORKFLOW_FULL_REF=$WORKFLOW_FULL_REF"
          echo "::notice ::WORKFLOW_REF=$WORKFLOW_REF"

          echo "WORKFLOW_REF=$WORKFLOW_REF" | tee -a $GITHUB_ENV

      - uses: actions/checkout@v4
        with:
          repository: duboisf/get-reusable-workflow-reference
          ref: ${{ env.WORKFLOW_REF }}

      - name: Run script from called workflow
        run: |
          ./scripts/example.sh

And here's how I call it:

name: Call workflow example

on:
  pull_request:

jobs:

  CallWorkflow:
    permissions:
      contents: read
      id-token: write
    uses: duboisf/get-reusable-workflow-reference/.github/workflows/example.yml@v1

Output:

Thank you for the wonderful cue. Implemented in the dev branch. Currently testing it.
Any feedback or help is more than welcome

Answer 32 · 2025-01-10T18:10:24.000Z

I would argue that the reusable workflow should have be a convenient way of referencing its own repository that it came from without having to call outside APIs or even having to check it out again. Afterall how the reusable workflow come into existence anyway? It had to be checked out by the calling workflow, so its repo must still be somewhere.