Dagger `install-only` doesn't persist `age-key`

Question

Dagger `install-only` doesn't persist `age-key`

Closed this issue 2 years ago · 11 comments

grouville commented 3 years ago

Problem

Several users have felt the need to use dagger with the install-only option for 2 reasons mainly:

They need to retrieve the context of the commit / PR
Their internal policy restricts the use of encrypted files containing secrets, regardless the security level of the encrypted file.

For the second case, no issue arises

People just push without a .dagger folder and literally add the inputs/secrets in the GitHub workflow:

# This is a basic workflow to help you get started with Actions

name: Dagger PR test

# Controls when the workflow will run
on:
  # Triggers the workflow on a pull request event
  pull_request:

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "dagger"
  dagger:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/checkout@v2

      - name: Install Dagger
        uses: dagger/dagger-action@v1
        with:
          install-only: true
          
      #- name: Show Dagger version
        #run: dagger version
        
      - name: Setup dagger environment "myenv"
        run: |
          dagger init
          dagger new myenv -p ./myenv
        
      - name: Pass PR number to Dagger
        run: dagger input text prID ${{github.event.number}} -e myenv
        
      - name: List Dagger inputs
        run: dagger input list -e myenv
        
      - name: Dagger up!
        run: dagger up -e myenv
        
      - name: List Dagger outputs
        run: dagger output list -e myenv

But, some people want to retrieve part of the Github context AND store their secrets in the sops file:

jobs:
  test_ci:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Install Dagger
        uses: dagger/dagger-action@v1
        with:
          age-key: ${{ secrets.DAGGER_AGE_KEY }}
          install-only: true

      - name: Pass Commit SHA to Dagger
        run: dagger input text githubCommitSha ${{github.sha}}

      - name: Dagger up!
        run: dagger up

The above code doesn't work, dagger doesn't find the sops key

Current Solution

After some debug, I found a way to make it work. It seems a little weird because the second time we added an input, the sops key gets found.

name: Dagger PR test

# Controls when the workflow will run
on:
  # Triggers the workflow on a pull request event
  push:
  pull_request:

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "dagger"
  dagger:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/checkout@v2

      - name: Install Dagger
        uses: dagger/dagger-action@v1
        with:
          install-only: true

      - name: Pass commit sha
        uses: dagger/dagger-action@v1
        with:
          age-key: ${{ secrets.DAGGER_AGE_KEY }}
          args: input text githubCommitSha ${{github.sha}}

      - name: Dagger new input
        run: dagger input text githubCommitNumber "ok"

      - name: Dagger up!
        run: dagger up

Answer 1 · 2022-02-21T15:01:47.000Z

Btw, this will become obsolete with dagger europa, as env var will be retrievable from the context

Answer 2 · 2022-02-21T15:54:23.000Z

As discussed with @helderco (thanks for your point of view), this creates a weird DX issue.

The doc needs to be updated + a PR extending it needs to be proposed (doing it right now)

Answer 3 · 2022-02-21T16:05:50.000Z

Can you put it in its own step?

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/checkout@v2

      - name: Install Dagger
        uses: dagger/dagger-action@v1
        with:
          install-only: true

      - name: Set up age key
        uses: dagger/dagger-action@v1  
        with:
          age-key: ${{ secrets.DAGGER_AGE_KEY }}

      - name: Pass commit sha
        run: dagger input text githubCommitSha ${{github.sha}}

      - name: Dagger new input
        run: dagger input text githubCommitNumber "ok"

      - name: Dagger up!
        run: dagger up

Answer 4 · 2022-02-21T16:12:08.000Z

That's an excellent idea. But this doesn't work:

Run dagger/dagger-action@v1
[8](https://github.com/grouville/test-ci/runs/5277022914?check_suite_focus=true#step:4:8)
Downloading https://dl.dagger.io/dagger/releases/0.2.0-alpha.5/dagger_v0.2.0-alpha.5_linux_amd64.tar.gz
[9](https://github.com/grouville/test-ci/runs/5277022914?check_suite_focus=true#step:4:9)
Extracting Dagger
[10](https://github.com/grouville/test-ci/runs/5277022914?check_suite_focus=true#step:4:10)
/usr/bin/tar xz --warning=no-unknown-keyword --overwrite -C /home/runner/work/_temp/ede94a77-8382-4663-9396-2effec74b1df -f /home/runner/work/_temp/cfda0e10-c31e-4040-a6f7-a5fc3fae864d
[11](https://github.com/grouville/test-ci/runs/5277022914?check_suite_focus=true#step:4:11)
Error: args input required

age-key needs an arg to work. Making an exception like that is possible though

      - name: Install Dagger
        uses: dagger/dagger-action@v1
        with:
          age-key: ${{ secrets.DAGGER_AGE_KEY }}
          install-only: true

Answer 5 · 2022-02-21T16:14:29.000Z

Ah, looking at the code it makes sense. Because everything else is trying to run a dagger command.

Yes it makes sense to me having the age-key being included as part of the installation process.

Answer 6 · 2022-02-21T16:23:52.000Z

Changing the order still fails, as the install-only returns and is the only option for which an arg is not required.
With the changes I was thinking about:

install only possible
install only + age-key possible
any command + arg possible

Answer 7 · 2022-02-23T10:19:05.000Z

I think we could mutualize the steps using an input list like:

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/checkout@v2

      - name: Dagger
        uses: dagger/dagger-action@v1
        with:
          age-key: ${{ secrets.DAGGER_AGE_KEY }}
          cmds: |
            input text githubCommitSha ${{github.sha}}
            input text githubCommitNumber "ok"
            up

So it would be self-contained in the same action. WDYT?

Answer 8 · 2022-02-23T12:26:06.000Z

I think we could mutualize the steps using an input list like:

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/checkout@v2

      - name: Dagger
        uses: dagger/dagger-action@v1
        with:
          age-key: ${{ secrets.DAGGER_AGE_KEY }}
          cmds: |
            input text githubCommitSha ${{github.sha}}
            input text githubCommitNumber "ok"
            up

So it would be self-contained in the same action. WDYT?

That feels smooth, you're right. And I can't seem to see a use-case where this would bring a DX issue.

One (probably dumb) question
As we default the version to latest when the version step is not specified. Wouldn't that break the current users configs or is it scoped to some versions ? In other words, wouldn't that bring retrocompatibility issues ?

Another question
Does that mean that we shall remove the install-only option ?
What about the workdir option ? As dagger has the --project CLI command, is it still relevant ?

Europa will also remove that necessity I believe

From your proposal, the new steps would be:

Following inputs can be used as step.with keys

Name	Type	Default	Description
version	String	latest	Dagger version
age-key	String		Dagger private key
workdir	String	.	Working directory (below repository root)
cmds	List<String>		List of Dagger commands to execute
cleanup	Bool	true	Cleanup Dagger home folder at the end of a job

WDYT @shykes ?

Answer 9 · 2022-02-24T21:47:05.000Z

I don't have a strong opinion since this is specific to 0.1 and we are releasing 0.2 in the coming weeks.

I am more concerned about how:

How to cleanly migrate users of this action from Dagger 0.1 to 0.2
How the migration interacts with #21

Answer 10 · 2022-02-24T23:54:54.000Z

How to cleanly migrate users of this action from Dagger 0.1 to 0.2

As GHA best practice, and as pointed out in Dagger docs, users have been (at least should have been) pinning to a specific version of this action rather than main. Excerpt from GHA tutorial on our docs site: uses: dagger/dagger-action@v1

We can safely publish a 0.2-based action as uses: dagger/dagger-action@v2 without breaking any 0.1 user.

Note that:

v1 / v2: that's the version of the GH action, reserved for breaking changes in the action definition itself (e.g. we might change the syntax because of dagger do)
Additionally, there's a version field in the action which defines which version of dagger to use (defaults to latest)

We can:

Force v1 to use the latest 0.1.x version (rather than latest)
Have a v2 with the latest syntax that points to latest (0.2.x)

How the migration interacts with #21

AFAIK (@crazy-max to confirm), it should run smoothly. GH sets up a "redirect" after renaming a repo. Even though the new canonical name will be different, the old name can still be used interchangeably.

Answer 11 · 2022-03-09T20:26:27.000Z

Not relevant anymore with new version