agola-io/agola

variables and secrets are not loaded in building environment

grapemix opened this issue · 8 comments

What happened:
In short, variables and secrets are not loaded in building environment. I run export cmd and the variables are set as empty string. The variables and secrets are set under projectgroup via cli and my project are under correct project group. Also, it is hard to debug and hence the user experience is not good.

What you expected to happen:
Secrects and variables are able to use in building environment.

How to reproduce it (as minimally and precisely as possible):
see detail.

Anything else we need to know?:

Environment:

  • Agola version: v0.8.0
  • Agola running environment (if useful to understand the bug):
    My setup is k8s + fluxcd + gitea + OIDC and installed via helm chart.
  • Detail:
    After installing via helm chart, I assume the installation and the OIDC integration are success because I can login and can fetch projects from my gitea server. I tried to create a projectgroup under my user via cli
agola projectgroup create -n PGNAME --parent user/USERNAME --token TOKEN

2023-12-12T21:53:59.488231024Z INF agola/cmd/agola/cmd/projectgroupcreate.go:77 > creating project group
2023-12-12T21:53:59.489983439Z FTL agola/cmd/agola/cmd/projectgroupcreate.go:33 > error="failed to create project group: remote error badrequest"

So I create my projectgroup via GUI, so what is the right cli to create a projectgroup under my user BTW? I can't find it in doc.

After that, I create my secrets and variable under my user's project group via cli:

agola projectgroup secret create  --projectgroup user/USERNAME/PGNAME --name SECNAME -f FILE --token TOKEN
agola projectgroup variable create  --projectgroup user/USERNAME/PGNAME --name SECNAME -f FILE2 --token TOKEN

The cli stdout looks like success and I can see secrets and variables name are shown in the project group and my project's setting page. At this point, I thought secrets and variable are stored correctly (but it wasn't).

By digging various examples, I try to clone my private repo with this config:

version: v0
runs:
  - name: build image
    tasks:
      - name: clone
        runtime:
          containers:
            - image: alpine/git
        steps:
          - clone:
          - save_to_workspace:
              contents:
                - source_dir: .
                  dest_dir: /
                  paths:
                    - '**'

But I can't clone the source. After digging the logs and the code, I pass my variables and secrets like this:

version: v0
runs:
  - name: build image
    tasks:
      - name: clone
        environment:
          AGOLA_SSHPRIVKEY:
            from_variable: gitea-rsa-private-key
          AGOLA_SKIPSSHHOSTKEYCHECK: "true"
        runtime:
          containers:
            - image: alpine/git
        steps:
          - clone:
          - save_to_workspace:
              contents:
                - source_dir: .
                  dest_dir: /
                  paths:
                    - '**'

Cloning is success and hence I think my secrets and variables are really working (but again, it wasn't). So I continue writing the remaining of my workflow.

After few days of debugging, I found out the root cause is my secrets and variables cannot passed to the building env.

I found it because I finally use the export cmd. I don't want to use cmd because I find out I cannot delete a build and I don't want to leak my creds and secrets.

We only needs to pass AGOLA_SKIPSSHHOSTKEYCHECK in env and the AGOLA_SSHPRIVKEY is being passed automatically like the following:

version: v0
runs:
  - name: build image
    tasks:
      - name: clone
        environment:
          AGOLA_SKIPSSHHOSTKEYCHECK: "true"
        runtime:
          containers:
            - image: alpine/git
        steps:
          - run: export
          - clone:
          - save_to_workspace:
              contents:
                - source_dir: .
                  dest_dir: /
                  paths:
                    - '**'

It seems I am not the only one who got stuck, others are also being blocked by private repo: https://talk.agola.io/t/cloning-private-repos/96/2.

Even worse, when I dig up the logs from k8s, those logs are not helpful. Most of them doesn't have traceback. I can't tell what is wrong.

From agola-runservice:

│ goroutine 7162952 [running]:                                                                                                                                                                                                                                                                                                                                                                                                          │
│ net/http.(*conn).serve.func1()                                                                                                                                                                                                                                                                                                                                                                                                        │
│     /usr/local/go/src/net/http/server.go:1854 +0xbf                                                                                                                                                                                                                                                                                                                                                                                   │
│ panic({0x2018ac0, 0x367f860})                                                                                                                                                                                                                                                                                                                                                                                                         │
│     /usr/local/go/src/runtime/panic.go:890 +0x263                                                                                                                                                                                                                                                                                                                                                                                     │
│ agola.io/agola/internal/services/runservice/api.(*RunByGroupHandler).ServeHTTP(0xc0000eff10, {0x2706c70, 0xc0003c67e0}, 0xc0003c5900)                                                                                                                                                                                                                                                                                                 │
│     /agola/internal/services/runservice/api/api.go:551 +0x50d                                                                                                                                                                                                                                                                                                                                                                         │
│ github.com/gorilla/mux.(*Router).ServeHTTP(0xc000b7e180, {0x2706c70, 0xc0003c67e0}, 0xc0003c5700)                                                                                                                                                                                                                                                                                                                                     │
│     /go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210 +0x1cf                                                                                                                                                                                                                                                                                                                                                                       │
│ github.com/gorilla/mux.(*Router).ServeHTTP(0xc000b7e300, {0x2706c70, 0xc0003c67e0}, 0xc0003c5500)                                                                                                                                                                                                                                                                                                                                     │
│     /go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210 +0x1cf                                                                                                                                                                                                                                                                                                                                                                       │
│ net/http.serverHandler.ServeHTTP({0xc000a41350?}, {0x2706c70, 0xc0003c67e0}, 0xc0003c5500)                                                                                                                                                                                                                                                                                                                                            │
│     /usr/local/go/src/net/http/server.go:2936 +0x316                                                                                                                                                                                                                                                                                                                                                                                  │
│ net/http.(*conn).serve(0xc000736ab0, {0x2707bd8, 0xc000bec150})                                                                                                                                                                                                                                                                                                                                                                       │
│     /usr/local/go/src/net/http/server.go:1995 +0x612                                                                                                                                                                                                                                                                                                                                                                                  │
│ created by net/http.(*Server).Serve                                                                                                                                                                                                                                                                                                                                                                                                   │
│     /usr/local/go/src/net/http/server.go:3089 +0x5ed                                                                                                                                                                                                                                                                                                                                                                                  │
│ 2023/12/04 19:19:20 http: panic serving 10.42.0.109:34418: runtime error: invalid memory address or nil pointer dereference                    

From agola-configstore:

2023-12-04T09:20:41.531619682Z ERR agola/internal/services/configstore/api/project.go:138 > error="apiError (kind: notexist): project \"user/USERNAME/PROJECTNAME/REPONAME\" doesn't exist"

From agola-gateway:

2023-12-04T19:19:49.480593118Z ERR agola/internal/services/gateway/api/run.go:264 > error="apiError (kind: internal): Get \"http://agola-runservice:4000/api/v1alpha/runs/group/%2Fproject%2F30019bb1-9009-41f7-9bba-a29bf49de779/15\": EOF"

So I re-deploy to try to reproduce it.

I found out that after I re-deploy, agola-gateway already gave me this error after connecting OIDC.

ERR agola/internal/services/gateway/api/run.go:211 > error="failed to determine permissions: apiError (kind: internal): remote error internal"

But how can an user to understand without traceback or detail?

At this point, I am not even sure my OIDC integration is success or not.

So in conclusion, the reason I have to spend so much time to troubleshoot is because:

  1. The logs are really not helpful, traceback or which part configs go wrong are needed.
  2. Debugging option is not available
  3. Errors are being silenced
  4. There has no cli to show the secrets and variables
  5. GUI only shows the key inside secrets and variables, but not values. We don't have have to show the entire values, we can only show prefix or suffix like credit card number.
  6. No way to delete build output

Let me know if you need more detail. Thanks.

  • I tried to create a projectgroup under my user via cli
agola projectgroup create -n PGNAME --parent user/USERNAME --token TOKEN

2023-12-12T21:53:59.488231024Z INF agola/cmd/agola/cmd/projectgroupcreate.go:77 > creating project group
2023-12-12T21:53:59.489983439Z FTL agola/cmd/agola/cmd/projectgroupcreate.go:33 > error="failed to create project group: remote error badrequest"

So I create my projectgroup via GUI, so what is the right cli to create a projectgroup under my user BTW? I can't find it in doc.

I tried to create the projectgroup with the command agola projectgroup create -n PGNAME --parent user/USERNAME --token USERTOKEN--gateway-url AGOLAGATEWAY it is working for me. Do you have used the admin token ? if yes I think it is the reason of the error.

@grapemix I need to see the log of the failed run to understand better.

I can see you are setting the AGOLA_SKIPSSHHOSTKEYCHECK variable; but it is passed automatically by the SkipSSHHostKeyCheck parameter you set in the remotesource .
You can try to update(or recreate) the remotesource with --skip-ssh-host-key-check (it is a boolean), if you want skip ssh host key check. And can you reply here if this solution fix the problem?

  • I tried to create a projectgroup under my user via cli
agola projectgroup create -n PGNAME --parent user/USERNAME --token TOKEN

2023-12-12T21:53:59.488231024Z INF agola/cmd/agola/cmd/projectgroupcreate.go:77 > creating project group
2023-12-12T21:53:59.489983439Z FTL agola/cmd/agola/cmd/projectgroupcreate.go:33 > error="failed to create project group: remote error badrequest"

So I create my projectgroup via GUI, so what is the right cli to create a projectgroup under my user BTW? I can't find it in doc.

I tried to create the projectgroup with the command agola projectgroup create -n PGNAME --parent user/USERNAME --token USERTOKEN--gateway-url AGOLAGATEWAY it is working for me. Do you have used the admin token ? if yes I think it is the reason of the error.

You are correct. That's the problem. After creating the a token from my user's profile and applying the token, I can create projectgroup from cli now. I can hardly guess the admin token is the problem. Thanks.

@grapemix I need to see the log of the failed run to understand better.

I can see you are setting the AGOLA_SKIPSSHHOSTKEYCHECK variable; but it is passed automatically by the SkipSSHHostKeyCheck parameter you set in the remotesource . You can try to update(or recreate) the remotesource with --skip-ssh-host-key-check (it is a boolean), if you want skip ssh host key check. And can you reply here if this solution fix the problem?

Thanks for the reply. Since I can hardcode the AGOLA_SKIPSSHHOSTKEYCHECK flag in environment section, although it takes me a few hrs to figure it out, the AGOLA_SKIPSSHHOSTKEYCHECK is not the problem.

What the problem is I CANNOT pass secret (which I cannot hardcode) in the environment section.

For hardcode var and internal var in the environment section, I can verify values have been passed successfully by the export step. But for any secrets, values CANNOT be passed. I can see the key, but value are all empty.

For example:

        environment:
          DOCKERAUTH:
            from_variable: gitea-reg-all-rw-pat

I need to set DOCKERAUTH in order to publish docker images. I can see "gitea-reg-all-rw-pat" 's key being shown in the variable table in the projectgroup and the project's setting page. But the problem is I cannot find much log to troubleshoot the problem. Can you provide detail on extracting the log? All logs from all pods seems not helpful.

One of the theory is because I use the admin token to create variables and secrets. So I uninstall and install the whole helm release include all pv and pvc to try again with user token. But I still cannot pass secrets.

I need to set DOCKERAUTH in order to publish docker images. I can see "gitea-reg-all-rw-pat" 's key being shown in the variable table in the projectgroup and the project's setting page. But the problem is I cannot find much log to troubleshoot the problem. Can you provide detail on extracting the log? All logs from all pods seems not helpful.

One of the theory is because I use the admin token to create variables and secrets. So I uninstall and install the whole helm release include all pv and pvc to try again with user token. But I still cannot pass secrets.

Is normal that you can not see the secret values in the projectgroup/project's setting page, but you can see the secret name/key, but not the values.
Use of admin token to create variables and secrets this is not the cause of the problem.

Is normal that you can not see the secret values in the projectgroup/project's setting page, but you can see the secret name/key, but not the values.
Use of admin token to create variables and secrets this is not the cause of the problem.

For hardcode var and internal var in the environment section, I can verify values have been passed successfully by the export step. But for any secrets, values CANNOT be passed. I can see the key, but value are all empty.

I don't understand what do you meen. Have you added some step to show the values(for example echo $DOCKERAUTH) to see the gitea-reg-all-rw-pat value?

Sorry for my late reply.

Have you added some step to show the values?
Yes. I add the export step in both task.

        steps:
          - run: export
          - clone:

I already posted the complete tasks above. Let me know if you need more information. Thanks for your follow up.

@grapemix Sorry from the original issue description and next answer I cannot understand the main point of the issue since you are asking multiple unrelated questions. So please open different issues/question/enhancements for every single point or it'll be impossible to follow the thread.