Consul template is not reading token in $HOME./vault

Question

Consul template is not reading token in $HOME./vault

hayoade1 opened this issue 6 years ago · 30 comments

I tried following the example in vault-agent-k8s-demo using gke instead of minikube. Consul-template container keeps failing, and when I run "kubectl logs vault-agent-example consul-template", I see the following error in the logs:

2019/01/01 16:18:14.061809 [WARN] (view) vault.read(secret/myapp/config): vault.read(secret/myapp/config): Error making API request.

URL: GET http://xx.xxx.xx.xx:8200/v1/secret/myapp/config
Code: 400. Errors:

missing client token (retry attempt 4 after "8s")

I'm able to see the token when I exec into the vault-agent container with the following commands

$ kubectl exec -it vault-agent-example --container vault-agent-auth sh

/# echo $(cat /home/vault/.vault-token)

Also, I'm able to successfully authenticate with the token obtain with the command above using the CLI.

The consul template container also works when I hard code a valid token in the nano example-k8s-spec.yml as an environment variable.

all files in the repo were unaltered except for the nano example-k8s-spec.yml file as I had to point the Vault url to my vault service.

Answer 1 · 2019-01-07T18:27:58.000Z

@hayoade1 Could you check and see which kv version is mounted at secret/?

vault secrets list -detailed

Since Consul Template doesn't seem to fully support kv-v2, this guide was written for kv. So, that might be the issue.

Answer 2 · 2019-01-07T19:48:03.000Z

@yhyakuna thanks for your comment. Its V1. See output of the command below:

secret/ kv kv_8c508b53 system system false replicated false map[version:1] key/value secret stor

Answer 3 · 2019-01-08T01:13:38.000Z

@hayoade1 I was able to reproduce the error. It seems that .vault_token is not a shared data in a GKE cluster while it was reachable from consul-template container in minikube.

To make this work, the sink must be modified to write the token as a shared data. The current vault agent configuration (vault-agent-config.hcl) is:

...
    sink "file" {
        config = {
            path = "/home/vault/.vault-token"
        }
    }

@tdsacilowski Do you have other comment/insight on this? Is my assumption correct?

Answer 4 · 2019-01-08T14:06:37.000Z

@yhyakuna and @tdsacilowski how can I configure the sink to write the token as a shared data? I do not see any info on it in the vault agent documentation.
Thank you for your assistance.

Answer 5 · 2019-01-08T18:00:36.000Z

@hayoade1 Reviewing the example-k8s-spec.yml, it is correctly setting the /home/vault to be a shared storage between the vault-agent-auth and consul-template containers (see line 35 & 60), so there's no reason why the Consul Template cannot read the token from /home/vault/.vault_token file. (It's not the sink setting. The sink path needs to point to a shared storage so that other containers in the pod can read.)

So, I'm puzzled :(

...
  containers:
    # Vault container
    - name: vault-agent-auth
      image: vault

      volumeMounts:
        - name: vault-token
          mountPath: /home/vault
       ...
     
    # Consul Template container
    - name: consul-template
      image: hashicorp/consul-template
      imagePullPolicy: Always

      volumeMounts:
        - name: vault-token
          mountPath: /home/vault
...

The fact that this works fine with minikube, this must be a GKE issue???

I'm a kubernetes beginner myself, so need to do some more research on GKE. FYI, this article explains creating shared storage in a pod.

Answer 6 · 2019-01-08T18:50:33.000Z

I tried what @hayoade1 did and set the VAULT_TOKEN environment variable in example-k8s-spec.yml:

...
    # Consul Template container
    - name: consul-template
      image: hashicorp/consul-template
      imagePullPolicy: Alway

     ...

      env:
        - name: VAULT_TOKEN
          value: $(cat /home/vault/.vault-token)
          ...

Now, I'm getting error performing token check error instead of missing client token error. :(

Answer 7 · 2019-01-08T19:35:23.000Z

@yhyakuna That didn't work when I tried it as well, but when I hardcoded a valid token in the the value field it worked. For example:

...
# Consul Template container
- name: consul-template
image: hashicorp/consul-template
imagePullPolicy: Alway

 ...

  env:
    - name: VAULT_TOKEN
      value: <enter valid token>
      ...

Answer 8 · 2019-01-29T20:16:26.000Z

This seems to be a GKE specific issue.
I just tested this successfully against a Kubernetes cluster running on Azure Kubernetes Services. :(

Answer 9 · 2019-03-04T11:28:16.000Z

This isn't just a GKE specific issue. I am seeing exactly the same behaviour on AWS.

I know the token created by the vault container is valid because I can exec into that container and read secrets that are allowed for the role successfully.

Unfortunately I can't exec into the consul-template container since it doesn't allow that nor can I ask it cat the contents of the token file which should be mounted at /home/vault/.vault-token so I can prove that its there.

What I did try was explicitly setting the location of the token file in the HCL config ..

vault_agent_token_file = "/home/vault/.vault-token"

Interestingly that now returns a permissions error, but again its hard to debug cos I can't get into the consul-template container ...

* permission denied (retry attempt 6 after "32s")
2019/03/04 11:27:05.322460 [WARN] (view) vault.read(kv/digital/ukd_services/apikey): vault.read(kv/digital/ukd_services/apikey): Error making API request.

Answer 10 · 2019-03-04T11:57:27.000Z

Ok, so using the hashicorp/consul-template:alpine image I can exec into the container and I can see that the .vault-token file is indeed mounted in /home/vault as expected.

Answer 11 · 2019-03-04T12:25:53.000Z

For the purpose of testing, if I explicitly set the VAULT_TOKEN inside the consul-template container I see this ...

export VAULT_TOKEN=$(cat /home/vault/.vault-token)

2019/03/04 12:20:20.981041 [WARN] vault.token: renewer returned (maybe the lease expired)
2019/03/04 12:20:35.981163 [WARN] (view) lease expired or is not renewable (retry attempt 5 after "16s")
2019/03/04 12:20:51.995734 [WARN] vault.token: failed to renew: Error making API request.

Answer 12 · 2019-03-07T22:03:51.000Z

@goffinf Thanks for the additional information! Seems like, this demo only works with Minikube and AKS cluster as-is. :(

I'll try your approach (vault_agent_token_file) and investigate some more.

Answer 13 · 2019-03-07T23:57:22.000Z

@yhyakuna (Yoko), I did get this working eventually. It appears in part to be related to the version of the image you use for consul-templates and likely the version of Vault installed.

In the end I used this image for the initContainer to get the Vault auth token and write it to the shared volume (vault:1.0.3). This is the HashiCorp Vault Agent which I configured to exit as soon as the token was writing to the sink.

For consul-template I used consul-template:0.19.6-dev-alpine (you can exec into the alpine ones) and I mounted vault-token at the path /home/consul-template rather than /home/vault (this seemed critical). After that everything just worked as expected.

I did try the latest consul-template images (0.20.0) but they just continuously return ‘missing client token’ or 403 unauthorised not matter what I tried so in the end I gave up since I do actually have a working solution now. I wondered whether it might have to do with whether v1 or v2 is set up for the Vault secrets backend, but that’s just a wild guess.

HTHs

Fraser.

Answer 14 · 2019-03-08T01:31:20.000Z

@goffinf (Fraser), thank you so much for this info!

Answer 15 · 2019-03-12T03:41:26.000Z

@yhyakuna can you confirm that the demo works for AKS? I'm still getting missing token errors :( , and I tried @goffinf 's suggestion to I'm getting errors pertaining to the consul template image consul-template:0.19.6-dev-alpine. Can you also confirm that it is correct?

Answer 16 · 2019-03-12T04:33:48.000Z

@hayoade1 Unlike GKE, this demo worked smoothly against AKS cluster for me. I had to demo this on AKS a few weeks ago, so I'm very sure of that.

I've been so busy with other work that I haven't had a chance to try the consul-template:0.19.6-dev-alpine image, yet. Hope to find the time later this week.

Answer 17 · 2019-03-12T04:39:36.000Z

@yhyakuna Looks like the consul-template image was recently updated. I will try again. wish me luck :)

Answer 18 · 2019-03-12T09:05:08.000Z

@hayoade1 I think the error I was seeing when using the latest consul-template images (0.20.x) are likely because of the version of Vault that is running and support for either the v1 and/or v2 APIs. I don’t have any control over that in my situation since Vault is provided as a platform level service to me by our CloudOps team. If you have it within your gift to use the latest version of Vault I would definitely give the latest consul-templates image a go.

HtH

Fraser.

Answer 19 · 2019-03-16T21:03:31.000Z

I'm having the same issue and I can confirm that in my case on minikube in it works only with consul-template:0.19.6-dev-alpine image. Both 0.19.5 and 0.20 don't work :(

Answer 20 · 2019-03-21T05:31:22.000Z

If you're using vault kv-v2 you need to update the .Data.data format in the consul-template-config.hcl file

consul-template loads the vault token using the consul-template config (see: hashicorp/consul-template@72366c2)

The reason you can't exec into the consul-template image is because it's a scratch container (see: https://github.com/hashicorp/consul-template/blob/master/docker/scratch/Dockerfile)

Haven't tested on AKS or GKE but this should work regardless.

Answer 21 · 2019-03-22T23:23:06.000Z

This was working for me perfectly before, but now I'm seeing the same issue (missing client token).

I tried consul-template:0.19.6-dev-alpine image as some of you suggested. When I invoked consul-template -template="<test.tpl>:<test.output>" -once command inside the consul-template container, it worked fine which means that Cosul Template used the ~/.vault-token as it should. So, the issue seems to be due to a race condition (consul-template executed before token was written)?

The approach @goffinf pointed out with initContainers worked for me (tested with Minikube as well as AKS). So, I've updated the demo files.

The learn will be updated shortly.

Answer 22 · 2019-03-26T18:50:03.000Z

@hayoade1 I just tested successfully against GKE cluster. So, I'm pretty convinced that this was a race condition issue. I'm now closing this, but if the solution doesn't work for you, feel free to re-open or create another issue report.

Thank you so much for bringing this to our attention!

Answer 23 · 2019-04-24T03:49:36.000Z

yeah, this got broken in 0.20.0; hashicorp/consul-template#1182 (comment)

Answer 24 · 2019-04-25T12:13:08.000Z

I am also getting the same error . Seems like consul template container is not able to read "/home/vault/.vault-token" . Is there any workaround available ?

Code: 403. Errors:

permission denied (retry attempt 1 after "1s")
2019/04/25 12:08:20.470880 [WARN] (view) vault.read(secret/testapp/config): vault.read(secret/testapp/config): Error making API request.
URL: GET https://example.default.svc:8200/v1/sys/internal/ui/mounts/secret/testapp/config

Answer 25 · 2019-04-25T22:18:59.000Z

yeah, go back to an older version of consul template (registry.hub.docker.com/sethvargo/consul-template:0.19.6.dev-alpine is the one I am using, for example).

Answer 26 · 2019-04-27T00:14:38.000Z

@nitishnigam1989 and @jurgenweber I just tested with hashicorp/consul-template:0.20.0-alpine but this demo worked fine. So, I have a couple of questions:

Is your secret path is enabled with kv v1 or kv v2? --> this demo assumes that secret is kv v1
@nitishnigam1989 Your error message indicates that you're trying to read from secret/testapp/config path instead of secret/myapp/config (testapp vs. myapp) --> Did you make sure to update the setup-k8s-auth.sh and consul-template-config.hcl to read from secret/testapp/config?

If using kv-v2 at secret, please update the /configs-k8s/consul-template-config.hcl as follow:

...
  <p>Some secrets:</p>
  {{- with secret "secret/data/myapp/config?version=1" }}
  <ul>
  <li><pre>username: {{ .Data.data.username }}</pre></li>
  <li><pre>password: {{ .Data.data.password }}</pre></li>
  </ul>
  {{ end }}
  </body>
  </html>
  EOH
}

Answer 27 · 2019-04-27T06:22:09.000Z

@yhyakuna For my POC purpose, i was using secret path as secret/testapp/config [(not myapp)] so I did changed all references .Also changed consul-template-config.hcl to read from /secret/testapp/config.
I will again try with hashicorp/consul-template:0.20.0-alpine and see if it works. Earlier I was using hashicorp/consul-template:alpine image.

@jurgenweber Thanks a lot, Yes ,using "registry.hub.docker.com/sethvargo/consul-template:0.19.5.dev-alpine" image for consul-template worked well for me :) .

Answer 28 · 2019-04-29T16:14:19.000Z

Hmmm... I was able to successfully test this demo with hashicorp/consul-template:alpine as well, so I'm puzzled. :(

Answer 29 · 2019-07-11T00:39:06.000Z

I have created a PR to resolve this in the later versions of consul-template; hashicorp/consul-template#1228

Answer 30 · 2024-01-19T15:42:02.000Z

May not be the same problem, but I recently hit a problem where the owner + permissions of /vault/.vault-token made it readable only by root, and the consul-template container ran as uid 100, and therefore failed to read it.