Vault Kubernetes auth role token is not renewed

[aleskiontherun]

Envconsul version

envconsul v0.13.0 (c9c55c9)

Configuration

renew_token = true should probably be redundant like with a plaintext token, as there's no agent involved. Anyway, it makes no difference and doesn't help to resolve the issue.

# /etc/envconsul.d/01-common.hcl
log_level = "warn"
exec {
  splay        = "5s"
  kill_timeout = "10s"
}
max_stale = "1m"
sanitize  = true
upcase    = true

vault {
  address     = "https://vault-active.vault.svc:8200"
  renew_token = true
  ssl {
    enabled = true
    verify  = true
    ca_cert = "/run/secrets/kubernetes.io/serviceaccount/ca.crt"
    server_name = "vault.vault.svc"
  }
}

# /etc/envconsul.d/02-config.hcl
vault {
  k8s_auth_role_name = "my-role"
}
secret {
  path      = "database/creds/my-database-role"
  no_prefix = true
  format    = "DATABASE_{{ key }}"
}
secret {
  path      = "secrets/some/secret"
  no_prefix = true
  format    = "SOME_PREFIX_{{ key }}"
}

Command

envconsul -config /etc/envconsul.d -log-level=debug sleep infinity

Debug output

For demo purpose I set the Vault Kubernetes role token initial TTL to 60 seconds. Max TTL is 1 hour, so it has no effect on the outcome. I also tried with different initial TTL, max TTL and period from a few minutes to a few hours, the result is always the same: once the initial TTL expires, envconsul starts to log 403s.
https://gist.github.com/dizeee/7bfa522a44ccab6852ecb8557b414eaf

Expected behavior

Vault token is renewed before its TTL expires. It would also be nice to rotate the token when max TTL is reached.

Actual behavior

403 errors appear in the log after the token TTL expires.

References

• GH-274

[eikenb]

Hey @Dizeee, thanks for reporting this.

The k8ts auth support was added in a PR and I'm hoping the people who contributed might have an idea. @Kryvchun, @ygworldr... any thoughts?

[eikenb]

Re-reading this I'm wondering if this could be related to #310? That bug would impact this type of auth as well. I'm going to merge the fix for that today.
Hey @Dizeee.. any chance you could build from head and try that after I get it merged?

[aleskiontherun]

@eikenb Tried with 0.13.1 with the change you mentioned. The token is still not renewed when it expires. The only difference is that now envconsul doesn't reload static secrets when they are changed (I was surprised to see such a change in behavior in a minor update, TBO), so if it only reads static secrets, it never fails. But once a dynamic secret is about to expire, envconsul tries to renew it and, if the token is already expired at this point, it starts to log 403s.

[eikenb]

Thanks for the followup @Dizeee.

The only difference is that now envconsul doesn't reload static secrets when they are changed [..]

Sorry, being a bit dense... Would you mind expanding on what you mean by static secrets here? Thanks.

[aleskiontherun]

@eikenb By static I mean secrets stored in a KV backend (we use KV v2). And by dynamic - secrets that are generated and managed by Vault secret engines, like MySQL or RabbitMQ credentials.

[eikenb]

Thanks. Secrets stored using the KV values should always be monitored and kept up to date. The default is to check every 5 minutes.

[aleskiontherun]

@eikenb 0.13.0 checked every few seconds, I guess it was a bug. It's not related to my issue anyway.

[eikenb]

Yeah. That was issue #310... the reason I released 0.13.1. :)

[eikenb]

Sorry for the red herring and back to the original issue @Dizeee...

Re-reading your original post I've got 2 questions.

1. Is K8s required to reproduce this?
2. Could you share your Vault setup (or an approximation of it)?

Thanks.

[Kryvchun]

@Dizeee Can you please check this issue in my fork?

• hashicorp/consul-template#1750
• #332