Lease TTL (with dynamic secrets Vault backend)

[zarko-a]

Hi there,

I started testing KES and it works quite well with static KV secrets (where retrieved secret is always the same). But as soon as I tried to use it with AWS secret engine from HashiCorp Vault I noticed a potential problem.

KES reads the secret based on the POLLER_INTERVAL_MILLISECONDS which defaults to 10s. HashiCorp Vault provisions AWS IAM user each time a secret is read and this causes multiplication of IAM users. In our specific case, we also need to restart the application pod to reload the secret, so it's not really practical to do it frequently.

TLDR:

1. I'm wondering if it's possible for KES to utilize lease TTL information for secret polling. Is this supported?
2. Is KES even meant to be used with dynamically provisioned AWS credentials (or only with static)?
3. Can KES use lease lookup to query if lease is active instead of polling the secret? Specifically Vault API provides lease lookup with expiry information. https://www.vaultproject.io/api/system/leases

I realize I can increase the global polling interval, but then rotating the credentials on demand would require a change to that value, at least temporarily. Also, if polling is missed due to some outage, the leases can expire without KES being aware and impacting applications.

[moolen]

Are you refering to this feature: https://www.vaultproject.io/docs/secrets/aws ?

So, you use vault to generate an IAM users ? That's not a use-case previously covered with KES. Generating AWS IAM users is (generally) considered an anti-pattern (there may be exceptions to this rule).

I'm wondering if it's possible for KES to utilize lease TTL information for secret polling

No, that's not supported. every secret/poller iteration get it's own new, fresh client to fetch secrets from.

Is KES even meant to be used with dynamically provisioned AWS credentials

If you're running on AWS there's no need to use static credentials, just attach an IAM Roles to KES. When running in EKS: you should use IRSA.

Can KES use lease lookup to query if lease is active instead of polling the secret?

I'm not sure if i correctly understand you use-case. I think supporting the AWS secrets engine in Vault would essentially be a new secret provider (RN we support the Vault KV engine only). To answer your question: sure, the new provider should preferably use the lease semantics!

[zarko-a]

Thanks for the reply @moolen

Are you refering to this feature: https://www.vaultproject.io/docs/secrets/aws ?

Yes.

So, you use vault to generate an IAM users ? That's not a use-case previously covered with KES. Generating AWS IAM users is (generally) considered an anti-pattern (there may be exceptions to this rule).

Yes, we generate IAM users and associated keys (keys are the main goal). Maybe I'm missing something, but if we have an application that runs outside AWS and needs access to AWS I don't see many ways of providing access. We are using Vault's AWS secrets engine to provision dynamic/ephemeral credentials for apps that don't have native Vault support.

I'm not sure if i correctly understand you use-case. I think supporting the AWS secrets engine in Vault would essentially be a new secret provider (RN we support the Vault KV engine only). To answer your question: sure, the new provider should preferably use the lease semantics!

Primary focus of this use case is providing AWS dynamic credentials to an app that has no Vault awareness. Hence we started looking into KES to update existing Kubernetes secrets automatically.
Thanks for confirming this is not really supported, I thought I was missing something.

A disappointed that only KV engine is supported in KES considering the main advantage of Vault lies in dynamically provisioned credentials.

[vaidik]

There is a use-case for lease based rotating credentials when using databases such as postgres. Vault can be configured to generate credentials when the lease is about to expire. Can KES handle that in a graceful way that credentials generated after the lease expires - the new credentials are exposed to pods transparently and there is a way to gracefully reload the application or restart the pods?

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[derbeneviv]

+1 to the request.
Common use-cases are:

dynamodb credentials (aws secret engine)
database credentials (https://www.vaultproject.io/docs/secrets/databases secret engine)
SSH keys (https://www.vaultproject.io/docs/secrets/ssh)

well, mostly all places, where HashiVault is used for dynamic secret generation

[Flydiverny]

See #864
That said a PR would still be possible

[vaidik]

Is this still being tracked actively for ESO? I think this will be a great feature to have. Literally the only blocker right now for us to roll out ESO in my company.

[justinas-b]

+1 interested in this

[Flydiverny]

Is this still being tracked actively for ESO? I think this will be a great feature to have. Literally the only blocker right now for us to roll out ESO in my company.

I think it's best to make a new issue in ESO repo if this isn't available in ESO!

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions]

This issue was closed because it has been stalled for 30 days with no activity.