Renewing AWS Backend credentials (or getting new ones)
Blackbaud-KevinHutson opened this issue · 29 comments
Hi there,
I'm trying to use spring-cloud-vault-config with an AWS backend. It works great on Spring startup, hitting the AWS backend and giving me an initial set of credentials. But the minute it tries to renew the lease it fails. After talking with my Vault administrator, he pointed me here:
According to Vault's "Lease, Renew and Revoke" section (https://www.vaultproject.io/docs/concepts/lease.html) it states "For example, with the AWS secret backend, the access keys will be deleted from AWS the moment a secret is revoked. This renders the access keys invalid from that point forward."
So, I'm trying to understand what my options are. He's giving me a TTL of only a few minutes for the S3 access for those cred's. I'm going to need to go back and get new cred's periodically. If I can't get the lease renewed, what can I do? It appears that spring-cloud-vault-config only hits that original endpoint on startup. Is there some other config or switch I can utilize that would get me new credentials? Or do I need to manually write something in Spring that hits that Vault endpoint (which would seem to defeat the purpose of this library)? Any thoughts?
Example error that I'm getting back: org.springframework.vault.client.VaultException: Cannot renew lease: Status 403 URI https://host:8200/v1/sys/renew/aws-dev/creds/S3FullAccess/bcad60d8-aad7-b01b-ee93-005eb67c4a56: permission denied
I'm using the configuration stated here:
http://cloud.spring.io/spring-cloud-vault-config/spring-cloud-vault-config.html#vault.config.backends.aws
with something like this in bootstrap.properties
spring.cloud.vault.aws.enabled=true
spring.cloud.vault.aws.role=S3FullAccess
spring.cloud.vault.aws.backend=aws-dev
spring.cloud.vault.token=token
spring.cloud.vault.host=host
spring.cloud.vault.port=8200
spring.cloud.vault.scheme=https
spring.cloud.vault.connection-timeout=5000
spring.cloud.vault.read-timeout=15000
Thanks so much! I feel like I must be missing something obvious.
I too, would like to see an answer for this. Thank you.
A short TTL is something you're looking for to renew the lease periodically, as long as your application is running. Spring Cloud Vault Config is lease-aware and renews leases by default (spring.cloud.vault.config.lifecycle.enabled=true) if leases are renewable. Renewing leases requires your token to be able to access /sys/renew which is a policy setting of your token (login role).
The message you get (permission denied) indicates that you're not authorized to renew tokens.
You don't want to obtain new credentials all over because you would be required to propagate the new credentials to various places without actually knowing, where these credentials are really used. Obtaining new credentials creates a lot of new users and Spring Cloud Vault Config exposes credentials just to the Environment. Properties are picked up from there by configuration components and are be held in components that are not under control of Spring Cloud Vault Config.
Thanks much @mp911de I'm asked my Vault admin to fix access to /sys/renew that and now I've moved from a 403 to a 400 during renewal. I'm probably missing 1 more thing. I will take that up tomorrow with them.
One more question if you don't mind. I cranked up DEBUG and see a lot of these. The first one starts with 360. Is that the lease duration coming back from Vault? Again, I don't think this is something I need to specify in spring-cloud-vault-config.
2017-02-28 22:59:30.746 INFO [ main] o.s.c.vault.config.VaultConfigTemplate : Fetching config from Vault at: https://myHost:8200/v1/aws-dev/creds/S3FullAccess
2017-02-28 22:59:30.751 DEBUG [ main] o.s.c.v.c.LeasingVaultPropertySource : Lease aws-dev/creds/S3FullAccess/c2cab04b-d314-2673-6447-ca2beca07560 qualified for renewal
2017-02-28 22:59:30.752 DEBUG [ main] aultPropertySource$LeaseRenewalScheduler : Scheduling renewal for lease aws-dev/creds/S3FullAccess/c2cab04b-d314-2673-6447-ca2beca07560, lease duration 360
...then..
2017-02-28 23:05:43.407 DEBUG [pool-1-thread-1] MyService : Beginning to stream files.
2017-02-28 23:09:30.796 DEBUG [g-Cloud-Vault-1] aultPropertySource$LeaseRenewalScheduler : Renewing lease aws-dev/creds/S3FullAccess/c2cab04b-d314-2673-6447-ca2beca07560
2017-02-28 23:09:30.811 DEBUG [g-Cloud-Vault-1] o.s.c.v.c.LeasingVaultPropertySource : Lease aws-dev/creds/S3FullAccess/c2cab04b-d314-2673-6447-ca2beca07560 qualified for renewal
2017-02-28 23:09:30.811 DEBUG [g-Cloud-Vault-1] aultPropertySource$LeaseRenewalScheduler : Scheduling renewal for lease aws-dev/creds/S3FullAccess/c2cab04b-d314-2673-6447-ca2beca07560, lease duration 119
2017-02-28 23:10:29.811 DEBUG [g-Cloud-Vault-2] aultPropertySource$LeaseRenewalScheduler : Renewing lease aws-dev/creds/S3FullAccess/c2cab04b-d314-2673-6447-ca2beca07560
2017-02-28 23:10:29.823 DEBUG [g-Cloud-Vault-2] o.s.c.v.c.LeasingVaultPropertySource : Lease aws-dev/creds/S3FullAccess/c2cab04b-d314-2673-6447-ca2beca07560 qualified for renewal
2017-02-28 23:10:29.824 DEBUG [g-Cloud-Vault-2] aultPropertySource$LeaseRenewalScheduler : Scheduling renewal for lease aws-dev/creds/S3FullAccess/c2cab04b-d314-2673-6447-ca2beca07560, lease duration 60
2017-02-28 23:10:39.825 DEBUG [g-Cloud-Vault-1] aultPropertySource$LeaseRenewalScheduler : Renewing lease aws-dev/creds/S3FullAccess/c2cab04b-d314-2673-6447-ca2beca07560
2017-02-28 23:10:39.841 DEBUG [g-Cloud-Vault-1] o.s.c.v.c.LeasingVaultPropertySource : Lease aws-dev/creds/S3FullAccess/c2cab04b-d314-2673-6447-ca2beca07560 qualified for renewal
2017-02-28 23:10:39.850 DEBUG [g-Cloud-Vault-1] aultPropertySource$LeaseRenewalScheduler : Scheduling renewal for lease aws-dev/creds/S3FullAccess/c2cab04b-d314-2673-6447-ca2beca07560, lease duration 50
2017-02-28 23:10:43.496 DEBUG [pool-1-thread-1] MyBucketPathResolver : Bucket Key Prefix, springApplicationName=myApp
2017-02-28 23:10:43.497 INFO [pool-1-thread-1] MyService : Begin scan for tombstones. bucket=myBucket, pendingBucketPath=int-apps/myApp/pending
2017-02-28 23:10:43.497 DEBUG [pool-1-thread-1] MyService : Beginning to stream files.
2017-02-28 23:10:49.851 DEBUG [g-Cloud-Vault-1] aultPropertySource$LeaseRenewalScheduler : Renewing lease aws-dev/creds/S3FullAccess/c2cab04b-d314-2673-6447-ca2beca07560
2017-02-28 23:10:49.862 DEBUG [g-Cloud-Vault-1] o.s.c.v.c.LeasingVaultPropertySource : Lease aws-dev/creds/S3FullAccess/c2cab04b-d314-2673-6447-ca2beca07560 qualified for renewal
2017-02-28 23:10:49.863 DEBUG [g-Cloud-Vault-1] aultPropertySource$LeaseRenewalScheduler : Scheduling renewal for lease aws-dev/creds/S3FullAccess/c2cab04b-d314-2673-6447-ca2beca07560, lease duration 40
2017-02-28 23:10:59.865 DEBUG [g-Cloud-Vault-2] aultPropertySource$LeaseRenewalScheduler : Renewing lease aws-dev/creds/S3FullAccess/c2cab04b-d314-2673-6447-ca2beca07560
2017-02-28 23:10:59.886 DEBUG [g-Cloud-Vault-2] o.s.c.v.c.LeasingVaultPropertySource : Lease aws-dev/creds/S3FullAccess/c2cab04b-d314-2673-6447-ca2beca07560 qualified for renewal
2017-02-28 23:10:59.887 DEBUG [g-Cloud-Vault-2] aultPropertySource$LeaseRenewalScheduler : Scheduling renewal for lease aws-dev/creds/S3FullAccess/c2cab04b-d314-2673-6447-ca2beca07560, lease duration 30
2017-02-28 23:11:09.889 DEBUG [g-Cloud-Vault-2] aultPropertySource$LeaseRenewalScheduler : Renewing lease aws-dev/creds/S3FullAccess/c2cab04b-d314-2673-6447-ca2beca07560
2017-02-28 23:11:09.901 DEBUG [g-Cloud-Vault-2] aultPropertySource$LeaseRenewalScheduler : Scheduling renewal for lease aws-dev/creds/S3FullAccess/c2cab04b-d314-2673-6447-ca2beca07560, lease duration 20
2017-02-28 23:11:09.901 DEBUG [g-Cloud-Vault-2] o.s.c.v.c.LeasingVaultPropertySource : Lease aws-dev/creds/S3FullAccess/c2cab04b-d314-2673-6447-ca2beca07560 qualified for renewal
2017-02-28 23:11:19.901 DEBUG [g-Cloud-Vault-2] aultPropertySource$LeaseRenewalScheduler : Renewing lease aws-dev/creds/S3FullAccess/c2cab04b-d314-2673-6447-ca2beca07560
2017-02-28 23:11:19.913 DEBUG [g-Cloud-Vault-2] o.s.c.v.c.LeasingVaultPropertySource : Lease aws-dev/creds/S3FullAccess/c2cab04b-d314-2673-6447-ca2beca07560 qualified for renewal
2017-02-28 23:11:19.913 DEBUG [g-Cloud-Vault-2] aultPropertySource$LeaseRenewalScheduler : Scheduling renewal for lease aws-dev/creds/S3FullAccess/c2cab04b-d314-2673-6447-ca2beca07560, lease duration 10
2017-02-28 23:11:29.914 DEBUG [g-Cloud-Vault-2] aultPropertySource$LeaseRenewalScheduler : Renewing lease aws-dev/creds/S3FullAccess/c2cab04b-d314-2673-6447-ca2beca07560
2017-02-28 23:11:29.926 DEBUG [g-Cloud-Vault-2] o.s.c.v.c.LeasingVaultPropertySource : Lease aws-dev/creds/S3FullAccess/c2cab04b-d314-2673-6447-ca2beca07560 qualified for renewal
2017-02-28 23:11:29.926 DEBUG [g-Cloud-Vault-2] aultPropertySource$LeaseRenewalScheduler : Scheduling renewal for lease aws-dev/creds/S3FullAccess/c2cab04b-d314-2673-6447-ca2beca07560, lease duration 0
2017-02-28 23:11:39.927 DEBUG [g-Cloud-Vault-2] aultPropertySource$LeaseRenewalScheduler : Renewing lease aws-dev/creds/S3FullAccess/c2cab04b-d314-2673-6447-ca2beca07560
2017-02-28 23:11:39.941 ERROR [g-Cloud-Vault-2] aultPropertySource$LeaseRenewalScheduler : Cannot renew lease aws-dev/creds/S3FullAccess/c2cab04b-d314-2673-6447-ca2beca07560
org.springframework.vault.client.VaultException: Cannot renew lease: Status 400 URI https://myHost:8200/v1/sys/renew/aws-dev/creds/S3FullAccess/c2cab04b-d314-2673-6447-ca2beca07560: lease not found or lease is not renewable
Cheers!
Had a quick thought. I know how to curl this manually.
curl -k -X GET -H "x-Vault-Token:MyToken" https://vaultHost:8200/v1/aws-dev/creds/S3FullAccess
{"request_id":"someUuid","lease_id":"aws-dev/creds/S3FullAccess/someUuid","renewable":true,"lease_duration":360,"data":{"access_key":"key","secret_key":"secret","security_token":null},"wrap_info":null,"warnings":null,"auth":null}
So, 360 is indeed coming from Vault. And renewable is true.
Just not sure why it won't renew.
Good times.
Your lease has a ttl (lease time) = 360 and max_ttl (max_lease) = 360 set and that's why you can't renew the lease beyond 360 seconds. The log output shows a decreasing TTL. Usually, the ttl is short and the max_ttl is long (a month or so). Max TTL is intended as hard limit to enforce credential rotation.
My Vault admin is telling me "Right now, the default TTL is set to 360 seconds and the Max TTL is set to 720 seconds". And he's saying that if it doesn't know how to request new creds when it can't extend the lease, then it will just break after whatever the max TTL is. Is that true? Or am I misunderstanding the difference between renewing a lease and getting new cred's? Remember that this is for the AWS backend and those credentials are temporary not static. The docs in Vault themselves get revoked at some point.
In addition to renewals, a lease can be revoked. When a lease is revoked, it invalidates that secret immediately and prevents any further renewals. For dynamic secrets, the secrets themselves are often immediately disabled. For example, with the AWS secret backend, the access keys will be deleted from AWS the moment a secret is revoked. This renders the access keys invalid from that point forward. from https://www.vaultproject.io/docs/concepts/lease.html
I'm going to play with this locally more today now that S3 is up :-)
720 seconds max ttl means that after two renewals (720 seconds after creating) AWS credentials, they will expire and will no longer be valid.
The Java AWS SDK uses a AWSCredentialsProvider to obtain credentials. I think you could take advantage of that API in combination with VaultTemplate but then you would lose lease renewal and the convenient injection of credentials. You can obtain AWS credentials yourself and expose these through AWSCredentialsProvider.
We will provide a user API in Spring Vault to work with renewable leases through Spring's Environment but we're not quite there yet.
Thanks again. Soo. This statement: "Max TTL is intended as hard limit to enforce credential rotation." I was just asked this. Does spring-cloud-vault-config handle key or credential rotation? If I set my max_ttl to a month (as you suggested), what do I do then? Reboot my app? :-) I think we are missing something here.
No, there's no rotation support here. The approach is to restart the application (or use Spring Cloud's @RefreshScope. The reason is similar to why we can't rotate AWS credentials. A connection pool that uses an existing set of credential might cause unforeseen consequences if updated with new credentials, while connections are use and the old credentials expire on the server side.
Ah. Thank you. Now it's more clear to us. We didn't know you didn't support key rotation.
There are at least some use cases where refreshing those credentials should not pose an issue. Our question is do you have any plans to support key rotation at all in the project? We need key rotation either way (especially for S3) and are not comfortable with using @RefreshScope or bouncing our apps on a schedule.
If we could continue this conversation with someone planning your project roadmap , we might be able to contribute an optional key rotation integration back into this project if that is desirable? While we could write our own solution, I think there's a lot to be gained by utilizing the existing Vault integration and the lease renewal features that are already there. We just need key rotation too. Would this be interesting to this project?
Key rotation is rather simple if you look only on lease/renewal/expiry. The complexity comes from the context: For Spring Cloud Vault (actually it's related to Spring Boot), you need to see how components are wired. Spring Boot initializes its components on startup and gathers configuration from various sources (application.properties, System Properties, Environment Variables, Vault, not necessarily in that order).
Boot applies the configuration to the components it bootstraps (i.e. copies the values obtained from Environment into configuration objects) on startup. These components do their things and use the configuration set by Spring Boot and no longer have a relation to the property source they were obtained from. Renewing a value/obtaining new credentials and put them into the Environment has no effect because Spring Boot does not re-initialize the bootstrapped components. The only way to work around this is using @RefreshScope by invalidating the object, so it's recreated on next use obtaining the current configuration.
If you limit credential rotation to a particular component (AWSCredentialProvider, a Basic-Auth provider for an HTTP client, …) that you have under control, it's possible to rotate credentials. There are stateful components (JDBC connection pools, Connections to message brokers) that don't have a clear boundary (like an HTTP client with request/response usage boundaries) so credential rotation would create harm to that components.
You can build key rotation for your specific problem using Spring Vault that gives you access to the Vault API. Ephemeral configuration isn't supported beyond @RefreshScope. We'll learn over time which components are safe to credential rotation and might add specific adapters to propagate benefits we get from Vault. However, we're still at the beginning.
@Blackbaud-KevinHutson can you explain why you "are not comfortable with using @RefreshScope"?
/cc @dsyer
FWIW: I pushed a draft of SecretLeaseContainer (see https://github.com/mp911de/spring-vault/tree/issue/50-lease-and-renew) to provide a managed container for secret retrieval with lease renewal and secrets rotation (request a secret once the lease hits max ttl). The difference to the current PropertySource is that SecretLeaseContainer is event-driven so a consumer can subscribe to secret creation/renewal/expiration events.
SecretLeaseContainer container = new SecretLeaseContainer(vaultOperations, taskScheduler);
final RequestedSecret requestedSecret = container.requestRotatingSecret("mysql/creds/my-role");
container.addLeaseListener(new LeaseListenerAdapter() {
public void onLeaseEvent(LeaseEvent leaseEvent) {
if (requestedSecret == leaseEvent.getSource()) {
if (leaseEvent instanceof LeaseCreatedEvent) {
}
if (leaseEvent instanceof LeaseExpiredEvent) {
}
}
}
});RefreshScope already has an event-driven model for updating credentials. Why do we need another model for the same feature?
RefreshScope is Spring Cloud specific, we should be able to do something similar in non-Boot/non-Cloud applications.
Vault lease renewal is intrinsic with regard to scheduled lease renewals as long as the application runs. Lease renewal is limited to Vault and specific to Vault. Renewing leases does not refresh the configuration, it's keeping the Vault lease alive, extending TTL. It does not require configuration refresh because credentials stay the same. It makes sense in that context to notify users about renewal, expiry and possible errors that happen. Only in the last stage of Vault leases, obtaining new secrets after the lease expired it makes sense to refresh credentials scoped to just the expired secret leaving other secrets untouched.
IIRC refreshing @RefreshScope refreshes all refresh-scoped beans. Triggering refresh requires either an external ping, a scheduler or a watcher (likeConsulConfigurationListener) to trigger refresh.
Maybe there is something at Framework level that we could do to combine aspects from RefreshScope with a tight focus to selectively update beans.
OK, so I can understand the need for a vault-specific event to refresh itself (so the client can continue to contact the server). If we restrict the focus to that, and also emit an ApplicationEvent, then Spring Cloud (and others) can listen for that and refresh if necessary. Is that the right way to think about it?
IIRC refreshing @RefreshScope refreshes all refresh-scoped beans
I don't think that's technically necessary - it might be the way the listener that looks for EnvironmentRefreshedEvent works, but if so then it's an implementation detail.
Yes, that's the right way to think about it. Spring Vault LeaseEvents extend ApplicationEvent and can be published through the ApplicationContext. Right now they are just published to registered listeners to reduce the scope but events can be broadcasted that way to notify others.
@spencergibb Sorry! It's been long time and I forgot to respond. Our team, after much discussion, overcame their issue with RefreshScope. That still seems to be the best option we have right now. So, we aren't blocked by this particular issue now. Thanks again!
Thanks for your response. We're not finally decided on how to proceed with credential rotation for the integrations (AWS, Databases, …). I'm closing this ticket for now and we'll revisit the topic at a later time.
I am running into the same issue using a database backend. Unable to get new creds once the lease expires
@gigenthomas Use SecretLeaseContainer to obtain and reconfigure your DataSource.
@mp911de - Wondering if you could provide some additional guidance on how to use SecretLeaseContainer ? Using a postgres backend
What am I doing for the LeaseExpiredEvent ?
`
@OverRide
public void afterPropertiesSet() throws Exception {
final RequestedSecret secret = RequestedSecret.renewable("postgres/creds/readonly");
leaseContainer.addLeaseListener(new LeaseListener() {
@Override
public void onLeaseEvent(SecretLeaseEvent secretLeaseEvent) {
if (secretLeaseEvent instanceof LeaseExpiredEvent) {
????
}
}
});
leaseContainer.addRequestedSecret(secret);
}
`
Thanks in advance !!
@gigenthomas Please post your question on StackOverflow.
Hi guys, I implemented Lease Rotation for MySQL (Hikari JDBC pool) using Spring Vault or Spring Cloud Vault and SecretLeaseContainer. If there is someone interested, please have a look at https://github.com/ivangfr/springboot-vault-examples. Thanks!
Hi guys, I implemented Lease Rotation for
MySQL(Hikari JDBC pool) usingSpring VaultorSpring Cloud VaultandSecretLeaseContainer. If there is someone interested, please have a look at https://github.com/ivangfr/springboot-vault-examples. Thanks!
Looks good, but be careful. Your solution will work only with Hikari JDBC pool 3.2.0+ See here
Also, how did you fix issue with
org.springframework.vault.VaultException: Cannot renew lease: lease not found
By the way, solution works for PostgreSQL too.
Hey @frombrest
Looks good, but be careful. Your solution will work only with Hikari JDBC pool 3.2.0+
You are right. My solution is very specific.
Also, how did you fix issue with
org.springframework.vault.VaultException: Cannot renew lease: lease not found
Sorry, but with just this line, it would be difficult to help you. If you have a small sample that I could clone and try to reproduce on my machine, I would be glad to help.
@ivangfr Don't worry... I've fixed my issue:
#Sets the duration that is at least required before renewing a lease.
spring.cloud.vault.config.lifecycle.min-renewal=10
#Sets the expiry threshold. A lease is renewed the configured period of time before it expires.
#Should be less than min-renewal period because we a going to use unrenewed credentials.
spring.cloud.vault.config.lifecycle.expiry-threshold=5
Thank you for your solution.
I've written a blog post about how to ensure that expiring Spring Cloud Vault dynamic database secrets are renewed, when reaching Hashicorp Vault's max_ttl: Hashicorp Vault max_ttl Killed My Spring App
There will be a follow-up using some more insights and inspirations e.g. from @ivangfr's proposal.
The follow-up post about how to rotate expiring relational Spring Cloud Vault database credentials without downtime is available: Heavy Rotation of Relational Hashicorp Vault Database Secrets in Spring