Rethink Vault Integration Implementation

[lawliet89]

There are several current (and future) integration in the user_data scripts that require reading secrets from Vault. We might want to consider refactoring this to better support Vault operations. In particular, many applications do not know the existence of Vault and may fail to renew the lease on their secrets.

This is usually not a problem for secrets passed to Nomad (except for the future Consul ACL token) because Nomad will update the secrets as needed.

Steps the user_data script should perform:

1. Check with Consul if Vault is up and the KV store to see if AWS authentication is enabled for the type of servers.
2. Get a Vault Token (via vault login with AWS auth) and save it to the usual spot at ~/.vault_token. This will be done as the Root user so /root/.vault_token.
3. Change existing Nomad Vault integration to derive a Token from this Token to pass to Nomad instead of using the vault login token.
4. Get additional secrets from Vault if needed (e,g, a future Consul ACL token)
5. Set up a cron job to renew the lease for the login token and any other secrets. Currently, only the token passed to Nomad for Vault integration will automatically get renewed by Nomad.

We might want to consider using consul_template to render and manage the renewal of these tokens.

Open Questions:

• What if we were to enable ACL in Consul and then have Vault provide a Consul secrets backend to provide a Consul ACL. Won't this cause a chicken and egg problem when we try to discover Vault? Possibilities: anonymous tokens or agent specific default token