Use case: Generate and store password immediately

[barskern]

This is a great plugin that I've recently added to my Vault setup, thank you for your great work! 😄

One use-case I would like to highlight, and ask about, is the possibility to generate and store the password immediately. There might be a better way of solving this, however to avoid and XY-problem I will explain my problem first, my solution and then ask for advice.

Problem

I have a dynamic infrastructure where Nomad administers all tasks into a cluster, where both Consul and Vault provide service discovery and secret management. As a part of these deployments I manage the credentials of numerous databases through Vault dynamic secrets (see here for the tutorial I followed). However as stated in that tutorial and in the docs page, it is best advised to give Vault a separate superuser than root, due to the fact that once root password rotation is done, one would no longer have any superuser access. Hence the challenge is to initialize the database with an initial root password which can be stored for later use in Vault.

My solution

To generate the password I use this plugin in a template in the nomad job file (similar to Vault Agent templates). As a part of this template code, I insert the generated password into Vault in the key-value storage. HOWEVER we have to be careful to not overwrite, so I only gave the postgres instance create permissions, and read permissions for the metadata (see policy rules below). The generated password will hence only be available the first time the container starts, then it will be cached in the database persistent data, and not accessible from the container itself.

The template script to do these shenanigans is complex (and attached below), because we have to ensure that we don't try to read a endpoint which doesn't exist (because secret causes and error which stops the template generation). Hence I cleverly used secrets to loop over an empty array if the endpoint doesn't exist, and if it exists we look through the metadata to check if the password is already stored.

{{- $generate_password := true -}}
{{- range secrets "secret/metadata/databases" -}}
	{{- if eq . "postgres_root_creds" }}
# Found the secret we searched for '{{ . }}'!
		{{- with printf "secret/metadata/databases/%s" . | secret -}}
			{{- if index .Data.versions (printf "%s" .Data.current_version) "destroyed" }}
# Secret destroyed, create a new one!
			{{- else }}
# Secret exists, hence should be cached by service
				{{- $generate_password = false -}}
			{{- end -}}
		{{- end -}}
	{{- else }}
# Secret '{{ . }}' was not what we were looking for
	{{- end -}}
	{{- else -}}
# Didn't find anything at all, generating credentials
{{- end -}}
{{- if $generate_password -}}
	{{- with secret "gen/password" "length=32" "symbols=0" -}}
		{{- if printf "password=%s" .Data.value | secret "secret/data/databases/postgres_root_creds" "username=postgres" }}
POSTGRESQL_POSTGRES_PASSWORD={{ .Data.value | toJSON }}
		{{- else }}
# ERROR! Unable to store generated credentials, using dummy value..
POSTGRESQL_POSTGRES_PASSWORD=insecure-password
		{{- end -}}
	{{- end -}}
{{- end -}}

rule {
  description  = "Allow dumping root database credentials to specified secret engine (BUT NOT OVERWRITE)"
  path         = "/secret/data/databases/+"
  capabilities = ["create"]
}

rule {
  description  = "Allow checking metadata to see if credentials exist"
  path         = "/secret/metadata/databases/+"
  capabilities = ["read"]
}

rule {
  description  = "Allow checking metadata to see if credentials exist"
  path         = "/secret/metadata/databases/*"
  capabilities = ["list"]
}

Conclusion

All-in-all, with this solution I'm able to generate a password which is only accessible the first time the container starts, which is then stored internally in the persistent storage of the container and in vault itself.

Ask for advice

1. How do you bootstrap servers with root passwords which we do not want to expose?
2. Perhaps this plugin could somehow write the value directly to the KV-store on request?
3. Should I perhaps go the "insecure password hardcoded at initialization which is rotated when the service is up" route?

[sethvargo]

with this solution I'm able to generate a password which is only accessible the first time the container starts

Have you considered using cubbyhole instead? That seems like a better approach since it has the semantics you want (delete on read) built in: https://www.vaultproject.io/docs/secrets/cubbyhole.

1. How do you bootstrap servers with root passwords which we do not want to expose?

Do you need the root passwords later? My proposal would be to randomly generate the password and, if provisioning fails for some reason, THEN write the password somewhere to be able to use for debugging.

2.\ Perhaps this plugin could somehow write the value directly to the KV-store on request?

Plugins have no way of communicating with each other, by design. There's no way for plugin A to communicate with plugin B sans shelling out to the system or calling the Vault API (which is probably doesn't have authorization to do). It's also invalid to assume that the KV store is even mounted/available.

3. Should I perhaps go the "insecure password hardcoded at initialization which is rotated when the service is up" route?

I'm a bit unclear why you need to store "boot" password.

[barskern]

Thank you for the quick and through reply!

Have you considered using cubbyhole instead? That seems like a better approach since it has the semantics you want (delete on read) built in: https://www.vaultproject.io/docs/secrets/cubbyhole.

Yes, I did consider this, however since it will only be u accessible to the token that created it, it will then only be accessible to the token given to the nomad cluster. Hence I would not have access to it, though it does have the semantics I'm looking for.

Do you need the root passwords later? My proposal would be to randomly generate the password and, if provisioning fails for some reason, THEN write the password somewhere to be able to use for debugging.

The only reason one would need the root password for later is if there are any maintenance that needs to be performed on the database. Further if, as an admin, I need to access the database, create a new superuser, perform other admin jobs, I need some way to get credentials to it. As stated in the tutorial for [database root password rotation][1], one should preferably create a separate superuser just for vault: NOTE: Once the root credential was rotated, only the Vault knows the new root password. This is the same for all root database credentials given to Vault. Therefore, you should create a separate superuser dedicated to the Vault usage which is not used for other purposes. [1]: https://learn.hashicorp.com/tutorials/vault/database-root-rotation#step-3-rotate-the-root-credentials

Plugins have no way of communicating with each other, by design. There's no way for plugin A to communicate with plugin B sans shelling out to the system or calling the Vault API (which is probably doesn't have authorization to do). It's also invalid to assume that the KV store is even mounted/available.

That makes sense in the context of plugins, as much isolation as possible.

I'm a bit unclear why you need to store "boot" password.

I would like to have a superuser access to the database from outside of vault, to do potential maintenance and governance of the database manually. This type of access and control on a database that is created dynamically is perhaps a bit too much to ask anyways. However I would think there would be a elegant solution to the problem of initializations. As a thought experiment, I was thinking that perhaps one could create a database role which grants superuser access to the database. Hence people with the right permissions could then generate a set of root credentials when needed, though one would have to be *really* careful not to give access to that path through a policy. It seems a bit too dangerous though.

[sethvargo]

This type of access and control on a database that is created dynamically is perhaps a bit too much to ask anyways. However I would think there would be a elegant solution to the problem of initializations.

As a thought experiment, I was thinking that perhaps one could create a database role which grants superuser access to the database. Hence people with the right permissions could then generate a set of root credentials when needed, though one would have to be really careful not to give access to that path through a policy. It seems a bit too dangerous though.

This is how I set this up personally, and how I've recommended customers to configure things in the past.

[barskern]

That makes sense, and it does seem a lot more elegant to do. Thank you for the swift feedback and valuable answers! Feel free to close issue.

[github-actions]

This issue has been automatically locked since there has not been any
recent activity after it was closed. Please open a new issue for
related bugs.