Spike: Investigate better options for secret management

[bmonkman]

Right now we give people a kustomize manifest to create a secret from a .env file but that leaves it to them to manage the contents of the env file which is not ideal.
Let's see if we can find a better solution that enables collaboration while emphasizing secure processes.
If we can use this same method for the initial setup of the project too that would be nice.

[bmonkman]

I'm leaning toward using external-secrets which basically syncs a secret from AWS Secrets Manager to a k8s secret, configured using a k8s custom resource.

The setup would be:

Create a secret where the keys and values will become the keys and values of the k8s secret. Maybe prefix with the environment:

aws secretsmanager create-secret --name "application/stage/app-name" --secret-string '{"API_KEY": "1234"} --description "k8s secret for app-name stage" --tags '[{"Key":"Environment","Value":"stage"}]''

Access can be restricted to only devs or operators allowed to view/edit application secrets, and the external-secrets pod which does the syncing.

Define the custom resource in the application code. For example, in backend-go we could have something like this that would get deployed normally with the application in the CI pipeline:
/kubernetes/base/external-secret.yaml

apiVersion: "kubernetes-client.io/v1"
kind: ExternalSecret
metadata:
  name: app-name
spec:
  backendType: secretsManager
  dataFrom:
  - application/stage/app-name

After this gets deployed, the secret will be created and kept in sync by the controller (by polling the secret value. Default interval is 10 seconds.)
When the secret is referenced using envFrom in the deplyoment spec, all the keys from the AWS secret will become env vars available to the pod.

We could also potentially use this instead of the code we have written to copy AWS secrets into k8s secrets for things like VPN PK, Cloudfront keypair, etc.

To edit a secret someone would have to update it through the CLI or AWS web UI. Editing through the CLI might not be a great experience though, I wonder if there's tooling to help.

A normal case might be to edit the secret, add a K/V pair, which would then be synced, then deploy the new version of the service which uses the new key.
Since editing the secret is an out-of-band procedure, there's some amount of user-error that's possible due to timing. For example, deploying before adding the K/V pair to the secret.

Cons:

• An additional pod running in the cluster
• Not free: about $1 / month / application

One other option I considered was Mozilla's sops which could work but I'm hesitant to use it because I'm a bit nervous about the idea of storing encrypted secrets in git (even though sops does a lot to mitigate the dangers) and because the CI pipeline would need access to decrypt the secrets. It would also be another tool users would have to have installed.

[davidcheung]

Question: does it merge AWS key value pairs into the secret? or just replaces the entire k8s-secret (can users tag alone non-conflicting keys in there)

Yeah having a single source of truth is definitely much nicer, right now it could be hard to troubleshoot if the k8s secret ends up wrong or mal-formatted. We just need to encourage users to build the habit to change it in the webconsole, and know if they update the k8s secret it will auto change back.

[bmonkman]

No, it syncs the entire contents to the secret. If you wanted something that users could edit in k8s you could make another secret and then just reference them both in the deployment. Not sure there would be much point in that though as you'd be missing out on the nice single source of truth.

[benjaminkvm]

I really like this approach; I think it is pretty important to have a solid secrets management solution out of the gate, even if it is not perfect -- so I am happy to see this! Don't let perfect be the enemy of good and all :)

I think this solution is definitely preferable for most startups over running a dedicated service like Vault, which can incur significant expense and has a stronger learning curve. Hadn't heard of external-secrets before, but it looks pretty clean!

Curious if you would provision the secrets with the Terraform provider? If so, we could maybe add some documentation around that / the process.