Supress Vault secret existence checks / "Failed secret modification" K8s events
Opened this issue · 2 comments
Hello, thank you for vault-crd
, it's been a really useful component in our stack!
We've run into a slight issue with getting a lot of FailedModification
K8s events, which are of type Failure
, whereas it doesn't seem anything "wrong" is happening. It's not a big problem, but it obscures other important K8s events in our monitoring.
Repro steps:
- Create KV2 secret in Vault
- Deploy
Vault
K8s resource with corresponding Vault KV2 secret path - Remove the Vault KV2 secret, without touching
.yaml
definitions in K8s (Vault
resource stays unchanged)
Not, every now and then, vault-crd
will refresh, checking for Vault secret existence and state. If it's not there, it brings up this notification:
This seems sensible, since the secret is not present. However, I believe our use case is a bit less obvious.
Our use case – default secrets + override
We have multiple development environments of the same apps. For convenience, we share a _default
KV2 secret in Vault for each service, which contains all default ENV VARs for the service. Then, if a developer wants to override them or add new, they can create a new Vault secret, specific to the environment. For example:
---
apiVersion: "koudingspawn.de/v1"
kind: Vault
metadata:
name: "kv-myservice-default-secrets"
spec:
path: "kv-myservice/_default"
type: "KEYVALUEV2"
changeAdjustmentCallback:
type: deployment
name: "myservice"
{{ end }}
# Vault secrets for env (override the default-secrets)
---
apiVersion: "koudingspawn.de/v1"
kind: Vault
metadata:
name: "kv-myservice-env-secrets"
spec:
path: "kv-myservice/myenvname"
type: "KEYVALUEV2"
changeAdjustmentCallback:
type: deployment
name: "myservice"
Then in Deployment, we do this:
envFrom:
- secretRef:
name: "kv-myservice-default-secrets"
- secretRef:
name: "kv-myservice-env-secrets"
optional: true
{{- end }}
—
So in our situation, it seems that having a Vault
resource with no correspondent Vault secret is an "acceptable" state (although only for override secrets; the _default
should always have a corresponding Vault secret).
Is it possible to somehow supress these events? Perhaps I missed something in the documentation. Otherwise, would you consider extending the resource API to cover such a use case? For example:
apiVersion: "koudingspawn.de/v1"
kind: Vault
metadata:
name: "kv-myservice-env-secrets"
spec:
path: "kv-myservice/myenvname"
checkVaultSecretExistence: false
type: "KEYVALUEV2"
changeAdjustmentCallback:
type: deployment
name: "myservice"
( checkVaultSecretExistence: false
)
It's just an example API I've come up with now, I'm sure there is a better name.
Env info
vault-crd version: 1.6.3
k8s version: 1.24 (EKS)
Hi @awinecki ,
To get it right your expectation is that we don't send out events when a secret is not available and "ignore" them simply?
So more like a "supress warning"?
@DaspawnW exactly.
I was suggesting the ability to disable the secret existence check. But suppressing the warning/log might be even better.
So something like:
apiVersion: "koudingspawn.de/v1"
kind: Vault
metadata:
name: "kv-myservice-env-secrets"
spec:
path: "kv-myservice/myenvname"
suppressFailedModificationLogs: true
type: "KEYVALUEV2"
changeAdjustmentCallback:
type: deployment
name: "myservice"