external-secrets/kubernetes-external-secrets

Application stops functioning after a few days

Closed this issue · 2 comments

jlact commented

Application seems to just stop functioning, usually after a few days. No action is taken to stop or terminate deployment. As far as kubernetes is concerned the pod is still running, and it does indeed respond to calls on the metrics endpoint, just doesn't manage the secrets anymore. Issuing a rolling restart on the deployment resolves it, but it's a manual fix.

version: 7.2.0
k8s: 1.19
backend: Azure Keyvault

Last entries in log files look like:

{"level":"info","message_time":"2021-05-21T22:12:49.695Z","pid":17,"hostname":"external-secrets-kubernetes-external-secrets-5bdc8f9947-8x7pg","message":"Stopping watch stream for namespace * due to event: END"}
{"level":"info","message_time":"2021-05-21T22:12:49.699Z","pid":17,"hostname":"external-secrets-kubernetes-external-secrets-5bdc8f9947-8x7pg","message":"stopping poller for ns1/notifications-secret"}
{"level":"info","message_time":"2021-05-21T22:12:49.699Z","pid":17,"hostname":"external-secrets-kubernetes-external-secrets-5bdc8f9947-8x7pg","message":"stopping poller for ns1/private-key"}
{"level":"info","message_time":"2021-05-21T22:12:49.699Z","pid":17,"hostname":"external-secrets-kubernetes-external-secrets-5bdc8f9947-8x7pg","message":"stopping poller for ns2/dns-config"}
{"level":"info","message_time":"2021-05-21T22:12:49.699Z","pid":17,"hostname":"external-secrets-kubernetes-external-secrets-5bdc8f9947-8x7pg","message":"stopping poller for ns3/external-dns-config"}

Metrics from kubernetes show CPU (and network) usage as 0 once it occurs, memory usage is also static. Green being the 'failed' pod and orange a new one from restarting the deployment:
image

Same problem here:

helm chart version: 8.0.1
k8s (AKS): 1.20.5
backend: Azure Keyvault

Should be resolved in 8.0.2 by #765