Application stops functioning after a few days
Closed this issue · 2 comments
Application seems to just stop functioning, usually after a few days. No action is taken to stop or terminate deployment. As far as kubernetes is concerned the pod is still running, and it does indeed respond to calls on the metrics endpoint, just doesn't manage the secrets anymore. Issuing a rolling restart on the deployment resolves it, but it's a manual fix.
version: 7.2.0
k8s: 1.19
backend: Azure Keyvault
Last entries in log files look like:
{"level":"info","message_time":"2021-05-21T22:12:49.695Z","pid":17,"hostname":"external-secrets-kubernetes-external-secrets-5bdc8f9947-8x7pg","message":"Stopping watch stream for namespace * due to event: END"}
{"level":"info","message_time":"2021-05-21T22:12:49.699Z","pid":17,"hostname":"external-secrets-kubernetes-external-secrets-5bdc8f9947-8x7pg","message":"stopping poller for ns1/notifications-secret"}
{"level":"info","message_time":"2021-05-21T22:12:49.699Z","pid":17,"hostname":"external-secrets-kubernetes-external-secrets-5bdc8f9947-8x7pg","message":"stopping poller for ns1/private-key"}
{"level":"info","message_time":"2021-05-21T22:12:49.699Z","pid":17,"hostname":"external-secrets-kubernetes-external-secrets-5bdc8f9947-8x7pg","message":"stopping poller for ns2/dns-config"}
{"level":"info","message_time":"2021-05-21T22:12:49.699Z","pid":17,"hostname":"external-secrets-kubernetes-external-secrets-5bdc8f9947-8x7pg","message":"stopping poller for ns3/external-dns-config"}
Metrics from kubernetes show CPU (and network) usage as 0 once it occurs, memory usage is also static. Green being the 'failed' pod and orange a new one from restarting the deployment:
Same problem here:
helm chart version: 8.0.1
k8s (AKS): 1.20.5
backend: Azure Keyvault
Should be resolved in 8.0.2 by #765