If a secret is deleted with a failed last sync, metric still exists
piyat opened this issue · 1 comments
piyat commented
Hi - thanks for exposing so many useful metrics out of the box! I just wanted to check my understanding of something I've observed and see if there's a way around it.
I made a prometheus rule to send an alert whenever an external secret had a failed last sync with the below:
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: external-secrets-prometheus-rules
namespace: external-secrets
spec:
groups:
- name: ExternalSecretsRules
rules:
- alert: ExternalSecretSyncFailure
annotations:
message: "Last Sync of {{`{{ $labels.name }}`}} in namespace {{`{{ $labels.exported_namespace }}`}} failed\n"
expr: sum((kubernetes_external_secrets_last_sync_call_state)) by (name, exported_namespace) < 1
labels:
severity: warning
I made a dummy secret which would always fail (upstream secret doesn't exist), and I received the alert - all good.
I then deleted the dummy secret, but the alert still fires. It seems as though if there was never a successful sync after the most recent failed sync, this metric is always present.
Is there a way to avoid this?
Flydiverny commented