Rate limit ineffective for kustomization reconciliation events with time stamps

[luk2038649]

Issue

Duplicate alerts sent to external service for successful Kustomization reconciliation with an unchanged revision.

Consequences

In our case, the github Provider would start to fail due to GitHub API rate limiting being applied.

To reproduce

enable rate limiting in notification controller

configure a Kustomization.interval to be less than the notification controller’s --rate-limit-interval

configure Alert for the Kustomization

Reason

The fluxcd/notification-controller provides rate limiting of duplicate events. The notification-controller uses Event.message as one of the fields to determine uniqueness. On successful reconciliation, the fluxcd/kustomize-controller emits an Event object with Event.message containing nanosecond precision duration of how long the reconcile took. The message has too much variance for the rate limiter implementation to meaningful dedupe Alerts.

Here are two example messages for the same Kustomization revision that successfully reconciled:

Reconciliation finished in 2.084434647s, next run in 1m0s

and

Reconciliation finished in 2.075916724s, next run in 1m0s

Version info

Flux version: 0.41.2
helm-controller: v0.31.2
image-automation-controller: v0.31.0
image-reflector-controller: v0.26.1
kustomize-controller: v0.35.1
notification-controller: v0.33.0
source-controller: v0.36.1

[leosunmo]

We're currently running in to this as well. We have a service that receives notifications as generic webhooks from Flux and deduplicates them itself by hashing various parts of it and storing it in a database. This is quite inconvenient and it would be great if this could be handled in Flux itself instead.

We are seeing the issue with Kustomizations as well. We get webhooks sent every time they reconcile right now, which is unusable.

[stefanprodan]

Drop the Reconciliation finished in in the Alert using https://fluxcd.io/flux/components/notification/alerts/#event-exclusion