grafana/synthetic-monitoring-app

Documentation about alert sensitivities is missing

mem opened this issue · 0 comments

mem commented

Back when we added alerts and alert sensitivities in particular we didn't do a good job of updating the user-facing documentation.

We need to capture:

  • Which values are known to the plugin (high, medium, low)
  • What does the plugin do with unknown values
  • How the known values map to the predefined alerts
  • How the known values map to thresholds

The short version of this is that we have a single recording rule for the success rate of the check that only looks at whether the alert_sensitivity label is not blank. If that's the case, the whole expression results in a value that gets recorded, otherwise it's ignored. In the alerts that we predefine, we look at the known values for the alert_sensitivity label and map those to thresholds.

If the user adds other values, the recording rule still produces a result, but we don't have predefined alerts for those. This allows the user to create custom rules, possibly with custom thresholds. They can use alert manager to route based on the labels.