Dummy 'summary' message for Pagerduty integration discussion
dputnamfr opened this issue · 0 comments
Slack thread: https://grafana.slack.com/archives/C043CEX9MBK/p1689199648823409
Summary:
I ran into a situation recently where the Pagerduty-Grafana integration was failing to send 'alert resolved' notifications to a Pagerduty contact point after the alert resolved. The problem was due to a golang template for the 'summary' section of the contact point being wrapped in an {{ if .Alerts.Firing }} tag (which of course would result in an empty summary upon resolution of an alert). Pagerduty was responding with a 400 every time this empty summary came through, resulting in a failure to auto-resolve a page.
I was asked by @yuri-tceretian to generate an issue so ya'll could discuss the addition of possible "protect the user from their own misconfiguration" dummy text upon empty summary.
My own thoughts:
The addition of golang templating to grafana alerts in grafana 9 looks really powerful, but it's still pretty complex and not entirely transparent to the userbase yet... I'm not sure I (as a user) would appreciate the addition of an extra 'sane default' in an ecosystem where there's already a lot that can go wrong and not quite as much transparency on the alert compilation process...
... but you guys are the experts, go get 'em. :)