/alertmanager-github-receiver

Prometheus Alertmanager webhook receiver that creates GitHub issues from alerts

Primary LanguageGoApache License 2.0Apache-2.0

alertmanager-github-receiver

Version Build Status Coverage Status GoDoc | Go Report Card

Not all alerts are an emergency. But, we want to track every one because alerts are always an actual problem. Either:

  • an actual problem in the monitored system
  • an actual problem in processes around the monitored system
  • an actual problem with the alert itself

The alertmanager github receiver creates GitHub issues using Alertmanager webhook notifications.

Build

make docker DOCKER_TAG=repo/imageName

This will build the binary and push it to repo/imageName.

Setup

Create GitHub access token

The github receiver uses user access tokens to create issues in an existing repository.

Generate a new access token:

Because this access token has permission to create issues and operate on repositories the access token user can access, protect the access token as you would a password.

Start GitHub Receiver

To start the github receiver locally:

docker run -it -p 9393:9393 measurementlab/alertmanager-github-receiver:latest
        -authtoken=$(GITHUB_AUTH_TOKEN) -org=<org> -repo=<repo>

Note: both the org and repo must already exist.

Configure Alertmanager Webhook Plugin

The Prometheus Alertmanager supports third-party notification mechanisms using the Alertmanager Webhook API.

Add a receiver definition to the alertmanager configuration.

- name: 'github-receiver-issues'
  webhook_configs:
  - url: 'http://localhost:9393/v1/receiver'

To publish a test notification by hand, try:

msg='{
  "version": "4",
  "groupKey": "fakegroupkey",
  "status": "firing",
  "receiver": "http://localhost:9393/v1/receiver",
  "groupLabels": {"alertname": "FoobarIsBroken"},
  "externalURL": "http://localhost:9093",
  "alerts": [
    {
      "labels": {"thing": "value"},
      "annotations": {"hint": "how to fix foobar"},
      "status": "firing",
      "startsAt": "2018-06-12T01:00:00Z",
      "endsAt": "2018-06-14T01:00:00Z"
    }
  ]
}'
curl -XPOST --data-binary "${msg}" http://localhost:9393/v1/receiver

Configuration

Alertmanager & Github Receiver

The Alertmanager configuration controls what labels are present on alerts delivered to the github-receiver. The github-receiver configuration must be compatible with these settings to work effectively.

For example, it is common for the Alertmanager to use alert routes that group_by: ['alertname']. See: https://github.com/prometheus/alertmanager#example

And, the github-receiver's default "title template" is {{ .Data.GroupLabels.alertname }}, which depends on an alertname group label.

If an alert does not include this label, the template will evaluate to <no value>. To prevent this, ensure that the github-receiver title template uses labels available in an Alertmanager Message.

Auto close

If -enable-auto-close is specified, the program will close each issue as its corresponding alert is resolved. It searches for matching issues by filtering open issues on the value of -alertlabel and then matching issue titles. The issue title template can be overridden using -title-template-file for greater (or lesser) specificity. The default template is {{ .Data.GroupLabels.alertname }}, which sets the issue title to the alert name. The template is passed a Message as its argument.

Repository

If the alert includes a repo label, issues will be created in that repository, under the GitHub organization specified by -org. If no repo label is present, issues will be created in the repository specified by the -repo option.