Alertmanager webhook logger

This is a simple webhook receiver for Alertmanager, that logs the alerts it receives as properly formatted JSON documents on stdout.

The goal of the receiver is to keep a trace of all the alerts that were generated by Alertmanager, for example to:

Get an history of all the alerts generated in your logging system and see when alerts were triggered and when they were resolved.
Create statistics using your favorite logging system to see which alerts come up the most, with which parameters.
Help to create Alertmanager templates, by showing the actual content and structure of the alerts sent.

It is available as a Docker image:

docker pull ghcr.io/multani/alertmanager-webhook-logger

Usage

Run the webhook logger somewhere where it can be reached by Alertmanager.

Add it as a new receiver to Alertmanager in its configuration:

route:
  receiver: webhook-logger # send all alerts to the webhook-logger

  group_by:
    - alertname

receivers:
  - name: webhook-logger
    webhook_configs:
    - url: http://webhook-logger:8000/alerts
      send_resolved: true
      max_alerts: 0 # 0=all alerts

Alerts sent by Alertmanager should appear on the webhook logger standart output.

Output format

The content of the alerts field is the content sent by Alertmanager
The log level of the overall log message will be:
- info: the alerts are resolved
- warn: the alerts are firing
- error: something is wrong with the webhook logger itself
The timestamp of the log message is when the webhook logger receives and prints the message.

{
  "level": "warn",
  "timestamp": "2022-07-27T15:54:18.487Z",
  "caller": "app/main.go:118",
  "message": "Alerts received",
  "alerts": {
    "receiver": "webhook",
    "status": "firing",
    "alerts": [
      {
        "status": "firing",
        "labels": {
          "alertname": "Test1",
          "branch": "HEAD",
          "goversion": "go1.18.4",
          "instance": "127.0.0.1:29591",
          "job": "prometheus",
          "revision": "b41e0750abf5cc18d8233161560731de05199330",
          "severity": "critical",
          "version": "2.37.0"
        },
        "annotations": {
          "summary": "Something bad happened"
        },
        "startsAt": "2022-07-27T09:09:25.147Z",
        "endsAt": "0001-01-01T00:00:00Z",
        "generatorURL": "http://localhost:9090/graph?g0.expr=prometheus_build_info+%3E+0&g0.tab=1",
        "fingerprint": "a6f5e9850a5c3760"
      }
    ],
    "groupLabels": {
      "alertname": "Test1"
    },
    "commonLabels": {
      "alertname": "Test1",
      "branch": "HEAD",
      "goversion": "go1.18.4",
      "job": "prometheus",
      "revision": "b41e0750abf5cc18d8233161560731de05199330",
      "severity": "critical",
      "version": "2.37.0"
    },
    "commonAnnotations": {
      "summary": "Something bad happened"
    },
    "externalURL": "http://localhost:9093"
  }
}

Deployment & security

You can run multiple instances of the webhook logger.

Alertmanager is supposed to do the de-duplication on its side, and even if different instances of the webhook logger receive the alerts, looking at all the logs of all the webhook logger should give you an overview of the alerting status of your system.
Don't send secrets into the alerts.

This is not specific to this Alertmanager receiver, but it will not try to conceal any information from your alerts. Any sensitive information part of the content of the alerts themselves will be displayed as it was sent by Alertmanager.

Similar projects

This project was influenced by TomTom's own alertmanager-webhook-logger, but follows a different approach, by trying to stick closer to the original payload sent by Alertmanager.