/prometheus-alerts-playground

Example for testing Prometheus Alerts + Alert Manager + SMTP server.

Prometheus Alerts Playground

This example run 3 containers:

  • Prometheus
  • AlertManager
  • A SMTP server.

It's created 2 alert rules with the file playground/prometheus/prometheus.rules.yml, and the Alert Manager container is configured to send mail to root@localhost for every alert.

One alert rule should be fired after a minute and the other one should be fired after 5 minutes.

Getting started

cd playground
docker-compose up -d

docker exec -it smtp tail -F /var/mail/mail

The expected behavior is that after the first mail will have been sent the file will be created and the tail command will handle it and print the mail content.

Debugging and Troubleshooting

  • Open container logs.
  • Try the queries defined on playground/prometheus/prometheus.rules.yml by using Prometheus Graph UI. Queries example.
  • On the Alert tab inspect the state of the alert rules.
  • Go to Alertmanager UI and check the alerts were properly received by AlertManager.
  • ssh to smtp container and inspect the local Exim.

Useful resources: