Alert analysis tool gathers alerts from webhook and Alertmanager API, stores it in ClickHouse database and provides dashboards for analyzing the data. Please refer to Cloudflare Blog for insights and experiences and PromCon 2023 in Berlin for the talk presentation.
This demo uses vector.dev to collect data from different sources and write the data in the datastore.
we use one http_server vector instance - to receive Alertmanager webhook notifications,
two http_client sources to query Alertmanager's alerts and silence API endpoints and
two sinks for writing all the state logs in ClickHouse into alerts and silences tables.
The docker-compose will bring up several containers:
Cadvisoris used to generate system metrics for monitoring.Prometheusis used to monitor and generate alerts.Alertmanageris to route alerts and provide the alert events via webhook and API.alertmanager_silenceis to create an Alertmanager silence.blackbox_exporteris for monitoring the sites and generating alerts.ClickHouseis used to write the Alertmanager alert events into the datastore for alert analysis.Vector.dev- to collect data from Alertmanager webhook, alerts and silences API, transform the data and write into ClickHouse.Grafanais used to visualize the logs.
docker
Bring up the containers using docker compose
docker compose up
Please wait for about 5 minutes for the alerts to be triggered and
visit http://localhost:3000/ to explore the Alerts and silences overview dashboard and play around.
