A cardinailty nanny for Promtheus.
Which will:
- Prevent high cardinality metrics from crashing prometheus (by dropping those labels)
- Remove old instances of the metric with high cardinality
TODO:
- Metric name cardinality as opposed to label cardinality
- Write notifications when cardinality is averted
- Better local setup
Open 3 terminals (sorry):
- Run
docker compose up
to start prometheus - Run
go run cmd/cardinality-injector/inject-cardinality.go
to inject some cardinality into prometheus - Run
go run cmd/cardinanny/cardinanny.go -cardinalityLabelLimit=200
to start cardinanny