metrics_analyzer

metrics_analyzer is a command line tools which helps checking metrics between dashboards of grafana, prometheus rules and prometheus metrics and outputs whitelisted metrics or metrics to drop.

Features

Connect via API to Grafana and get all metrics which are use in any dashboards
Connect via API to Prometheus and get all metrics which are use in any rules
Connect via API to Prometheus and check exist metrics
Generates regex output to add to service monitor

Installation

Prerequisites

at least Python 3.10

Stable release

To install metrics_analyzer, run this command directly from repository:

$ pip install git+https://github.com/oleksii-boiko-ua/metrics_analyzer.git

Usage

To use this tool locally, you will need to create port-forwards for your grafana and prometheus services.

$ kubectl port-forward svc/grafana 3000:80
$ kubectl port-forward svc/prometheus 9090:9090

Also, if you use an API key for Grafana you should set it in you environment.

$ export GRAFANA_KEY=...

Now you should be able to run the following script:

$ metrics_analyzer metrics-to-whitelist

Output example:

We can whitelist 10 metrics from the job argocd-metrics

process_cpu_seconds_total|ALERTS|go_goroutines|workqueue_depth|process_start_time_seconds|process_resident_memory_bytes|workqueue_adds_total|up|argocd_app_info|workqueue_queue_duration_seconds_bucket


We can whitelist 55 metrics from the job kube-state-metrics

kube_job_status_start_time|kube_deployment_status_replicas|kube_statefulset_status_replicas_ready|kube_pod_container_resource_requests|kube_node_spec_taint|kube_namespace_created|kube_node_status_allocatable|kube_pod_container_resource_limits|kube_deployment_labels|kube_node_info


We can whitelist 10 metrics from the job prometheus-grafana

process_cpu_seconds_total|grafana_http_request_duration_seconds_bucket|go_goroutines|process_start_time_seconds|grafana_build_info|process_resident_memory_bytes|grafana_http_request_duration_seconds_sum|grafana_stat_totals_dashboard|up|grafana_http_request_duration_seconds_count

example of metric relabelings for service monitor to whitelist recomended metrics

metricRelabelings:
  - action: keep
    regex: 'process_cpu_seconds_total|go_goroutines|prometheus_operator_watch_operations_total'
    sourceLabels: [__name__]