statsd metrics exporter not working

[ratovarius]

Version

cli version: 0.42.0
cluster version: 0.42.0

Description

I am exporting custom metrics using python statsd client.
I can't find the custom metrics on grafana.

Configuration

- name: classifier
  kind: RealtimeAPI
  pod:
    port: 8080
    containers:
      - name: api-classifier
        image: xxx.dkr.ecr.us-east-1.amazonaws.com/api-classifier
        compute:
          cpu: 200m
          mem: 200M
      - name: tfs-cpu
        image: xxx.dkr.ecr.us-east-1.amazonaws.com/tfs-cpu
        compute:
          cpu: 1
          mem: 1G
      - name: classifier
        image: xxx.dkr.ecr.us-east-1.amazonaws.com/classifier
        compute:
          cpu: 1340m
          mem: 3700000Ki

Steps to reproduce

1. include statsd==3.3.0 in the app requirements

2. python snippet pushing custom metric

import statsd

metrics = statsd.StatsClient(host='prometheus-statsd-exporter.prometheus.svc.cluster.local', port=9125)

n_clean = 0
n_error = 0

n_clean += 1

with metrics.pipeline() as pipe:
    pipe.incr(stat="api_n_clean" count=n_clean)
    pipe.incr(stat="api_n_error", count=n_error)

3. Check for metrics in prometheus-statsd-exporter

kubectl port-forward -n prometheus prometheus-statsd-exporter 9102
curl localhost:9102/metrics
# HELP api_n_clean Metric autogenerated by statsd_exporter.
# TYPE api_n_clean counter
api_n_clean{region="us-east-1"} 1.797706e+06
# HELP api_n_error Metric autogenerated by statsd_exporter.
# TYPE api_n_error counter
api_n_error{region="us-east-1"} 337182

prometheus-statsd-exporter is not logging any error or warning,

k logs prometheus-statsd-exporter
level=info ts=2022-02-03T22:04:48.184Z caller=main.go:321 msg="Starting StatsD -> Prometheus Exporter" version="(version=0.21.0, branch=HEAD, revision=ef6627b9f05350d54cd3bfea5afe36617d7eb5a4)"
level=info ts=2022-02-03T22:04:48.184Z caller=main.go:322 msg="Build context" context="(go=go1.16.5, user=root@8ace135a0329, date=20210610-07:24:32)"
level=info ts=2022-02-03T22:04:48.184Z caller=main.go:361 msg="Accepting StatsD Traffic" udp=:9125 tcp=:9125 unixgram=
level=info ts=2022-02-03T22:04:48.184Z caller=main.go:362 msg="Accepting Prometheus Requests" addr=:9102

prometheus-statsd-exporter seems to be working ok in prometheus targets

kubectl port-forward -n prometheus prometheus-prometheus-0 9090
I can't see the custom metrics in the prometheus metrics explorer either.

Expected behavior

I should see the custom metrics in grafana, under Metrics browser

Actual behavior

I can't find the custom metrics

[ratovarius]

For custom metrics exporting via statsd, the metrics name must start with cortex_.

Found in prometheus config,

- job_name: serviceMonitor/prometheus/operator/0
...
  metric_relabel_configs:
  - source_labels: [__name__]
    separator: ;
    regex: cortex_(.+)
    replacement: $1
    action: keep

import statsd

metrics = statsd.StatsClient(host='prometheus-statsd-exporter.prometheus.svc.cluster.local', port=9125)

n_clean = 0
n_error = 0

n_clean += 1

with metrics.pipeline() as pipe:
    pipe.incr(stat="cortex_api_n_clean" count=n_clean)
    pipe.incr(stat="cortex_api_n_error", count=n_error)

Now the metrics are shown correctly in grafana

[deliahu]

Awesome, thanks for following up on this!