Cannot get metrics from collector
flenoir opened this issue · 10 comments
flenoir commented
Hi,
I did try to use your Opentelemetry APM Grfaana Dashboard but i can get same metrics as yours. Listing my opentelemetry-collector metrics, i can't find duration_milliseconds_count or duration_seconds_count
Here are my metrics :
# HELP otelcol_exporter_queue_capacity Fixed capacity of the retry queue (in batches)
# TYPE otelcol_exporter_queue_capacity gauge
otelcol_exporter_queue_capacity{exporter="loki",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 1000
otelcol_exporter_queue_capacity{exporter="otlp/tempo",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 1000
otelcol_exporter_queue_capacity{exporter="otlphttp",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 1000
otelcol_exporter_queue_capacity{exporter="prometheusremotewrite",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 10000
# HELP otelcol_exporter_queue_size Current size of the retry queue (in batches)
# TYPE otelcol_exporter_queue_size gauge
otelcol_exporter_queue_size{exporter="loki",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 0
otelcol_exporter_queue_size{exporter="otlp/tempo",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 0
otelcol_exporter_queue_size{exporter="otlphttp",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 0
otelcol_exporter_queue_size{exporter="prometheusremotewrite",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 0
# HELP otelcol_exporter_send_failed_metric_points Number of metric points in failed attempts to send to destination.
# TYPE otelcol_exporter_send_failed_metric_points counter
otelcol_exporter_send_failed_metric_points{exporter="otlphttp",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 785867
otelcol_exporter_send_failed_metric_points{exporter="prometheusremotewrite",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 43399
# HELP otelcol_exporter_sent_metric_points Number of metric points successfully sent to destination.
# TYPE otelcol_exporter_sent_metric_points counter
otelcol_exporter_sent_metric_points{exporter="debug",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 785867
otelcol_exporter_sent_metric_points{exporter="otlphttp",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 0
otelcol_exporter_sent_metric_points{exporter="prometheusremotewrite",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 742468
# HELP otelcol_exporter_sent_spans Number of spans successfully sent to destination.
# TYPE otelcol_exporter_sent_spans counter
otelcol_exporter_sent_spans{exporter="debug",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 1.545405e+06
otelcol_exporter_sent_spans{exporter="otlp/tempo",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 1.545405e+06
otelcol_exporter_sent_spans{exporter="otlphttp",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 1.545405e+06
# HELP otelcol_kafka_receiver_current_offset Current message offset
# TYPE otelcol_kafka_receiver_current_offset gauge
otelcol_kafka_receiver_current_offset{name="",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 1.71299006e+08
# HELP otelcol_kafka_receiver_messages Number of received messages
# TYPE otelcol_kafka_receiver_messages counter
otelcol_kafka_receiver_messages{name="",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 21429
# HELP otelcol_kafka_receiver_offset_lag Current offset lag
# TYPE otelcol_kafka_receiver_offset_lag gauge
otelcol_kafka_receiver_offset_lag{name="",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 0
# HELP otelcol_kafka_receiver_partition_close Number of finished partitions
# TYPE otelcol_kafka_receiver_partition_close counter
otelcol_kafka_receiver_partition_close{name="",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 4
# HELP otelcol_kafka_receiver_partition_start Number of started partitions
# TYPE otelcol_kafka_receiver_partition_start counter
otelcol_kafka_receiver_partition_start{name="",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 8
# HELP otelcol_process_cpu_seconds Total CPU user and system time in seconds
# TYPE otelcol_process_cpu_seconds counter
otelcol_process_cpu_seconds{service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 190.44
# HELP otelcol_process_memory_rss Total physical memory (resident set size)
# TYPE otelcol_process_memory_rss gauge
otelcol_process_memory_rss{service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 2.91172352e+08
# HELP otelcol_process_runtime_heap_alloc_bytes Bytes of allocated heap objects (see 'go doc runtime.MemStats.HeapAlloc')
# TYPE otelcol_process_runtime_heap_alloc_bytes gauge
otelcol_process_runtime_heap_alloc_bytes{service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 1.31552104e+08
# HELP otelcol_process_runtime_total_alloc_bytes Cumulative bytes allocated for heap objects (see 'go doc runtime.MemStats.TotalAlloc')
# TYPE otelcol_process_runtime_total_alloc_bytes counter
otelcol_process_runtime_total_alloc_bytes{service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 5.72070242e+10
# HELP otelcol_process_runtime_total_sys_memory_bytes Total bytes of memory obtained from the OS (see 'go doc runtime.MemStats.Sys')
# TYPE otelcol_process_runtime_total_sys_memory_bytes gauge
otelcol_process_runtime_total_sys_memory_bytes{service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 5.17147e+08
# HELP otelcol_process_uptime Uptime of the process
# TYPE otelcol_process_uptime counter
otelcol_process_uptime{service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 687.527554317
# HELP otelcol_processor_batch_batch_send_size Number of units in the batch
# TYPE otelcol_processor_batch_batch_send_size histogram
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="10"} 84
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="25"} 380
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="50"} 889
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="75"} 1509
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="100"} 1918
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="250"} 5130
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="500"} 6736
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="750"} 8358
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="1000"} 8359
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="2000"} 8360
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="3000"} 8362
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="4000"} 8371
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="5000"} 8384
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="6000"} 8426
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="7000"} 8426
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="8000"} 8426
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="9000"} 8426
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="10000"} 8426
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="20000"} 8426
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="30000"} 8426
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="50000"} 8426
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="100000"} 8426
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="+Inf"} 8426
otelcol_processor_batch_batch_send_size_sum{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 2.3312719999999995e+06
otelcol_processor_batch_batch_send_size_count{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 8426
# HELP otelcol_processor_batch_batch_send_size_bytes Number of bytes in batch that was sent
# TYPE otelcol_processor_batch_batch_send_size_bytes histogram
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="10"} 0
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="25"} 0
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="50"} 0
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="75"} 0
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="100"} 0
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="250"} 0
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="500"} 0
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="750"} 2
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="1000"} 6
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="2000"} 11
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="3000"} 22
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="4000"} 38
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="5000"} 49
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="6000"} 65
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="7000"} 83
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="8000"} 112
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="9000"} 162
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="10000"} 199
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="20000"} 1062
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="30000"} 1981
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="50000"} 3186
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="100000"} 4431
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="200000"} 5597
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="300000"} 6648
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="400000"} 8232
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="500000"} 8359
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="600000"} 8359
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="700000"} 8359
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="800000"} 8381
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="900000"} 8383
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="1e+06"} 8383
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="2e+06"} 8419
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="3e+06"} 8426
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="4e+06"} 8426
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="5e+06"} 8426
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="6e+06"} 8426
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="7e+06"} 8426
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="8e+06"} 8426
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="9e+06"} 8426
otelcol_processor_batch_batch_send_size_bytes_bucket{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",le="+Inf"} 8426
otelcol_processor_batch_batch_send_size_bytes_sum{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 1.309194729999994e+09
otelcol_processor_batch_batch_send_size_bytes_count{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 8426
# HELP otelcol_processor_batch_batch_size_trigger_send Number of times the batch was sent due to a size trigger
# TYPE otelcol_processor_batch_batch_size_trigger_send counter
otelcol_processor_batch_batch_size_trigger_send{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 6508
# HELP otelcol_processor_batch_timeout_trigger_send Number of times the batch was sent due to a timeout trigger
# TYPE otelcol_processor_batch_timeout_trigger_send counter
otelcol_processor_batch_timeout_trigger_send{processor="batch",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0"} 1918
# HELP otelcol_receiver_accepted_metric_points Number of metric points successfully pushed into the pipeline.
# TYPE otelcol_receiver_accepted_metric_points counter
otelcol_receiver_accepted_metric_points{receiver="kafka/metrics_fab",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",transport="kafka"} 409437
otelcol_receiver_accepted_metric_points{receiver="kafka/metrics_prod",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",transport="kafka"} 170486
otelcol_receiver_accepted_metric_points{receiver="prometheus/receiver",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",transport="http"} 3034
# HELP otelcol_receiver_accepted_spans Number of spans successfully pushed into the pipeline.
# TYPE otelcol_receiver_accepted_spans counter
otelcol_receiver_accepted_spans{receiver="kafka/traces_fab",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",transport="kafka"} 963220
otelcol_receiver_accepted_spans{receiver="kafka/traces_prod",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",transport="kafka"} 582245
# HELP otelcol_receiver_refused_metric_points Number of metric points that could not be pushed into the pipeline.
# TYPE otelcol_receiver_refused_metric_points counter
otelcol_receiver_refused_metric_points{receiver="kafka/metrics_fab",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",transport="kafka"} 0
otelcol_receiver_refused_metric_points{receiver="kafka/metrics_prod",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",transport="kafka"} 0
otelcol_receiver_refused_metric_points{receiver="prometheus/receiver",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",transport="http"} 0
# HELP otelcol_receiver_refused_spans Number of spans that could not be pushed into the pipeline.
# TYPE otelcol_receiver_refused_spans counter
otelcol_receiver_refused_spans{receiver="kafka/traces_fab",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",transport="kafka"} 0
otelcol_receiver_refused_spans{receiver="kafka/traces_prod",service_instance_id="c23a66b9-6654-4d41-9367-6566b4f7906b",service_name="otelcol-contrib",service_version="0.91.0",transport="kafka"} 0
I did adapt my collector config (version 0.91) to use connectors but i may have missed something
In my collector, my pipeline is set as this:
pipelines:
traces:
receivers: [otlp, jaeger, kafka/traces_fab, kafka/traces_prod, zipkin]
processors: [batch, span/statuscode]
exporters: [debug, otlp/tempo, otlphttp, spanmetrics]
metrics:
receivers: [otlp, kafka/metrics_fab, kafka/metrics_prod, prometheus/receiver, spanmetrics]
processors: [batch]
exporters: [debug, prometheusremotewrite, otlphttp]
How can i get duration_milliseconds_count or duration_seconds_count metrics ? Could they have a prefix ?
thanks for this great dashboard
blueswen commented
Hi @flenoir ,
Could you provide your full collector config? The metrics you provided look like from the metrics endpoint, but you are using prometheusremotewrite as the exporter. Could you describe how you get the metrics?
Or you can create another minimized metrics pipeline and use prometheus as the exporter to ensure the metrics are generated correctly.
flenoir commented
Hi,
Yes, please find the prometheus remote write part, frommy collector config below
prometheusremotewrite:
endpoint: http://ti-prometheus.z6-dev.k8s.mycompany.intra/api/v1/write
resource_to_telemetry_conversion:
enabled: true
"mycomapny" is not the real dns name for confidentiallity reason
this is a local prometheus pod on my cluster
the metrics, on my inital post, are comming out of my collector pod
As i checked your connector config, i guess it may be related to the use of "connectors", right ?
blueswen commented
I think maybe spanmetrics is not working correctly. How about creat another pipeline with prometheus exporter to check ?
exporters:
prometheus:
endpoint: "0.0.0.0:8889"
pipelines:
metrics/spanmetrics:
receivers: [spanmetrics]
processors: [batch]
exporters: [prometheus]zdelagrange commented
same experience on my end, the spanmetrics connector doesn't seem to produce those metrics with a very similar configuration. I only am getting duration_bucket duration_count duration_sum
blueswen commented
Hi @zdelagrange,
Could you check that your OpenTelemetry Collector version is 0.85.0+? The metrics generated by the Span Metrics connector already change the metrics name from duration_count to duration_milliseconds_count or duration_seconds_count according to the unit of the metrics.
zdelagrange commented
this particular instance is using 0.88, but i will upgrade to latest to see if it works out.
zdelagrange commented
awesome, i updated to 0.94.0 and those metrics are now available.
sarathskumar commented
Hi @blueswen
I am using opentelemetry collector chart version is 0.73.1. There i am getting these metrics latency_milliseconds_count instead of duration_milliseconds_count. Did i miss anything in the config to get duration metrics?
blueswen commented
Hi @sarathskumar,
Could you update opentelemetry collector chart to the latest version? If not, you can replace the metrics name with latency_milliseconds_count in dashboard json.
sarathskumar commented
Hi @blueswen
I replaced all metrics in the dashboard with latency_milliseconds_count. Thanks for your effort