processor label missing in otel metrics

[nikmel2803]

Hi! Thanks for the great dashboard!

I noticed one issue, not sure what's the reason for this.
The dashboard relies on processor label in some queries but i actually can't see this label in any of otel collector metrics. I'm operating latest version of collector under otel/opentelemetry-collector-contrib:0.97.0 image.

Does this label come from some older versions or am I missing something?

I have detailed log level enabled:

    service:
      telemetry:
        metrics:
          level: detailed

Here is result of my search for processor label:

I think it's easy to workaround, just want to clarify the reason first.

Thanks!

[jangaraj]

It works for me:

Do you have any processor in your pipelines?

Tested with otel/opentelemetry-collector-contrib:0.97.0 (https://github.com/monitoringartist/grafana-opentelemetry)

[nikmel2803]

hmm that's strange. yes, i have:

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [k8sattributes, resource/cluster, tail_sampling]
          exporters: [otlp]
      telemetry:
        metrics:
          level: detailed

[jangaraj]

I think that not all processors produce those processor metrics. Does it causing any visible UX issue, when you use dashboard with your setup, please?

[nikmel2803]

these are the metrics i have (on the /metrics endpoint):

i see that dashboard is looking for metric prefixed with otelcol_processor_accepted_spans:

but i actually can't see metrics with such name at all:

# produces empty result
curl -s http://localhost:59467/metrics | grep otelcol_processor_accepted_spans

this is how dashboard looks for me:

and the processors variable is empty:

curious what's causing this difference between our metrics 🧐

[jangaraj]

I would say good practise is to have memory limiter as first processor (to handle memory pressure in better way than OOM killer) and last batch processor (to have better performance) - that's included in the dashboard.

It looks like sampling processor has own metric set, so maybe you can create new row for this specific processor metrics, which will be collapsed by default (as Kubernetes row)- feel free to create PR for that.

[kevinnoel-be]

I've stumbled on this issue too when using this dashboard, it seems the processor variable doesn't have "allValue": ".*" specified here (similar example for receiver variable here)

Created following PR: #7