Sending OpenTelemetry traces from Collector to Prometheus to create RED Dashboards
Closed this issue · 2 comments
In your setup, all the Otel instruments' traces are sent to Tempo, and from there, we can "Explore" them from Grafana.
I found difficulties in digesting the info from Tempo to generate Dashboards like this one:
I think they are called RED metrics.
I asked the Grafana community, and they explained to me that we have to expose the Otel traces in the Collector to Prometheus (on top of sending them to Tempo) so we can generate metrics dashboards as the suggested.
What are your thoughts about this? Do you think it should be included in your setup?
I think that maybe we don't need to digest the traces to generate RED metrics.
We can get them from the metrics that the prometheus_client
is generating:
- http_server_request_duration_seconds_bucket
- http_server_request_duration_seconds_count
- http_server_request_duration_seconds_sum
- http_server_requests_total
I see in the gif of the project'sREADME there is a kind of a RED metrics dashboard. Very nice one. I see it connects to Logs and from there to Traces.. wow.. this is great. Is this set up explained in the repo?
I managed to adapt the above dashboard to use the metrics sent by the prometheus-client
:
It has some errors, but I think it is caused by my inexperience, and they are solvable.