grafana/mimir-proxies

datadog-agent stop sending metrics

k-cloudit opened this issue · 2 comments

k-cloudit commented

I'm using mimir proxies main branch to send metrics from datadog-agent to mimir.
This this how datadog is configured :

additional_endpoints:
  "http://IP:8009/datadog":
  - grafana-labs

it works, mimir is receiving metrics but after few minutes I have this message in datadog-agent :

2023-09-07 13:34:51 CEST | CORE | ERROR | (pkg/forwarder/transaction/transaction.go:350 in internalProcess) | Error code "400 Bad Request" received while sending transaction to "http://10.0.0.47:8009/datadog/api/v1/check_run": "can't translate series: client_error: client error: 404\n", dropping it
2023-09-07 13:34:51 CEST | CORE | ERROR | (pkg/forwarder/transaction/transaction.go:350 in internalProcess) | Error code "400 Bad Request" received while sending transaction to "http://10.0.0.47:8009/datadog/api/v1/series": "can't translate series: client_error: client error: 404\n", dropping it

and this log message on mimir-proxies :

level=info ts=2023-09-07T11:34:51.057594497Z elapsed=2.481926ms traceID=71652a924ff0db99 sampled=false orgID=fake method=POST uri=/datadog/api/v1/series status=400
level=warn ts=2023-09-07T11:35:06.054259542Z traceID=5e64c8ef0da08f03 request_uri=/datadog/api/v1/check_run method=POST msg="can't translate series" response_code=400 err="client_error: client error: 404"
image
mimir-proxies

I attached a grafana dashboard show how the metrics are ingested, no mimir-proxies or datadog-agent have been restarted.

is there any solution to this behavior ?

alograg commented

@k-cloudit confirm that the problem no longer occurs after making the following updates:

  • httpsnoop v1.0.3 -> httpsnoop v1.0.4
  • mux v1.8.0 -> mux v1.8.1
  • clientgolang v1.16.0 -> clientgolang v1.17.0
  • common v0.44.0 -> common v0.45.0
  • otel v1.14.0 -> otel v1.19.0
  • opentracing v1.14.0 opentracing v1.19.0
  • sdk v1.14.0 -> sdk v1.19.0
  • trace v1.14.0 -> trace v1.19.0
  • net v0.15.0 -> net v0.19.0
  • golangprotobufextensions v1.0 -> golangprotobufextensions v2
  • clientmodel v0.4.0 -> clientmodel v0.4.1-0.202307181
  • procfs v0.10.1 -> procfs v0.11.1
  • metric v0.37.0 -> metric v1.19.0
  • crypto v0.13.0 -> crypto v0.16.0
  • oauth2 v0.8.0 -> oauth2 v0.12.0
  • sync v0.2.0 -> sync v0.3.0
  • sys v0.12.0 -> sys v0.15.0
  • text v0.13.0 -> text v0.14.0
  • protobuf v1.30.0 -> protobuf v1.31.0