BUG: Prometheus metrics are broken in td-agent 3.8.0
Houlistonm opened this issue · 0 comments
Houlistonm commented
We upgraded from td-agent-3.6.0 to td-agent-3.8.0 and our Grafana/Prometheus was reporting NO DATA for the fluent pipeline.
Our td-agent,log had this message in it.
2020-07-03 14:50:47 +0000 [error]: #0 #<Async::Task:0x3fd56383151c incoming connection #<Addrinfo: 127.0.0.1:39414 TCP> [fd=153] (failed)>Broken pipe
There were no other messages to indicate the source process/task to we started reviewing the change logs and recent issues with the embedded gems.
Found this issue : fluent/fluent-plugin-prometheus#159
We hand patched GEM with the fix in this PR: fluent/fluent-plugin-prometheus#161
And our metrics began working and the broken pipe errors have not returned.