treasure-data/kafka-fluentd-consumer

Consumer appears to spam connections to fluentd in-forward plugin causing system instability

Closed this issue · 1 comments

I'm having a hard time pinning down exactly what is happening to a machine using this consumer in concert with fluentd's in-forward plugin, but with low flush times and a slow flush executable, it appears fluentd's in-forward plugin begins to be spammed with connection attempts to flush the kafka-fluentd-consumer buffers despite fluentd reporting its buffers are full. Is it possible to ask this consumer to initiate connections to the in-forward plugin at a slower clip and more successful clip? This might leave behind fewer lingering connections and perhaps improve stability.

This issue an be closed. It was later identified we were experiencing hardware problems.