kafka_group not working with kafka > 2.1
splushii opened this issue · 9 comments
Using the kafka_group consumer with a Kafka 2.3 cluster results in lost messages, duplicated messages and abnormally high load and low throughput. Using a Kafka 2.1 cluster works well. I am not sure of the reason but I suspect that the consumer offset change (introduced in Kafka 2.2) is not supported by ruby-kafka (due to the compatibility chart). But these are just speculations that made me try out Kafka 2.1, nothing I have confirmed.
The complete plugin config for reference:
<source>
@type kafka_group
brokers log-kafka-bootstrap.log.svc.cluster.local:9093
consumer_group fluentd
topics test
format json
time_source record
time_format "%iso8601"
start_from_beginning false
max_wait_time 5
max_bytes 1048576
username "#{ENV['KAFKA_USERNAME']}"
password "#{ENV['KAFKA_PASSWORD']}"
scram_mechanism sha512
ssl_ca_cert /etc/ssl/certs/kafka/ca.crt
</source>
I am running multiple fluentd instances with the same config.
Hmm... if ruby-kafka keeps limited support with recent versions, we need to consider rdkafka based consumers.
i have the same issue. my version is td-agent-3.5.1-0.el7.x86_64 and kafka 2.12-2.3.0，when i fallback kafka to 2.12-2.2.0，everything is ok.
Having the same issue. I would upvote for rdkafka based input plugin too. With ruby_kafka, there have been some or the other issue with kafka upgrades, and their compatibility charts are so convoluted and doesn't give much details so you don't know what's going to break.
As a mitigation in the README at least, see zendesk/ruby-kafka#839 - please add to that PR's comments if you have any more specifics.
what's the error message you are getting ?
Interesting if it is similar to mine #346
rdkafka binding sounds good. iirc fluent-bit is already using it
rdkafka input plugin will be handled in #350
This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days
This issue was automatically closed because of stale in 30 days