Upgrade Kafka client to 2.7
mbarretta opened this issue · 4 comments
The current client is 2.4, which is 18mo old. A number of optimizations have been introduced since (especially 2.5), which can improve the performance of Logstash.
Expanding this to include support for Static Membership, which was introduced in Kafka 2.3: https://www.confluent.io/blog/kafka-rebalance-protocol-static-membership/#unnecessary-rebalance (<-- closest anchor, scroll down a bit). We will need a way to set the group.instance.id setting.
Related PR I opened in old logstash-kafka-input repo - logstash-plugins/logstash-input-kafka#341.
The static membership concept is proposed and explained in KIP-345.
When a client of a consumer group reconnects to a cluster it triggers a rebalance, to assign partition to consumers and this consumer is assigned an id named member.id which is randomly generated.
If a client is member of a consumer groups, and reconnects frequently is mean it triggers frequent rebalances.
To avoid this a static member.id could be passed by the consumer, to uniquely identify it inside the consumer group and in stored on both sides (broker and consumer). In this way the static group.instance.id avoids rebalances in case of continuous disconnections and reconnections.
Also with Static Membership however, rebalances could happen when a new member joins the group or a leader re-joins, for a full list of cases where rebalances happen also with static membership, please check KIP-345