logstash-plugins/logstash-input-kafka

don't join group until pipeline is initialized to prevent premature rebalancing

bczifra opened this issue · 1 comments

When a Kafka input is used it cannot begin processing until the rest of the pipeline has been initialized. However, its own initialization can trigger rebalancing within Kafka. With a large complex pipeline that may take several minutes to initialize, that Kafka input can be blocked for quite a while resulting in a delay of message processing from the assigned partition. In such cases, it would be useful for the Kafka input to wait to join the group until the pipeline is actually read to process events.

This issue is mitigated indirectly with a separate change that went into Logstash 7.5.2 and 7.6.0, in which the inputs to a pipeline are not started until the worker-loop is fully initialized. See elastic/logstash#11492