Kafka input plug-in skip reading from single partition from assigned topic
patelprakashp opened this issue · 0 comments
Logstash information:
Please include the following information:
- Logstash version : 6.8.14
- Logstash installation source : Docker
- How is Logstash being run : Docker
- How was the Logstash Plugin installed: Default
- Kafka-Input Plug-in : 9.1.0
- Kafka client : 2.3.0
I have logstash running on Kubernetes and consuming events from 100+ topics.Each topic have 3+ partitions. Thus Single consumer group subscribed to 100+ topics and consumes events from 500+ partitions. Below are the Kafka input configuration.I am running 10 instances of logstash on kubernetes so total 100 consumer threads consuming from 500 partitions.
auto_offset_reset => "earliest"
enable_auto_commit => "true"
group_id => "test_logstash"
consumer_threads => "10"
max_poll_records => "500"
heartbeat_interval_ms => "9000"
session_timeout_ms => "30000"
fetch_max_bytes => "10485760"
max_partition_fetch_bytes => "524288"
client_id => "test_logstash"
decorate_events => true
partition_assignment_strategy => "org.apache.kafka.clients.consumer.RoundRobinAssignor"
Kafka consumer stops reading data from single partition although its subscribed and assigned to consumer group.