Rebalancing problem with Faust Streaming
arcanjo45 opened this issue · 0 comments
Checklist
- I have included information about relevant versions
- I have verified that the issue persists when using the
master
branch of Faust.
Steps to reproduce
When my Dev Kafka cluster goes down and we receive this error :
The Cluster ID _BIvovDtTg6eUb755um5cg doesn't match stored clusterId Some(KC3jQfsuTgqeVtWd6NHkuA)
After this we change the ID in the kafka properties to match the new one but then when I go to my consumer micro service it gives the following logs in a infinite loop and even after restarting the service and the consumer this still happens:
2024-01-12 14:37:10.155 GMT [2024-01-12 14:37:10,153] [11] [ERROR] [^---AIOKafkaConsumerThread]: Aiokafka has not sent fetch request for TP(topic='transactions-dev-processor-__assignor-__leader', partition=0) since start (started 9.57 minutes ago)
2024-01-12 14:37:10.156 GMT [2024-01-12 14:37:10,155] [11] [ERROR] [^---AIOKafkaConsumerThread]: Aiokafka has not sent fetch request for TP(topic='dev_reporting_2', partition=0) since start (started 9.57 minutes ago)
2024-01-12 14:37:10.232 GMT [2024-01-12 14:37:10,231] [11] [ERROR] Failed fetch offsets from 0: [Error 6] NotLeaderForPartitionError: TopicPartition(topic='transactions-dev-processor-__assignor-__leader', partition=0)
Shouldn't Faust rebalance the consumer metadata to mathch the new one and accept this partition as leader?
Bellow I can show the output of the describe topic command:
Topic: dev_reporting_2 TopicId: AhWVV8osQkKBpSl6ZZqaSA PartitionCount: 1 ReplicationFactor: 1 Configs: Topic: dev_reporting_2 Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Versions
- Python version = 3.12.1
- Faust version = 0.10.21