[QUERY]Stick one EventHub consumer to read a specific partition from AEH using spring-messaging-azure-eventhubs SDK
hpunjan1 opened this issue · 2 comments
Query/Question
How can we stick one EventHub consumer instance to read a specific partition from Azure EventHub with using the Spring Azure EventHub SK- spring-messaging-azure-eventhubs 4.1.0?
Our current codebase is based on this:
- We have multiple instances of consumer process running reading from same EventHub with 10 partitions and 1 consumer group.
- We checkpoint manually into separate storage account containers.
- It is working fine but since we checkpoint for individual event hub partitions in separate container locations and our consumers can start/stop randomly, multiple consumers potentially read from older checkpoints and reload the same data, resulting in duplicates. Because of this issue, we want to stick each consumer to particular partition
- We have 10 partitions on our event hub
Why is this not a Bug or a feature Request?
We are already using the Spring Azure EventHub SDK open-sourced by Microsoft. We only have a query related to the SDK usage.
Setup (please complete the following information if applicable):
- OS: Linux
- IDE: Eclipse
- Library/Libraries: spring-messaging-azure-eventhubs version 4.1.0
Hi @hpunjan1 , thanks for reaching out.
spring-messaging-azure-eventhubs using EventProcessorClient, which is self-balancing, does not support sticking to a particular partition.
Hi,
Thanks for your reply.
What flavor of Spring integration would support BOTH?
- specifying a particular partition per consumer
- specifying a particular offset to read from that partition