How to consumer from multiple Kafka topic partitions in parallel?
Rap70r opened this issue · 0 comments
Rap70r commented
Hello,
Is there a specific parameter to enable Spark to consume from multiple Kafka topic partitions in parallel?
I'm using the following in my config.yaml:
triggerMode: ProcessingTime
triggerDuration: 30 seconds
batchMode: true
And for the input section I have:
options:
startingOffsets: earliest
maxOffsetsPerTrigger: 100000
I'm running Spark with spark.dynamicAllocation.enabled set to true.
Thank you