YotpoLtd/metorikku

How to consumer from multiple Kafka topic partitions in parallel?

Rap70r opened this issue · 0 comments

Hello,

Is there a specific parameter to enable Spark to consume from multiple Kafka topic partitions in parallel?

I'm using the following in my config.yaml:

triggerMode: ProcessingTime
triggerDuration: 30 seconds
batchMode: true

And for the input section I have:

options:
startingOffsets: earliest
maxOffsetsPerTrigger: 100000

I'm running Spark with spark.dynamicAllocation.enabled set to true.

Thank you