morganstanley/modern-cpp-kafka

Single Message Consuming

KevDi opened this issue · 2 comments

KevDi commented

Currently only the Batch Consuming of Messages is possible with the consumer.
I could consume one Message at each call when i set the max.pool.records. But internally it still does a batch poll.
Is it possible to do a single polling of Messages like in the RDKafka High Level Consumer?

kenneth-jia commented

Hi, @KevDi
It is possible, as long as librdkafka supports the "single polling" (since RDKafka High Level Consumer does, I guess it is the case). The only problem is "why do we want it", or "why do we need to care about the underlying batch poll"?
You know, we are able to commit the offsets for each message, (1) manually commit the offset after processing, OR (2) if we want enable.auto.commit=true, set max.poll.records=1 as well

KevDi commented

@kenneth-jia this issue could be closed.
I was doing some test regarding performance to see if there are any difference in using the librdkafka Wrapper or directly the librdkafka library.
But after i set some of the Parameters (especially the batch.size) i got nearly the same performance.

  • 👍1