Consumer can be rebalanced if poll is not called before `max.poll.interval.ms`

[allantl]

Starting from Kafka 0.10.1.0, there is max.poll.interval.ms setting:

The maximum delay between invocations of poll() when using consumer group management. This places an upper bound on the amount of time that the consumer can be idle before fetching more records. If poll() is not called before expiration of this timeout, then the consumer is considered failed and the group will rebalance in order to reassign the partitions to another member.

Since, monix-kafka backpressures until all the records has been processed. This could be a problem if processing takes time.

One solution could be to always poll every x second. Then, pause the partition when consumer is still busy processing so no new records are fetched (but it will reset the timer).

void pause(Collection<TopicPartition> partitions) 
Suspend fetching from the requested partitions.

Alpakka-kafka and fs2-kafka are doing this

https://github.com/ovotech/fs2-kafka/blob/master/modules/core/src/main/scala/fs2/kafka/internal/KafkaConsumerActor.scala#L332

https://github.com/akka/alpakka-kafka/blob/master/core/src/main/scala/akka/kafka/internal/KafkaConsumerActor.scala#L435

This will probably require some change, we should just mention in readme to reduce max.poll.records if it hits the limit.

[Avasil]

Thanks!
I will leave the issue open, we will have to do something similar

[thiloplanz]

The README now says

It is resolved in 1.0.0-RC8 by introducing an internal poll heartbeat interval that runs in the background keeping the consumer alive.

But I cannot find that release on Maven Central (or the release tag here).