Parameter guaranteeOrdering=true reduces producer throughput

Question

Parameter guaranteeOrdering=true reduces producer throughput

Opened this issue 5 years ago · 4 comments

When the KafkaProducer's parameter guaranteeOrdering is true the operator automatically sets max.in.flight.requests.per.connection to 1 when retries are enabled (retries > 0).
Allowing only one in-flight PRODUCE request, reduces throughput to the broker.

The better choice would be enable.idempotence=true, which allows max.in.flight.requests.per.connection <= 5 for maximum throughput.

https://medium.com/@stephane.maarek/new-features-of-kafka-2-1-33fb5396b546

Answer 1 · 2019-06-26T07:14:56.000Z

As a dependency, acks must be set to all, when enable.idempotence is set true. Otherwise a ConfigException will be thrown.

Answer 2 · 2019-09-26T12:49:11.000Z

When using Kafka 0.10.2 (still supported):

org.apache.kafka.common.errors.UnsupportedVersionException: Attempting to use idempotence with a broker which does not support the required message format (v2). The broker must be version 0.11 or later.

Answer 3 · 2019-09-26T13:30:23.000Z

Using the idempotent producer requires dropping the support of Kafka 0.10.2 when the guaranteeOrdering parameter is used with true.

Answer 4 · 2019-11-26T12:59:46.000Z

resolved in Release 3.0.0.