morganstanley/modern-cpp-kafka

How can I create a producer with a larger maximum message size?

adammpolak opened this issue · 4 comments

adammpolak commented

What is the API for the producer to create large messages?

kenneth-jia commented

If you want to handle large messages (e.g. > 1MB), you have to configure for both the producer and the brokers,
https://www.baeldung.com/java-kafka-send-large-message#:~:text=Kafka%20configuration%20limits%20the%20size,default%2C%20this%20limit%20is%201MB.

adammpolak commented

See message.max.bytes in https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md

Thank you but how would I use this field? In the consumer or the producer, or is it some other way I set this variable?

kenneth-jia commented

Here's an exact example for the customized message.max.bytes setting, https://github.com/morganstanley/modern-cpp-kafka/blob/main/tests/integration/TestKafkaProducer.cc#L594
In short, just need to add (kafka::clients::producer::Config::MESSAGE_MAX_BYTES, "2000000") into the Properties which is used to initialize the producer. (and no other way to set it after the producer being created)