Restricting Message size from no-kafka Producer

Question

Restricting Message size from no-kafka Producer

VineetKillerX opened this issue 7 years ago · 10 comments

Hi,
We want to restrict the message size being posted to Kafka Topic. This is the File Upload Scenario where the contents of the file are sent by the producer to the Broker. What specific properties need to be set in the Producer Config. We want to restrict message from Going to Kafka if the file size is greater than 6Mb.

Answer 1 · 2017-10-18T12:07:22.000Z

Why not restrict message size in your own code before sending it with producer.send()?

Answer 2 · 2017-10-18T12:56:26.000Z

Okay, We can do that as well but my main concern is over what limit can the no-kafka producer send the message( in size) ? I get this error message frequently.

"MessageSizeTooLarge","message":"The server has a configurable maximum message size to avoid unbounded memory allocation. This error is thrown if the client attempt to produce a message larger than this maximum.

But at times when I upload the same file(of around 5Mb) it reaches Kafka.

I checked the broker configs and all the size related parameters are set to 6MB.

Answer 3 · 2017-10-18T13:02:37.000Z

The only parameters that control size of sent messages in producer are batch options: https://github.com/oleksiyk/kafka#producer-options

With batch mode disabled all messages will be sent as they are and all responsibility on the message size is on library user.

Answer 4 · 2017-10-18T13:07:13.000Z

Okay so that means if I disable the batch option, then I will be able to send messages upto the limit set up on Broker side?

Answer 5 · 2017-10-18T13:11:09.000Z

No, it means batch mode possibly was the reason if your batch.size option was greater than 6MB.

Answer 6 · 2017-10-18T13:13:00.000Z

Okay so when do we generally get this Error ?

"MessageSizeTooLarge","message":"The server has a configurable maximum message size to avoid unbounded memory allocation. This error is thrown if the client attempt to produce a message larger than this maximum.

Is there some producer configuration we need to take care of?

Answer 7 · 2017-10-18T13:14:17.000Z

Please read my above replies.

Answer 8 · 2017-10-18T13:20:45.000Z

I read those replies but I didnt get any configuration for max.request.size.
Any way we can configure this? or this Change can help us to avoid MessageSizeTooLarge Error?

Answer 9 · 2017-10-18T13:34:05.000Z

Producer will send messages as they were passed to send() method.
Don't forget send() accepts an array of messages which will always be sent in one single request to the broker.

Answer 10 · 2017-10-18T13:37:57.000Z

So,

make sure you don't use batch mode which accumulates messages
make sure you don't send multiple files at once, e.g. like send([5MB, 5MB])
make sure you don't send any single messages larger than a limit configured in broker