confluentinc/kafka-rest

REST proxy change request JSON format

mithunmanohar opened this issue · 3 comments

mithunmanohar commented

This is more of a question. We are trying to connect an external system to Kafka REST proxy. But that system cannot post the messages in the format specified by REST proxy like:
{ "records": [ { "key": "recordKey", "value": "recordValue" } ] }

Is it possible to modify the REST proxy config to make it accept simple JSON message like { "key": "recordKey", "value": "recordValue" } ?

ruslanruslanruslan commented

Same problem. External systems don't support this crazy format. Why don't you allow free json messages?

domibay-hugo commented

I can only make a supposition but it seems that the JSON structure was meant for batching the data records like:

{"records": [
{"key": "recordKey1", "value": "recordValue1"}, 
{"key": "recordKey2", "value": "recordValue2"},
{"key": "recordKey3", "value": "recordValue3"}
]}

Batching can be desirable to mitigate the protocol overhead of HTTP and improve the data throughput.

To submit the data with 1 HTTP Request -> 1 Topic Record will multiply the protocol overhead with the growth of the data volume and lead to scalability issues much sooner.

alpacino12 commented

I'm too facing the same issue , not able to consume data from external team in this format. Is there any resolution provided for it?