mostafa/xk6-kafka

magic byte is added when using AVRO encoding without schema registry

thmshmm opened this issue · 3 comments

thmshmm commented

Hi,

is it intended that the schema id is also added even in the case when no schema registry is used?

xk6-kafka/serdes.go

Line 52 in 63b5c93

return k.encodeWireFormat(bytesData, container.Schema.ID)

If intended, it might be good to have a small hint in the docs. As a workaround message.value.splice(0, 5); can be used to remove it.

Kr Thms

mostafa commented

Hey @thmshmm,

The value of this integer field, ID, is populated only when calling getSchema or createSchema, hence, its value will be 0 in other cases, which results in a 5-bytes zero padding being added. This will be skipped automatically in the call to deserialize, and later decodeWireFormat. Can you elaborate?

thmshmm commented

I just stumbled upon this because I wanted to test producers and consumers that don't use the schema registry and also don't have the 5-byte padding. It was just not intuitive to me that the bytes were added even when not using the registry. As this is a technical necessity of the registry, not of using AVRO itself.

It's not an issue or a bug. Just something to be aware of.

  • 👀1
mostafa commented

@thmshmm
Thanks for the clarification, and feel free to update the docs as you see fit. 🙏

  • 👍1