loadmill/loadmill-kafka-relay

Deocde Error on Long type when message is in Avro.

yonathan-shtekel opened this issue · 1 comments

Hey guys,
For Avro messages when message contain Long type the decode is failing.
I'm not a javascript professional but there is a different between long type in Avro and bigInt in javascript and maybe need to create a new logicalType that make this conversion.

any way this is the error:
[2023-09-06 23:11:19.376] ERROR: [KAFKA]: {"timestamp":"2023-09-06T20:11:19.376Z","logger":"kafkajs","message":"Error when calling eachMessage","topic":"some-topic-name","partition":21,"offset":"14","stack":"Error: potential precision loss\n at LongType._read (/Users/yonathan.shtekel/Riskified/helpers/loadmill-kafka-relay/node_modules/avsc/lib/types.js:918:11)\n at RecordType.readEventMetaData [as _read] (eval at RecordType._createReader (/Users/yonathan.shtekel/Riskified/helpers/loadmill-kafka-relay/node_modules/avsc/lib/types.js:2296:10), <anonymous>:5:8)\n at RecordType.readLinkingFeaturesExtractionEvent [as _read] (eval at RecordType._createReader (/Users/yonathan.shtekel/Riskified/helpers/loadmill-kafka-relay/node_modules/avsc/lib/types.js:2296:10), <anonymous>:5:8)\n at readValue (/Users/yonathan.shtekel/Riskified/helpers/loadmill-kafka-relay/node_modules/avsc/lib/types.js:2914:17)\n at RecordType.Type.fromBuffer (/Users/yonathan.shtekel/Riskified/helpers/loadmill-kafka-relay/node_modules/avsc/lib/types.js:601:13)\n at SchemaRegistry.decode (/Users/yonathan.shtekel/Riskified/helpers/loadmill-kafka-relay/node_modules/@kafkajs/confluent-schema-registry/src/SchemaRegistry.ts:331:25)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)","error":{}}

way to reproduce:
Create a scheme with long type property and produce and consume the message from topic.
This is happening at the subscribe end point, when the service is actually consuming the messages.

Solved in #15