firehose event gets truncated/appended garbage characters
Closed this issue · 3 comments
I am playing with this plugin with packetbeat, some of the messages send to firehose gets truncated or appended garbage characters. Which fails deliver to ES. Not sure why it is happening
firehose raw data: eyJAdGltZXN0YW1wIjoiMjAxOC0wNS0yMVQyMzoyODo0MC4wMDBaIiwiQG1ldGFkYXRhIjp7ImJlYXQiOiJwYWNrZXRiZWF0IiwidHlwZSI6ImRvYyIsInZlcnNpb24iOiI2LjIuNCJ9LCJmaW5hbCI6ZmFsc2UsImJlYXQiOnsibmFtZSI6ImlwLTEtMTEtMy0yMDIiLCJob3N0bmFtZSI6ImlwLTEtMTEtMy0yMDIiLCJ2ZXJzaW9uIjoiNi4yLjQifSwic291cmNlIjp7ImlwdjYiOiJmZTgwOjo0NDo3YmZmOmZlZDg6YTIyZSIsInBvcnQiOjU0Niwic3RhdHMiOnsibmV0X3BhY2tldHNfdG90YWwiOjEsIm5ldF9ieXRlc190b3RhbCI6MTE2fX0sImxhc3RfdGltZSI6IjIwMTgtMDUtMjFUMjM6Mjg6MDkuMTc5WiIsInN0YXJ0X3RpbWUiOiIyMDE4LTA1LTIxVDIzOjI4OjA5LjE3OVoiLCJ0eXBlIjoiZmxvdyIsImZsb3dfaWQiOiJRQUwvLy8vLy8vOEEvLzhnLy84QUFBSCtnQUFBQUFBQUFBQkVlLy8rMktJdS93SUFBQUFBQUFBQUFBQUFBQUVBQWlJQ0l3SSIsImZpZWxkcyI6eyJsb2dfc291cmNlIjoidGF5bG9yLXBhY2tldGJlYXQifSwibWV0YSI6eyJjbG91ZCI6eyJhdmFpbGFiaWxpdHlfem9uZSI6ImV1LXdlc3QtMWIiLCJwcm92aWRlciI6ImVjMiIsImluc3RhbmNlX2lkIjoiaS0wYWRjZmUxMTg1MmViMmQxNiIsIm1hY2hpbmVfdHlwZSI6InQyLm1lZGl1bSIsInJlZ2lvbiI6ImV1LXdlc3QtMSJ9fSwidHJhbnNwb3J0IjoidWRwIiwiZGVzdCI6eyJpcHY2IjoiZmYwMjo6MToyIiwicG9ydCI6NTQ3fX1BQUFGL0FBQUJmdw
json:
{"@timestamp":"2018-05-21T23:28:40.000Z","@metadata":{"beat":"packetbeat","type":"doc","version":"6.2.4"},"final":false,"beat":{"name":"ip-1-11-3-202","hostname":"ip-1-11-3-202","version":"6.2.4"},"source":{"ipv6":"fe80::44:7bff:fed8:a22e","port":546,"stats":{"net_packets_total":1,"net_bytes_total":116}},"last_time":"2018-05-21T23:28:09.179Z","start_time":"2018-05-21T23:28:09.179Z","type":"flow","flow_id":"QAL///////8A//8g//8AAAH+gAAAAAAAAABEe//+2KIu/wIAAAAAAAAAAAAAAAEAAiICIwI","fields":{"log_source":"packetbeat"},"meta":{"cloud":{"availability_zone":"eu-west-1b","provider":"ec2","instance_id":"i-xxxx","machine_type":"t2.medium","region":"eu-west-1"}},"transport":"udp","dest":{"ipv6":"ff02::1:2","port":547}}AAAF/AAABfw
@di1214 Hi. I believe I spotted the source of issue.
This is data corruption. awsbeats is passing output from beat json encoder's output to the record sent to Firehose. This is wrong - beat's json encoder isn't designed to be used this way. It should copy the output before further usage.
So, instead of:
return &firehose.Record{Data: serializedEvent}, nilWe should write:
buf := make([]byte, len(serializedEvent))
copy(buf, serializedEvent)
serializedEvent = buf
return &firehose.Record{Data: serializedEvent}, nil