Debezium Connecotrs are failing While Reading Binlog due to Connection Reset/EOF File
Opened this issue · 2 comments
Hi Can you Please help up with an Issue we are facing when we are reading bin-log suddenly during high traffic we get EOF Exception/ Connection Reset Exception
I Read many blogs and saw the issue was caused due to Server ID but in debezium Config we have set database.server.id different for all the MySQL connectors.
Architecture : We have one server with 7 schema and we are using seven MYSQL Connectors to read the servers bin-log, with debezium whitelist property so that each connector will pass data on concerning to it's schema.
In MySQL logs we got this : 2020-09-03T09:34:35.745673Z 87255342 [Note] Aborted connection 87255342 to db: 'unconnected' user: 'kafka_source_user' host: '172.X.X.X' (Failed on my_net_write())
[2020-08-26 04:41:25,967] ERROR Error while deserializing binlog event at offset {ts_sec=1598429128, file=mysql-bin-changelog.179047, pos=120108368, server_id=284246538, event=394}.
Use the mysqlbinlog tool to view the problematic event: mysqlbinlog --start-position=120174225 --stop-position=120174457 --verbose mysql-bin-changelog.179047 (io.debezium.connector.mysql.BinlogReader)
[2020-08-26 04:41:25,967] ERROR Error during binlog processing. Last offset stored = {ts_sec=1598429125, file=mysql-bin-changelog.179047, pos=114769194, row=1, server_id=284246538, event=522}, binlog reader near position = mysql-bin-changelog.179047/120174122 (io.debezium.connector.mysql.BinlogReader)
[2020-08-26 04:41:25,967] ERROR Failed due to error: Error processing binlog event (io.debezium.connector.mysql.BinlogReader)
org.apache.kafka.connect.errors.ConnectException: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1598429128000, eventType=EXT_WRITE_ROWS, serverId=284246538, headerLength=19, dataLength=213, nextPosition=120174457, flags=0}
at io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:230)
at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:208)
at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:508)
at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1095)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:943)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:580)
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:825)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1598429128000, eventType=EXT_WRITE_ROWS, serverId=284246538, headerLength=19, dataLength=213, nextPosition=120174457, flags=0}
at io.debezium.connector.mysql.BinlogReader.handleServerIncident(BinlogReader.java:575)
at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:492)
... 5 more
Caused by: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1598429128000, eventType=EXT_WRITE_ROWS, serverId=284246538, headerLength=19, dataLength=213, nextPosition=120174457, flags=0}
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:300)
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:223)
at io.debezium.connector.mysql.BinlogReader$1.nextEvent(BinlogReader.java:224)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:922)
... 3 more
Caused by: java.io.EOFException
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.read(ByteArrayInputStream.java:190)
at java.io.InputStream.read(InputStream.java:170)
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.fill(ByteArrayInputStream.java:96)
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.read(ByteArrayInputStream.java:89)
at io.debezium.connector.mysql.RowDeserializers.deserializeDatetimeV2(RowDeserializers.java:407)
at io.debezium.connector.mysql.RowDeserializers$WriteRowsDeserializer.deserializeDatetimeV2(RowDeserializers.java:207)
at com.github.shyiko.mysql.binlog.event.deserialization.AbstractRowsEventDataDeserializer.deserializeCell(AbstractRowsEventDataDeserializer.java:178)
at com.github.shyiko.mysql.binlog.event.deserialization.AbstractRowsEventDataDeserializer.deserializeRow(AbstractRowsEventDataDeserializer.java:138)
at com.github.shyiko.mysql.binlog.event.deserialization.WriteRowsEventDataDeserializer.deserializeRows(WriteRowsEventDataDeserializer.java:64)
at com.github.shyiko.mysql.binlog.event.deserialization.WriteRowsEventDataDeserializer.deserialize(WriteRowsEventDataDeserializer.java:56)
at com.github.shyiko.mysql.binlog.event.deserialization.WriteRowsEventDataDeserializer.deserialize(WriteRowsEventDataDeserializer.java:32)
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:294)
... 6 more
[2020-08-26 04:41:25,968] ERROR Error during binlog processing. Last offset stored = {ts_sec=1598429125, file=mysql-bin-changelog.179047, pos=114769194, row=1, server_id=284246538, event=522}, binlog reader near position = mysql-bin-changelog.179047/120174457 (io.debezium.connector.mysql.BinlogReader)
I have encountered a similar problem. Have you solved this problem?
No we were not able to resolve it. Mostly it looked like we were using a single auto scalable disks.. so the connecter where timing out while the other connectors were writing to the disk. When we moved to an architecture with multiple disks.. the issue was minimised but not completely resolved.