exception of "java.io.IOException: Stumbled upon long even though int expected" when deserialize GTID Transaction event
Opened this issue · 1 comments
shenjunstar commented
background: we've encounted a exception "java.io.IOException: Stumbled upon long even though int expected" when using debezium mysql connector to do binlog processing.
exception :
io.debezium.DebeziumException: Failed to deserialize data of EventHeaderV4{timestamp=1709171907000, eventType=GTID, serverId=28775006, headerLength=19, dataLength=62, nextPosition=80207890, flags=0}
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.wrap(MySqlStreamingChangeEventSource.java:1091)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$ReaderThreadLifecycleListener.onEventDeserializationFailure(MySqlStreamingChangeEventSource.java:1144)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1081)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:648)
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:949)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1709171907000, eventType=GTID, serverId=28775006, headerLength=19, dataLength=62, nextPosition=80207890, flags=0}
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:343)
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:246)
at io.debezium.connector.mysql.strategy.AbstractBinaryLogClientConfigurator$1.nextEvent(AbstractBinaryLogClientConfigurator.java:110)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1068)
... 3 more
Caused by: java.io.IOException: Stumbled upon long even though int expected
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.readPackedInteger(ByteArrayInputStream.java:157)
at com.github.shyiko.mysql.binlog.event.deserialization.GtidEventDataDeserializer.deserialize(GtidEventDataDeserializer.java:90)
at com.github.shyiko.mysql.binlog.event.deserialization.GtidEventDataDeserializer.deserialize(GtidEventDataDeserializer.java:27)
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:337)
... 6 more
we've found that there is a problem that when processing a big transaction , if the transaction length is larger than the max value of Integer which is most probably occurred in production. Here is the code below :
// com.github.shyiko.mysql.binlog.event.deserialization.GtidEventDataDeserializer#deserialize
if (inputStream.available() >= TRANSACTION_LENGTH_MIN_LENGTH) {
transactionLength = inputStream.readPackedInteger();
}
// com.github.shyiko.mysql.binlog.io.ByteArrayInputStream#readPackedInteger
public int readPackedInteger() throws IOException {
Number number = readPackedNumber();
if (number == null) {
throw new IOException("Unexpected NULL where int should have been");
}
if (number.longValue() > Integer.MAX_VALUE) {
throw new IOException("Stumbled upon long even though int expected");
}
return number.intValue();
}
so we add a function to read packed long in ByteArrayInputStream
to resolve this:
/**
* @see #readPackedNumber()
* @throws IOException in case of malformed number, eof, null
* @return long
*/
public long readPackedLong() throws IOException {
Number number = readPackedNumber();
if (number == null) {
throw new IOException("Unexpected NULL where long should have been");
}
return number.longValue();
}
shenjunstar commented
a pr request submitted
#141