julien-duponchelle/python-mysql-replication

Decode Exception due to Inaccurate Length from self.query_length Leading to Byte Sequence Truncation

Closed this issue · 3 comments

zj5220924 commented

Version

Please specify the versions you are using. Exact version numbers are preferred.

  • Pymyrepl (1.1.0):
  • OS (Centos 7):
  • Pymyrepl (1.0.6):
  • Database and version (Remove unnecessary options):
    • MySQL: 8.0.32

System Variables

List relevant system variables using the query SHOW VARIABLES LIKE '<variable_name_here>';

  • log_bin: YES
  • binlog_format: ROW
  • binlog_row_image: FULL
  • enforce_gtid_consistency: True
  • gtid_mode: YES

If you are using MySQL 8.0.14 or later, please also provide the following variables:

  • binlog_row_metadata: FULL
  • binlog_row_value_options: Null

error massage:
Traceback (most recent call last):
File "/ROOT/tmp/my2sql/my2sql.py", line 144, in
binlog2sql.process_binlog()
File "/ROOT/tmp/my2sql/my2sql.py", line 72, in process_binlog
for binlog_event in stream:
File "/usr/local/python3.11/lib/python3.11/site-packages/pymysqlreplication/binlogstream.py", line 617, in fetchone
binlog_event = BinLogPacketWrapper(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11/lib/python3.11/site-packages/pymysqlreplication/packet.py", line 115, in init
self.event = event_class(
^^^^^^^^^^^^
File "/usr/local/python3.11/lib/python3.11/site-packages/pymysqlreplication/event.py", line 885, in init
self.query = self.packet.read(self.query_length).decode("utf-8")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 205-206: unexpected end of data

cat /....../pymysqlreplication/event.py
......
def init(self, from_packet, event_size, table_map, ctl_connection, **kwargs):
super(RowsQueryLogEvent, self).init(
from_packet, event_size, table_map, ctl_connection, **kwargs
)
self.query_length = self.packet.read_uint8()
self.query = self.packet.read(self.query_length).decode("utf-8")
print(self.query_length)
print(self.query)
......

After testing, it was found that the length obtained from self.query_length is inaccurate, causing the byte sequence read by self.packet.read to be truncated, leading to decoding exceptions.

dongwook-chan commented

@zj5220924
Thank you for reporting this issue.

the length obtained from self.query_length is inaccurate

You're indeed right. The length must be ignored without being used in parsing the query. code from msql-server suggests so.

Other clients in different languages implement the RowsQueryEvent the way mysql-server does.

I'll have this fixed ASAP.

dongwook-chan commented

@zj5220924
Could you please clarify steps to reproduce so that I could add test cases?
Please provide at least 1 query.

zj5220924 commented

@dongwook-chan
22159.log
This is my test data. thx~

  • 👍1