LookupError: unknown encoding: utf8mb3
blackoctop opened this issue · 8 comments
I have super simple example of BinLogStreamReader:
mysql_settings =
{'host': host,
'port': port,
'user': user,
'passwd': passw
}
stream = BinLogStreamReader(
connection_settings=mysql_settings,
server_id=1024,
is_mariadb=True,
# blocking=True,
only_events=[DeleteRowsEvent, UpdateRowsEvent, WriteRowsEvent],
# resume_stream=True,
only_tables=[table],
only_schemas=[schema]
)
which goes on exception on line "for row in binlogevent.rows:"
for binlogevent in stream:
for row in binlogevent.rows:
print(row)
stream.close()
Traceback (most recent call last):
File "C:\Users\user\Desktop\pyProjects\replica\binlogreader\test.py", line 93, in
print(binlogevent.rows)
File "C:\Users\user\Desktop\pyProjects\replica.env\lib\site-packages\pymysqlreplication\row_event.py", line 459, in rows
self._fetch_rows()
File "C:\Users\user\Desktop\pyProjects\replica.env\lib\site-packages\pymysqlreplication\row_event.py", line 454, in _fetch_rows
self.__rows.append(self._fetch_one_row())
File "C:\Users\user\Desktop\pyProjects\replica.env\lib\site-packages\pymysqlreplication\row_event.py", line 545, in _fetch_one_row
row["after_values"] = self._read_column_data(self.columns_present_bitmap2)
File "C:\Users\user\Desktop\pyProjects\replica.env\lib\site-packages\pymysqlreplication\row_event.py", line 154, in _read_column_data
values[name] = self.__read_string(2, column)
File "C:\Users\user\Desktop\pyProjects\replica.env\lib\site-packages\pymysqlreplication\row_event.py", line 250, in __read_string
string = string.decode(encoding)
LookupError: unknown encoding: utf8mb3
I added these lines in pymsql/charset.py and LookupError was fixed.
_charsets.add(Charset(256, "utf8mb3", "utf8mb3_general_ci", "Yes"))
_charsets.add(Charset(257, "utf8mb3", "utf8mb3_bin", ""))
but I am not sure if this is a good practice...
Hello, I have the exact same issue.
I am also looking for a more long-term fix rather than juste tweaking the pymysql library.
@blackoctop did you try to submit a PR to pymysql to include your fix?
@the4thdoctor I opened issue and got this answer:
methane commented 24 days ago
This is pymysqlreplication issue.
@methane methane closed this as [not planned]
🤦
We also got this issue when I updated MySQL RDS from version 5.7 to 8.0.
In MySQL 8.0, the utf8mb3
character set is deprecated, so tables/columns with the utf8mb3
character set gave the error. So we have changed the character set of our tables from utf8mb3
to utf8mb4
.
ALTER TABLE sh.subjects
DEFAULT CHARACTER SET utf8mb4,
MODIFY subject_desc varchar(50)
CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL;
Ref. https://dev.mysql.com/doc/refman/8.0/en/mysql-nutshell.html#mysql-nutshell-deprecations
I added these lines in pymsql/charset.py and LookupError was fixed. _charsets.add(Charset(256, "utf8mb3", "utf8mb3_general_ci", "Yes")) _charsets.add(Charset(257, "utf8mb3", "utf8mb3_bin", ""))
but I am not sure if this is a good practice...
this worked for me! was wondering if anybody found a better solution though?
I added these lines in pymsql/charset.py and LookupError was fixed. _charsets.add(Charset(256, "utf8mb3", "utf8mb3_general_ci", "Yes")) _charsets.add(Charset(257, "utf8mb3", "utf8mb3_bin", ""))
but I am not sure if this is a good practice...
thanks!