Missing events and TableMetadataUnavailableError exception raised due to column schema empty in table map - MySql 8.0.34
ShayYannay21 opened this issue · 7 comments
Hi,
Noticed a strange behavior in mysql 8.0.34 of missing binlog events when running against MySql 8.0.34 with mysql-replication package of 0.43.0 (currently the latest).
I ran some tests that first create a Table in MySql than insert a row, in that phase the BinLogStreamReader able to consume the events but on second Table creation (different table) and insert row, though the event is reaching the stream in BinLogStreamReader but the rows property is empty.
Found out that the reason of for that is in RowEvent object , table_map.columns_schema is not updated causing the rows to be empty, see code in here, the RowEvent is marked as incomplete.
When I set configuration fail_on_table_metadata_unavailable indeed the exception of TableMetadataUnavailableError raised as expected.
BTW after restarting mysql (tested this with docker) I was able to consume the missing event so it look like that when TableMap event occur the in memory data structure that the python-mysql-replication project relays on is not updated.
Saw this previous issue that setting auto commit to true can resolve this, I've try that but it didn't worked.
Wonder if python-mysql-replication project support MySql 8 in the first place since according to the project readme MySql 5.7 is the latest supported version, MySql 5.7 is reaching end of life and supporting MySql 8 can benefit allot of people using this great project.
Thanks for the help.
https://github.com/julien-duponchelle/python-mysql-replication/pull/431/files
hello @ShayYannay21
I think may be this PR is relation with your problem. (This pr Merged but not updated at latest version)
We had an issue where a connection was being created along with the opening of a transaction. This led to a problem with looking at the snapshot before creating the table. We have since resolved this issue
can you edit code like this? or run code on main branch
let me know How works!
Thanks @sean-k1 i've created package from master which contains the changed in the pr you mentioned above.
Tested it and it works!
Do you know when it will be available in pypi releases ?
note that main branch is set to 0.43.0 but its not in sync with the code in mysql-replication in pypi version 0.43.0
@ShayYannay21 Oh that's good! but I dont know when it will be available pypi release version
@julien-duponchelle Hello julien can u let me know, when this PR will be applied on PyPI?
Could you help me by doing a pr to prepare an updated changelog . Their is so much changes 🤣
@julien-duponchelle oh okay I will try it
@julien-duponchelle Can you closed this Isuue? I think issue is resolved~
@sean-k1 @julien-duponchelle thanks! for the help