Replication Is Broken With Gramine
harunkucuk5 opened this issue · 0 comments
harunkucuk5 commented
Hi, we observed that when we use MariaDB replication with Gramine it fails with the following error
Relay log read failure: Could not parse relay log event entry. The possible reasons are: the master's binary log is corrupted (you can check this by running 'mysqlbinlog' on the binary log), the slave's relay log is corrupted (you can check this by running 'mysqlbinlog' on the relay log), a network problem, or a bug in the master's or slave's MariaDB code. If you want to check the master's binary log or slave's relay log, you will be able to know their names by issuing 'SHOW SLAVE STATUS' on this slave
Without Gramine all is good, but with Gramine we have this error. This is not also related to the sgx because we get the same error with gramine-direct as well.
steps to reproduce the error.
Clone the repository
git pull https://github.com/enclaive/enclaive-docker-mariadb-sgx.gitBuild docker image
docker build -t gramine-mariadb:v2 .then create 2 instances with this docker image and create replication between them.
docker run -it --name master --security-opt seccomp=/root/docker_seccomp_aug_2022.json gramine-mariadb:v2
docker run -it --name replica --security-opt seccomp=/root/docker_seccomp_aug_2022.json gramine-mariadb:v2
CHANGE MASTER TO MASTER_HOST='<master-ip>', MASTER_USER='<user>',MASTER_PASSWORD='<password>',MASTER_USE_GTID = slave_pos;Once the replication is started all is good. But some time later it fails with this error. You can just run the following sysbench test and reproduce this replication error
sysbench oltp_read_only --mysql-host=127.0.0.1 --tables=10 --table_size=50000 --mysql-port=3306 --mysql-user=<user>--mysql-password="<password>" --mysql-db=sbtest preparereplication status
MariaDB [sbtest]> show slave status \G
*************************** 1. row ***************************
Slave_IO_State:
Master_Host: 172.17.0.2
Master_User: root
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: ON.000001
Read_Master_Log_Pos: 7060656
Relay_Log_File: mysqld-relay-bin.000002
Relay_Log_Pos: 5487680
Relay_Master_Log_File: ON.000001
Slave_IO_Running: No
Slave_SQL_Running: No
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 1594
Last_Error: Relay log read failure: Could not parse relay log event entry. The possible reasons are: the master's binary log is corrupted (you can check this by running 'mysqlbinlog' on the binary log), the slave's relay log is corrupted (you can check this by running 'mysqlbinlog' on the relay log), a network problem, or a bug in the master's or slave's MariaDB code. If you want to check the master's binary log or slave's relay log, you will be able to know their names by issuing 'SHOW SLAVE STATUS' on this slave.
Skip_Counter: 0
Exec_Master_Log_Pos: 5487388
Relay_Log_Space: 7061258
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 1595
Last_IO_Error: Relay log write failure: could not queue event from master
Last_SQL_Errno: 1594
Last_SQL_Error: Relay log read failure: Could not parse relay log event entry. The possible reasons are: the master's binary log is corrupted (you can check this by running 'mysqlbinlog' on the binary log), the slave's relay log is corrupted (you can check this by running 'mysqlbinlog' on the relay log), a network problem, or a bug in the master's or slave's MariaDB code. If you want to check the master's binary log or slave's relay log, you will be able to know their names by issuing 'SHOW SLAVE STATUS' on this slave.
Replicate_Ignore_Server_Ids:
Master_Server_Id: 1
Master_SSL_Crl:
Master_SSL_Crlpath:
Using_Gtid: Slave_Pos
Gtid_IO_Pos: 0-1-33
Replicate_Do_Domain_Ids:
Replicate_Ignore_Domain_Ids:
Parallel_Mode: optimistic
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State:
Slave_DDL_Groups: 14
Slave_Non_Transactional_Groups: 0
Slave_Transactional_Groups: 17When we checked the binary logs in the server we verified that they are healthy but the relay logs on the replica is broken. However, if we do the same without gramine it works without any issues.
if we check the relay logs with the following command we see they are corrupted.
show relaylog events IN 'mysqld-relay-bin.000002';
ERROR 1220 (HY000): Error when executing command SHOW BINLOG EVENTS: Wrong offset or I/O error
show relaylog events IN 'mysqld-relay-bin.000002' limit 10,3;
+-------------------------+------+------------+-----------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Log_name | Pos | Event_type | Server_id | End_log_pos | Info |
+-------------------------+------+------------+-----------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| mysqld-relay-bin.000002 | 961 | Query | 1 | 756 | create database test |
| mysqld-relay-bin.000002 | 1048 | Gtid | 1 | 798 | GTID 0-1-4 |
| mysqld-relay-bin.000002 | 1090 | Query | 1 | 1207 | use `test`; CREATE TABLE test_table (
id INT AUTO_INCREMENT PRIMARY KEY,
name VARCHAR(255) NOT NULL,
email VARCHAR(255) NOT NULL,
phone VARCHAR(20),
address VARCHAR(255),
city VARCHAR(100),
state VARCHAR(100),
postal_code VARCHAR(20),
country VARCHAR(100),
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
) |
+-------------------------+------+------------+-----------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
3 rows in set (0.000 sec)