Issue getting latest Galera3 RPM to work with 5.7.29 on RHEL

Question

Issue getting latest Galera3 RPM to work with 5.7.29 on RHEL

smithjenne opened this issue 5 years ago · 10 comments

I have a 3 node Galera cluster currently running MySQL 5.7.26 on RHEL7.
It is using the mysql-wsrep-5.7.26-25.18-linux-x86_64.tar.gz file that I got from https://galeracluster.com/downloads/.

For years I've been able to successfully update the MySQL Software every quarter. I've also been able to update the Galera Replication Library RPMs on from time to time. This quarter I decided to update both (mysql-wsrep-5.7.29-25.21-linux-x86_64.tar.gz & galera-3-25.3.29-1.el7.x86_64.rpm) using the Binary downloads.

The first thing I did was update the Galera Replication Library RPM to galera-3-25.3.29 on all 3 (MySQL 5.7.26) nodes. I was able to update one node at a time with no cluster outage.

Next I attempted to update the database sofware using mysql-wsrep-5.7.29-25.21-linux-x86_64.tar.gz.

This is where I started having issues. I've tried a few things unsuccessfully.

First I installed MySQL 5.7.29 to one of the nodes (while the other 3 are running).

Shutdown
Updated software
Restart the node, but it won't start. Below is from the error log.
2020-01-31T16:02:06.717405Z mysqld_safe Logging to '/mysql/admin/logs/mysqld_vftest571GC.log'.
2020-01-31T16:02:06.746279Z mysqld_safe Starting mysqld daemon with databases from /mysql/data/vftest571GC
nohup: ignoring input
2020-01-31T16:02:06.944526Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2020-01-31T16:02:06.944600Z 0 [Warning] 'NO_ZERO_DATE', 'NO_ZERO_IN_DATE' and 'ERROR_FOR_DIVISION_BY_ZERO' sql modes should be used with strict mode. They will be merged with strict mode in a future release.
2020-01-31T16:02:06.944604Z 0 [Warning] 'NO_AUTO_CREATE_USER' sql mode was not set.
2020-01-31T16:02:06.954800Z 0 [ERROR] WSREP: Initializing SSL context failed: 336236705: 'error:140A90A1:SSL routines:SSL_CTX_new:library has no ciphers': 22 (Invalid argument)
at galerautils/src/gu_asio.cpp:ssl_init_options():90
2020-01-31T16:02:06.954823Z 0 [ERROR] WSREP: wsrep::init() failed: 7, must shutdown
2020-01-31T16:02:06.954832Z 0 [ERROR] Aborting

Second I shutdown all 3 nodes and just tried to bootstrap one node

Shutdown the other 2 nodes (MySQL 5.6.26)
Disabled SSL (removed socket.ssl_cert, socket.ssl_key from the wsrep_provider_options)
Bootstrapped the MySQL 5.7.29 node but it won't start. Below is from the error log.

2020-01-31T16:07:46.245081Z mysqld_safe Starting mysqld daemon with databases from /mysql/data/vftest571GC
nohup: ignoring input
2020-01-31T16:07:46.432085Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2020-01-31T16:07:46.432168Z 0 [Warning] 'NO_ZERO_DATE', 'NO_ZERO_IN_DATE' and 'ERROR_FOR_DIVISION_BY_ZERO' sql modes should be used with strict mode. They will be merged with strict mode in a future release.
2020-01-31T16:07:46.432172Z 0 [Warning] 'NO_AUTO_CREATE_USER' sql mode was not set.
2020-01-31T16:07:46.445969Z 0 [ERROR] WSREP: context: library has no ciphers
2020-01-31T16:07:46.445993Z 0 [ERROR] WSREP: wsrep::init() failed: 7, must shutdown
2020-01-31T16:07:46.445999Z 0 [ERROR] Aborting

Third. I restarted again with debug mode & SSL disabled

Left SSL disabled
Enabled debug mode (added debug=yes to wsrep_provider_options)
Bootstrapped the MySQL 5.7.29 node but it won't start. Below is from the error log.

2020-01-31T16:07:46.215721Z mysqld_safe Logging to '/mysql/admin/logs/mysqld_vftest571GC.log'.
2020-01-31T16:07:46.245081Z mysqld_safe Starting mysqld daemon with databases from /mysql/data/vftest571GC
nohup: ignoring input
2020-01-31T16:07:46.432085Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2020-01-31T16:07:46.432168Z 0 [Warning] 'NO_ZERO_DATE', 'NO_ZERO_IN_DATE' and 'ERROR_FOR_DIVISION_BY_ZERO' sql modes should be used with strict mode. They will be merged with strict mode in a future release.
2020-01-31T16:07:46.432172Z 0 [Warning] 'NO_AUTO_CREATE_USER' sql mode was not set.
2020-01-31T16:07:46.445969Z 0 [ERROR] WSREP: context: library has no ciphers
2020-01-31T16:07:46.445993Z 0 [ERROR] WSREP: wsrep::init() failed: 7, must shutdown
2020-01-31T16:07:46.445999Z 0 [ERROR] Aborting

2020-01-31T16:07:46.454452Z mysqld_safe mysqld from pid file /mysql/data/vftest571GC/mysqld.pid ended
2020-01-31T16:11:55.924433Z mysqld_safe Logging to '/mysql/admin/logs/mysqld_vftest571GC.log'.
2020-01-31T16:11:55.952586Z mysqld_safe Starting mysqld daemon with databases from /mysql/data/vftest571GC
nohup: ignoring input
2020-01-31T16:11:56.127156Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2020-01-31T16:11:56.127227Z 0 [Warning] 'NO_ZERO_DATE', 'NO_ZERO_IN_DATE' and 'ERROR_FOR_DIVISION_BY_ZERO' sql modes should be used with strict mode. They will be merged with strict mode in a future release.
2020-01-31T16:11:56.127231Z 0 [Warning] 'NO_AUTO_CREATE_USER' sql mode was not set.
2020-01-31T16:11:56.140132Z 0 [ERROR] WSREP: galera/src/wsrep_provider.cpp:galera_init():50: context: library has no ciphers
2020-01-31T16:11:56.140173Z 0 [ERROR] WSREP: wsrep::init() failed: 7, must shutdown
2020-01-31T16:11:56.140188Z 0 [ERROR] Aborting

One final thing. If I change the wsrep_provider back to using the old galera-25.3.12-2.el7.x86_64 on the MySQL 5.7.29 node. I can bootstrap it just fine.

Any help in figuring out what is going on would be appreciated.

Thanks,
Jenn

Answer 1 · 2020-02-09T11:07:49.000Z

I got the same problem.

2020-02-09T10:28:41.808787Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2020-02-09T10:28:41.808879Z 0 [Note] --secure-file-priv is set to NULL. Operations related to importing and exporting data are disabled
2020-02-09T10:28:41.808909Z 0 [Note] /usr/local/mysql/bin/mysqld (mysqld 5.7.29-log) starting as process 33472 ...
2020-02-09T10:28:41.811959Z 0 [Note] WSREP: Read nil XID from storage engines, skipping position init
2020-02-09T10:28:41.811969Z 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera-3/libgalera_smm.so'
2020-02-09T10:28:41.815590Z 0 [Note] WSREP: wsrep_load(): Galera 3.28(rb3295e6) by Codership Oy info@codership.com loaded successfully.
2020-02-09T10:28:41.815610Z 0 [Note] WSREP: CRC-32C: using hardware acceleration.
2020-02-09T10:28:41.815864Z 0 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 1
2020-02-09T10:28:41.815925Z 0 [Note] WSREP: Skipped GCache ring buffer recovery: could not determine history UUID.
2020-02-09T10:28:41.816721Z 0 [Note] WSREP: Passing config to GCS: base_dir = /mysql_a/wsrep; base_host = 192.128.126.130; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /mysql_a/wsrep; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /mysql_a/wsrep/galera.cache; gcache.page_size = 128M; gcache.recover = TRUE; gcache.size = 256M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = FALSE; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout =
2020-02-09T10:28:41.817764Z 0 [Note] WSREP: MemPool(SlaveTrxHandle): hit ratio: 0, misses: 0, in use: 0, in pool: 0
2020-02-09T10:28:41.818261Z 0 [Note] WSREP: Flushing memory map to disk...
2020-02-09T10:28:41.818673Z 0 [ERROR] WSREP: context: library has no ciphers
2020-02-09T10:28:41.818686Z 0 [ERROR] WSREP: wsrep::init() failed: 7, must shutdown
2020-02-09T10:28:41.818690Z 0 [ERROR] Aborting

2020-02-09T10:28:41.818697Z 0 [Note] Binlog end
2020-02-09T10:28:41.818816Z 0 [Note] /usr/local/mysql/bin/mysqld: Shutdown complete

Answer 2 · 2020-02-10T08:10:56.000Z

I have tried galera-3-25.3.29-1.el7.x86_64.rpm + mysql-wsrep-5.7.27-25.19-linux-x86_64.tar.gz, it works fine. but galera-3-25.3.29-1.el7.x86_64.rpm with mysql-wsrep-5.7.28-25.20-linux-x86_64.tar.gz, or mysql-wsrep-5.7.29-25.21-linux-x86_64.tar.gz will run into this kind of problem

Answer 3 · 2020-02-18T11:54:07.000Z

Hi,

Thanks for reporting this!

Upstream MySQL switched to OpenSSL exclusively starting from 5.7.28, and it seems to interfere with Galera SSL operations. Could you post your SSL related configuration settings for both MySQL and Galera for troubleshooting?

Answer 4 · 2020-02-19T13:22:33.000Z

I have not tweaked any SSL related settings in my.cnf , just a few basic settings like port,server_id and some wsrep variables, then "--initialize-insecure" and "mysqld_safe --wsrep-new-cluster &", problem occured.

Answer 5 · 2020-02-19T20:24:23.000Z

Hello,

Here is what I'm using for SSL config settings.

ssl-ca=/mysql/newcerts/ca-cert.pem
ssl-cert=/mysql/newcerts/server-cert.pem
ssl-key=/mysql/newcerts/server-key.pem

wsrep_provider_options = "gmcast.listen_addr=tcp://dba-mdb-d05.andrew.local.cmu.edu:4306;socket.ssl_cert=/mysql/newcerts/server-cert.pem; socket.ssl_key=/mysql/newcerts/server-key.pem; socket.ssl_ca=/mysql/newcerts/ca-cert.pem"

However, note that when I use galera-3-25.3.29-1.el7.x86_64.rpm with mysql-wsrep-5.7.29-25.21-linux-x86_64.tar.gz, I can't bootstrap the cluster even with all SSL setting turned off.

Thanks,
Jenn

Answer 6 · 2020-02-20T15:14:40.000Z

Accidentally closed this issue when I added a comment. I'm reopening because the issue still exists.

Answer 7 · 2020-03-19T13:50:42.000Z

Hello Temeo,

Is there any more information with this?

Thanks,
Jenn

Answer 8 · 2020-04-17T05:58:33.000Z

Hi,

This issue happens because MySQL-wsrep server in binary tarball and Galera library in RPM have been linked with different versions of OpenSSL.

For the next release we aim to release also Galera binary tarball which has been linked with the same OpenSSL as the MySQL-wsrep server, which should fix the problem.

Meanwhile to work around this problem you could use mysqld binary from MySQL-wsrep RPM package built for RHEL/CentOS 7.

Answer 9 · 2020-06-22T08:37:05.000Z

New MySQL-wsrep 5.7.30-25.22 release (http://releases.galeracluster.com/mysql-wsrep-5.7.30-25.22/) is out with Galera 3.30. In this release we introduced a binary tarball package for Galera, which can be found here: http://releases.galeracluster.com/galera-3.30/binary/. The Galera library in binary package has been compiled/linked with the same OpenSSL version as the latest mysql-wsrep binary package, so they should now be compatible.

@smithjenne @ddhe9527 Could you give it a try if it fixes the problem in your case?

Answer 10 · 2020-06-22T14:44:04.000Z

Hello there.
Yes I downloaded the 5.7.30 MySQL-wsrep software along with the 3.30 Galera Binary tarball and that worked.

Thanks,
Jenn