Issue with Upgrading Zabbix from 6.0.2 to 6.4: Database Upgrade Failing in HA Mode
alemsas opened this issue · 19 comments
I am currently running Zabbix 6.0.2 in a Docker environment and attempting to upgrade to Zabbix 6.4. My setup includes Zabbix server and agent containers, with the Zabbix server using MySQL (Percona XtraDB Cluster). I have followed the upgrade instructions, but the upgrade process fails because the Zabbix server still attempts to operate in HA mode during the database upgrade.
Current Setup:
Database: Percona XtraDB Cluster
My configuration:
cat docker-compose.yml
version: "3.9"
services:
zabbix-server-1:
image: 'zabbix/zabbix-server-mysql:ubuntu-6.4-latest'
# ports:
# - "10051:10051"
volumes:
- /etc/localtime:/etc/localtime:ro
- /var/lib/snmp/mibs:/var/lib/zabbix/mibs:ro
- './usr/lib/zabbix/alertscripts:/usr/lib/zabbix/alertscripts:ro'
- './usr/lib/zabbix/externalscripts:/usr/lib/zabbix/externalscripts:ro'
environment:
- DB_SERVER_HOST=172.31.63.1
- DB_SERVER_PORT=3306
- ZBX_HANODENAME=zabbix-server-1
- ZBX_NODEADDRESS=172.31.63.1
env_file:
- ./envs/.env_mysql
- ./envs/.env_server
container_name: zabbix-server-1
hostname: zbxSRV-1
restart: unless-stopped
network_mode: host
ulimits:
nproc: 65535
nofile:
soft: 20000
hard: 40000
stop_grace_period: 30s
zabbix-agent-1:
image: 'zabbix/zabbix-agent:ubuntu-latest'
volumes:
- /etc/localtime:/etc/localtime:ro
- './zabbix-server/zabbix_agentd.d:/etc/zabbix/zabbix_agentd.d:ro'
- './zabbix-server/var/lib/zabbix:/var/lib/zabbix:ro'
env_file:
- ./envs/.env_agent
environment:
- ZBX_SERVER_HOST=127.0.0.1,172.31.63.1
- ZBX_ACTIVE_ALLOW=false
privileged: true
container_name: zabbix-agent-1
hostname: zbxAgent-1
restart: unless-stopped
pid: "host"
network_mode: host
stop_grace_period: 5s
also here is my docker logs for zabbix-server
docker logs -f --tail=30 zabbix-server-1
** Updating '/etc/zabbix/zabbix_server.conf' parameter "TLSCipherPSK13": ''...removed
** Updating '/etc/zabbix/zabbix_server.conf' parameter "TLSKeyFile": ''...removed
** Updating '/etc/zabbix/zabbix_server.conf' parameter "TLSPSKIdentity": ''...removed
** Updating '/etc/zabbix/zabbix_server.conf' parameter "TLSPSKFile": ''...removed
** Updating '/etc/zabbix/zabbix_server.conf' parameter "ServiceManagerSyncFrequency": ''...removed
** Updating '/etc/zabbix/zabbix_server.conf' parameter "HANodeName": 'zabbix-server-1'...updated
** Updating '/etc/zabbix/zabbix_server.conf' parameter "NodeAddress": '172.31.63.1'...updated
** Updating '/etc/zabbix/zabbix_server.conf' parameter "User": 'zabbix'...updated
Starting Zabbix Server. Zabbix 6.4.14 (revision 0a50e61).
Press Ctrl+C to exit.
7:20240516:232828.012 Starting Zabbix Server. Zabbix 6.4.14 (revision 0a50e61).
7:20240516:232828.012 ****** Enabled features ******
7:20240516:232828.012 SNMP monitoring: YES
7:20240516:232828.012 IPMI monitoring: YES
7:20240516:232828.012 Web monitoring: YES
7:20240516:232828.012 VMware monitoring: YES
7:20240516:232828.012 SMTP authentication: YES
7:20240516:232828.012 ODBC: YES
7:20240516:232828.012 SSH support: YES
7:20240516:232828.012 IPv6 support: YES
7:20240516:232828.012 TLS support: YES
7:20240516:232828.012 ******************************
7:20240516:232828.012 using configuration file: /etc/zabbix/zabbix_server.conf
7:20240516:232828.044 Zabbix supports only "utf8_bin,utf8mb3_bin,utf8mb4_bin" collation(s). Database "zabbix" has default collation "utf8mb4_0900_ai_ci"
7:20240516:232828.080 current database version (mandatory/optional): 06000000/06000020
7:20240516:232828.080 required mandatory version: 06040000
7:20240516:232828.080 mandatory patches were found
7:20240516:232828.082 cannot perform database upgrade in HA mode: all nodes need to be stopped and Zabbix server started in standalone mode for the time of upgrade.
7:20240516:232828.082 Zabbix Server stopped. Zabbix 6.4.14 (revision 0a50e61).
Steps Taken:
- Dynamically in MySQL:SET GLOBAL log_bin_trust_function_creators = 1;
- Updated Docker Image in docker-compose.yml:
- Cleared the ha_node Table:
4.Set Environment Variable to Disable HA Mode with remove below into my compose!
- ZBX_HANODENAME=zabbix-server-1
- ZBX_NODEADDRESS=172.31.63.1
Request for Help: What additional steps or configurations are necessary to ensure the Zabbix server starts in standalone mode to complete the database upgrade? Are there any known issues or additional configurations required when upgrading from 6.0.2 to 6.4 that I might have overlooked?
Also i increase debug level
and stiil i get this
6:20240517:093156.843 End of zbx_db_connect():0
6:20240517:093156.844 In zbx_dbms_version_info_extract()
6:20240517:093156.844 End of zbx_dbms_version_info_extract() version:80036
6:20240517:093156.845 In DBcheck_version()
6:20240517:093156.845 In zbx_db_connect() flag:0
6:20240517:093156.875 End of zbx_db_connect():0
6:20240517:093156.876 query [txnlev:0] [show tables like 'dbversion']
6:20240517:093156.877 query [txnlev:0] [select mandatory,optional from dbversion]
6:20240517:093156.878 current database version (mandatory/optional): 06000000/06000020
6:20240517:093156.878 required mandatory version: 06040000
6:20240517:093156.879 mandatory patches were found
6:20240517:093156.879 query [txnlev:0] [show tables like 'ha_node']
6:20240517:093156.880 query [txnlev:1] [begin;]
6:20240517:093156.881 query [txnlev:1] [select unix_timestamp(),ha_failover_delay from config]
6:20240517:093156.882 query [txnlev:1] [select lastaccess,name from ha_node where status not in (1,2) order by ha_nodeid for update]
6:20240517:093156.882 query [txnlev:1] [commit;]
6:20240517:093156.883 cannot perform database upgrade in HA mode: all nodes need to be stopped and Zabbix server started in standalone mode for the time of upgrade.
6:20240517:093156.883 End of DBcheck_version():FAIL
6:20240517:093156.884 Zabbix Server stopped. Zabbix 6.4.14 (revision 0a50e61).
No one can help?!
hi! Please provide full container log without debug and without defined vars ZBX_HANODENAME and ZBX_NODEADDRESS.
@dotneft
I have some changes, and I run databse with galera, also Here is my galera configuration:
version: "3.9"
services:
mariadb-galera-1:
image: 'bitnami/mariadb-galera:latest'
volumes:
- /etc/localtime:/etc/localtime:ro
- '/home/zabbix-cluster/mariadb-galera/Data/zabbix:/bitnami/mariadb'
- '/home/zabbix-cluster/mariadb-galera/conf/my_custom.cnf:/opt/bitnami/mariadb/conf/my_custom.cnf:ro'
env_file:
- /home/zabbix-cluster/mariadb-galera/env_vars/.env_db
environment:
- MARIADB_GALERA_CLUSTER_BOOTSTRAP=yes
- MARIADB_GALERA_FORCE_SAFETOBOOTSTRAP=yes
- MARIADB_GALERA_CLUSTER_ADDRESS=gcomm://172.31.63.1:4567,172.31.63.2:4567,172.31.63.44:4567
- MARIADB_GALERA_NODE_ADDRESS=172.31.63.1
- MARIADB_EXTRA_FLAGS=--max-connect-errors=1000 --max_connections=1000
ulimits:
nproc: 65535
nofile:
soft: 20000
hard: 40000
network_mode: host
container_name: mariadb-galera-1
hostname: ba1-DC-SRV1
restart: unless-stopped
and without debug and without ZBX_HANODENAME and ZBX_NODEADDRESS, I got:
docker logs -f --tail=30 zabbix-server-1
** Updating '/etc/zabbix/zabbix_server.conf' parameter "TLSCipherPSK13": ''...removed
** Updating '/etc/zabbix/zabbix_server.conf' parameter "TLSKeyFile": ''...removed
** Updating '/etc/zabbix/zabbix_server.conf' parameter "TLSPSKIdentity": ''...removed
** Updating '/etc/zabbix/zabbix_server.conf' parameter "TLSPSKFile": ''...removed
** Updating '/etc/zabbix/zabbix_server.conf' parameter "ServiceManagerSyncFrequency": ''...removed
** Updating '/etc/zabbix/zabbix_server.conf' parameter "HANodeName": 'zabbix-server-1'...updated
** Updating '/etc/zabbix/zabbix_server.conf' parameter "NodeAddress": '172.31.63.1'...updated
** Updating '/etc/zabbix/zabbix_server.conf' parameter "User": 'zabbix'...updated
Starting Zabbix Server. Zabbix 6.4.14 (revision 0a50e61).
Press Ctrl+C to exit.
7:20240516:232828.012 Starting Zabbix Server. Zabbix 6.4.14 (revision 0a50e61).
7:20240516:232828.012 ****** Enabled features ******
7:20240516:232828.012 SNMP monitoring: YES
7:20240516:232828.012 IPMI monitoring: YES
7:20240516:232828.012 Web monitoring: YES
7:20240516:232828.012 VMware monitoring: YES
7:20240516:232828.012 SMTP authentication: YES
7:20240516:232828.012 ODBC: YES
7:20240516:232828.012 SSH support: YES
7:20240516:232828.012 IPv6 support: YES
7:20240516:232828.012 TLS support: YES
7:20240516:232828.012 ******************************
7:20240516:232828.012 using configuration file: /etc/zabbix/zabbix_server.conf
7:20240516:232828.044 Zabbix supports only "utf8_bin,utf8mb3_bin,utf8mb4_bin" collation(s). Database "zabbix" has default collation "utf8mb4_0900_ai_ci"
7:20240516:232828.080 current database version (mandatory/optional): 06000000/06000020
7:20240516:232828.080 required mandatory version: 06040000
7:20240516:232828.080 mandatory patches were found
7:20240516:232828.082 cannot perform database upgrade in HA mode: all nodes need to be stopped and Zabbix server started in standalone mode for the time of upgrade.
7:20240516:232828.082 Zabbix Server stopped. Zabbix 6.4.14 (revision 0a50e61).
Also I try these steps:
- Run galera in standalone mode with remove bootstrap,gcom
- Fill my database with my backup from old database machine
- Run Zabbix-Server without HA enviornment
- Then stop zabbix server and galera, and run galera with cluster mode and add HA to zabbix-server, but still have same error.
it is starting WITH ZBX_HANODENAME and ZBX_NODEADDRESS:
** Updating '/etc/zabbix/zabbix_server.conf' parameter "HANodeName": 'zabbix-server-1'...updated
** Updating '/etc/zabbix/zabbix_server.conf' parameter "NodeAddress": '172.31.63.1'...updated
@dotneft
Thanks, but I remove or comment
- ZBX_HANODENAME=zabbix-server-1
- ZBX_NODEADDRESS=172.31.63.1
in my zabbix server compose as well as zabbix.conf but I still get it!
I import the zabbix server configuration by .env
**.env_server**
# ZBX_LISTENIP=
# ZBX_LISTENBACKLOG=
# ZBX_HISTORYSTORAGEURL=http://elasticsearch:9200/ # Available since 3.4.5
# ZBX_HISTORYSTORAGETYPES=uint,dbl,str,log,text # Available since 3.4.5
ZBX_ALLOWUNSUPPORTEDDBVERSIONS=1 # Available since 6.0.0
# ZBX_DBTLSCONNECT=required # Available since 5.0.0
# ZBX_DBTLSCAFILE=/run/secrets/root-ca.pem # Available since 5.0.0
# ZBX_DBTLSCERTFILE=/run/secrets/client-cert.pem # Available since 5.0.0
# ZBX_DBTLSKEYFILE=/run/secrets/client-key.pem # Available since 5.0.0
# ZBX_DBTLSCIPHER= # Available since 5.0.0
# ZBX_DBTLSCIPHER13= # Available since 5.0.0
ZBX_AUTOHANODENAME=hostname # Allowed values: fqdn, hostname. Available since 6.0.0
# ZBX_HANODENAME= # Available since 6.0.0
# ZBX_AUTONODEADDRESS=fqdn # Allowed values: fqdn, hostname. Available since 6.0.0
# ZBX_NODEADDRESSPORT=10051 # Allowed to use with ZBX_AUTONODEADDRESS variable only. Available since 6.0.0
# ZBX_NODEADDRESS=localhost:10051 # Available since 6.0.0
# ZBX_DEBUGLEVEL=3
ZBX_STARTPOLLERS=600
# ZBX_IPMIPOLLERS=4
ZBX_STARTPREPROCESSORS=32 # Available since 3.4.0
ZBX_STARTPOLLERSUNREACHABLE=200
ZBX_STARTTRAPPERS=24
ZBX_STARTPINGERS=150
# ZBX_STARTDISCOVERERS=1
# ZBX_STARTHTTPPOLLERS=1
ZBX_STARTHISTORYPOLLERS=32 # Available since 5.4.0
# ZBX_STARTTIMERS=1
# ZBX_STARTESCALATORS=1
ZBX_STARTALERTERS=6 # Available since 3.4.0
ZBX_STARTLLDPROCESSORS=32
# ZBX_JAVAGATEWAY_ENABLE=true
# ZBX_JAVAGATEWAY=zabbix-java-gateway
# ZBX_JAVAGATEWAYPORT=10052
# ZBX_STARTJAVAPOLLERS=5
# ZBX_STARTVMWARECOLLECTORS=0
# ZBX_VMWAREFREQUENCY=60
# ZBX_VMWAREPERFFREQUENCY=60
# ZBX_VMWARECACHESIZE=8M
# ZBX_VMWARETIMEOUT=10
ZBX_ENABLE_SNMP_TRAPS=true
# ZBX_SOURCEIP=
# ZBX_HOUSEKEEPINGFREQUENCY=1
# ZBX_MAXHOUSEKEEPERDELETE=5000
# ZBX_PROBLEMHOUSEKEEPINGFREQUENCY=60
# ZBX_SENDERFREQUENCY=30
ZBX_CACHESIZE=8G
# ZBX_CACHEUPDATEFREQUENCY=60
ZBX_STARTDBSYNCERS=24
ZBX_HISTORYCACHESIZE=2G
ZBX_HISTORYINDEXCACHESIZE=2G
# ZBX_HISTORYSTORAGEDATEINDEX=0
ZBX_TRENDCACHESIZE=2G
ZBX_TRENDFUNCTIONCACHESIZE=2G
ZBX_VALUECACHESIZE=2G
ZBX_TIMEOUT=6
# ZBX_TRAPPERTIMEOUT=300
# ZBX_UNREACHABLEPERIOD=45
# ZBX_UNAVAILABLEDELAY=60
# ZBX_UNREACHABLEDELAY=15
ZBX_LOGSLOWQUERIES=7000
# ZBX_EXPORTFILESIZE=
# ZBX_STARTPROXYPOLLERS=1
# ZBX_PROXYCONFIGFREQUENCY=3600
# ZBX_PROXYDATAFREQUENCY=1
# ZBX_LOADMODULE="dummy1.so,dummy2.so,dummy10.so"
# ZBX_TLSCAFILE=
# ZBX_TLSCRLFILE=
# ZBX_TLSCERTFILE=
# ZBX_TLSKEYFILE=
# ZBX_VAULTDBPATH=
# ZBX_VAULTURL=https://127.0.0.1:8200
# VAULT_TOKEN=
# ZBX_STARTREPORTWRITERS=0
# ZBX_WEBSERVICEURL=http://zabbix-web-service:10053/report
# ZBX_SERVICEMANAGERSYNCFREQUENCY=60#
where are they come?!I am really confuse!
when you change ENV vars, you must recreate container.
@dotneft
I do that!I know about this!
docker compose down or docker rm -f zabbix-server-1
and then
docker compose up
also i dont have any volume!
you have the same env in compose file as well:
environment:
- ZBX_HANODENAME=zabbix-server-1
- ZBX_NODEADDRESS=172.31.63.1
@dotneft
As I said I removed in compose or comment them and also comment them into my .env!but still same!
show this:
docker compose -f docker-compose.yml config
also why you have this one:
ZBX_AUTOHANODENAME=hostname # Allowed values: fqdn, hostname. Available since 6.0.0
docker compose -f docker-compose.yml config
name: zabbix-server-mysql
services:
zabbix-agent-1:
container_name: zabbix-agent-1
environment:
ZBX_ACTIVE_ALLOW: "false"
ZBX_SERVER_HOST: 127.0.0.1,172.31.60.117
hostname: zbxSRV-1
image: zabbix/zabbix-agent:ubuntu-6.0.17
network_mode: host
pid: host
privileged: true
restart: unless-stopped
stop_grace_period: 5s
volumes:
- type: bind
source: /etc/localtime
target: /etc/localtime
read_only: true
bind:
create_host_path: true
- type: bind
source: /home/master/zabbix-server-mysql/zabbix_agentd.d
target: /etc/zabbix/zabbix_agentd.d
read_only: true
bind:
create_host_path: true
- type: bind
source: /home/master/zabbix-server-mysql/var/lib/zabbix
target: /var/lib/zabbix
read_only: true
bind:
create_host_path: true
zabbix-server-1:
container_name: zabbix-server-1
environment:
DB_SERVER_HOST: 172.31.60.117
DB_SERVER_PORT: "3306"
MYSQL_DATABASE: zabbix
MYSQL_PASSWORD: my_sql_pass
MYSQL_ROOT_PASSWORD: root_pass
MYSQL_USER: root
ZBX_ALLOWUNSUPPORTEDDBVERSIONS: "1"
ZBX_AUTOHANODENAME: hostname
ZBX_CACHESIZE: 8G
ZBX_DEBUGLEVEL: "4"
ZBX_ENABLE_SNMP_TRAPS: "true"
ZBX_HISTORYCACHESIZE: 2G
ZBX_HISTORYINDEXCACHESIZE: 2G
ZBX_LOGSLOWQUERIES: "7000"
ZBX_STARTALERTERS: "6"
ZBX_STARTDBSYNCERS: "24"
ZBX_STARTHISTORYPOLLERS: "32"
ZBX_STARTLLDPROCESSORS: "32"
ZBX_STARTPINGERS: "150"
ZBX_STARTPOLLERS: "600"
ZBX_STARTPOLLERSUNREACHABLE: "200"
ZBX_STARTPREPROCESSORS: "32"
ZBX_STARTTRAPPERS: "24"
ZBX_TIMEOUT: "6"
ZBX_TRENDCACHESIZE: 2G
ZBX_TRENDFUNCTIONCACHESIZE: 2G
ZBX_VALUECACHESIZE: 2G
hostname: zbxSRV-1
image: arvan/zabbix-server-mysql:ubuntu-6.0.17
network_mode: host
restart: unless-stopped
stop_grace_period: 30s
ulimits:
nofile:
soft: 20000
hard: 40000
nproc: 65535
volumes:
- type: bind
source: /etc/localtime
target: /etc/localtime
read_only: true
bind:
create_host_path: true
- type: bind
source: /var/lib/snmp/mibs
target: /var/lib/zabbix/mibs
read_only: true
bind:
create_host_path: true
- type: bind
source: /home/master/zabbix-server-mysql/usr/lib/zabbix/alertscripts
target: /usr/lib/zabbix/alertscripts
read_only: true
bind:
create_host_path: true
- type: bind
source: /home/master/zabbix-server-mysql/usr/lib/zabbix/externalscripts
target: /usr/lib/zabbix/externalscripts
read_only: true
bind:
create_host_path: true
it is completely different one what you show before. Here is 6.0, while in first message you had 6.4.
finally, remove or comment this one: "ZBX_AUTOHANODENAME=hostname # Allowed values: fqdn, hostname. Available since 6.0.0" and try again.
it is completely different one what you show before. Here is 6.0, while in first message you had 6.4.
Yes, you are right, I put the Docker image here differently, but the configuration is the same, and I only updated the images to 6.4(or newer version)
There is no difference in the configuration with the new version that I am trying to update
finally, remove or comment this one: "ZBX_AUTOHANODENAME=hostname # Allowed values: fqdn, hostname. Available since 6.0.0" and try again.
I will test this and share the results with you Thank you for your time and consideration.
finally, remove or comment this one: "ZBX_AUTOHANODENAME=hostname # Allowed values: fqdn, hostname. Available since 6.0.0" and try again.
Yes, it worked!thanks!