EnterpriseDB/repmgr

Upstream connection: CRITICAL witness is not attached to expected upstream node

mattbunter opened this issue · 0 comments

Repmgr 5.1
Postgres 12
CentOS 7.9.2009 (Core)

After two switchovers, node 1 to node 2 then node 2 to node 1, the witness is reporting an error :

Upstream connection: CRITICAL (node "pgwitness.themis.uk.azure" (ID: 577739013) is not attached to expected upstream node "pgsql1.themis.uk.azure" (ID: 577714693))

Tried forcing register and also unregister then register :

[postgres@pgwitness ~]$ repmgr -f /etc/repmgr/12/repmgr.conf witness unregister --dry-run
INFO: connecting to node "pgwitness.themis.uk.azure" (ID: 577739013)
INFO: prerequisites for unregistering the witness node are met
[postgres@pgwitness ~]$ repmgr -f /etc/repmgr/12/repmgr.conf witness unregister
INFO: connecting to node "pgwitness.themis.uk.azure" (ID: 577739013)
INFO: unregistering witness node 577739013
INFO: witness unregistration complete
DETAIL: witness node with ID 577739013 successfully unregistered
[postgres@pgwitness ~]$
[postgres@pgwitness ~]$
[postgres@pgwitness ~]$ repmgr -f /etc/repmgr/12/repmgr.conf witness register -h pgsql1.themis.uk.azure
INFO: connecting to witness node "pgwitness.themis.uk.azure" (ID: 577739013)
INFO: connecting to primary node
INFO: "repmgr" extension is already installed
ERROR: witness node is already initialised and contains node records
HINT: use option -F/--force to reinitialise the node
[postgres@pgwitness ~]$ repmgr -f /etc/repmgr/12/repmgr.conf witness register -F -h pgsql1.themis.uk.azure
INFO: connecting to witness node "pgwitness.themis.uk.azure" (ID: 577739013)
INFO: connecting to primary node
INFO: "repmgr" extension is already installed
INFO: witness registration complete
NOTICE: witness node "pgwitness.themis.uk.azure" (ID: 577739013) successfully registered
[postgres@pgwitness ~]$ node_check
Node "pgwitness.themis.uk.azure":
Server role: OK (node is witness)
Replication lag: OK (N/A - node is witness)
WAL archiving: OK (0 pending archive ready files)
Upstream connection: CRITICAL (node "pgwitness.themis.uk.azure" (ID: 577739013) is not attached to expected upstream node "pgsql1.themis.uk.azure" (ID: 577714693))
Downstream servers: OK (this node has no downstream nodes)
Replication slots: OK (node has no physical replication slots)
Missing physical replication slots: OK (node has no missing physical replication slots)
Configured data directory: OK (configured "data_directory" is "/data/12")
[postgres@pgwitness ~]$ node_status
Node "pgwitness.themis.uk.azure":
PostgreSQL version: 12.2
Total data size: 32 MB
Conninfo: host=10.94.5.99 user=repmgr dbname=repmgr
Role: witness
WAL archiving: enabled
Archive command: /bin/true
WALs pending archiving: 0 pending files
Replication connections: 0 (of maximal 10)
Replication slots: 0 physical (of maximal 10; 0 missing)
Replication lag: n/a

Not sure what to do now. Would appreciate advice. Unable to upgrade repmgr at this time.