repmgr database "forgets" the repmgrd PID during an auto-failover sequence on a surviving standby (repmgr 4.4)
Opened this issue · 0 comments
ckolovson commented
We have come across an odd problem with repmgr (we’re using 4.4) that we wanted to bring to your attention. What we noticed is that after an automatic failover by repmgrd (promotion of a standby), for a short period of time, the “repmgr daemon status” command is falsely reporting that the repmgrd daemon is not running, when in fact it was running the whole time.
In our use case, we have a user-defined script for the “repmgr standby follow” command, because in that script we not only call “repmgr standby follow” but also a script to change the new primary database IP address on the clients that are accessing the database. Therefore, our custom “follow script” takes about 20 sec. During that 20 sec, SELECT repmgr.get_repmgrd_pid() returns -1 for the repmgrd PID! Not sure why this happens. Note that we are not using a PID file for repmgrd because we use systemd to control it, and systemd doesn’t work with PID files. We also see in the repmgr log that repmgrd automatically restarts postgres on the surviving standby that is following the newly promoted primary.
To summarize, what we’re seeing appears to be due to:
- On an auto-failover, repmgrd restarts postgres on the surviving standby and the repmgr metadata in the postgres DB forgets the PID of repmgrd for some period of time (until the promotion and standby follow commands have completed).
- Repmgrd can rediscover the PID but only after it finishes executing our custom follow script, which takes approx. 20 seconds.
See the sequence of events below.
Is there something you can do to make it so that repmgrd does not forget its PID in the database while an auto-failover sequence is happening?
On a related topic, we have also noticed that sometimes on the newly promoted primary, for a short period of time it transitions from:
Role Status
----- ---------
standby | running
To…
standby | ! running as primary
To…
primary | * running
Why does this intermediate state exist where it shows up as “standby | ! running as primary”, which we usually see on an old primary that has come back up in a cluster where a new primary has already been promoted?
Thanks in advance.
root@bos1-vcloud-static-161-49 [ ~ ]# su - postgres -c "date; repmgr cluster show; repmgr daemon status; echo 'PID of repmgrd:'; psql -d repmgr -A -t -c 'SELECT repmgr.get_repmgrd_pid();' "
Tue Mar 3 23:20:16 UTC 2020
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
-------+---------------------------+---------+-----------+---------------------------+----------+----------+----------+----------------------------------------------------------------
5556 | bos1-vcloud-static-161-48 | standby | running | bos1-vcloud-static-161-47 | default | 100 | 5 | host=172.18.36.190 user=repmgr dbname=repmgr connect_timeout=2
18346 | bos1-vcloud-static-161-47 | primary | * running | | default | 100 | 5 | host=172.18.36.189 user=repmgr dbname=repmgr connect_timeout=2
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-47 | default | 100 | 5 | host=172.18.36.191 user=repmgr dbname=repmgr connect_timeout=2
ID | Name | Role | Status | Upstream | repmgrd | PID | Paused? | Upstream last seen
----+---------------------------+---------+-----------+---------------------------+---------+-------+---------+--------------------
5556 | bos1-vcloud-static-161-48 | standby | running | bos1-vcloud-static-161-47 | running | 14380 | no | 0 second(s) ago
18346 | bos1-vcloud-static-161-47 | primary | * running | | running | 733 | no | n/a
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-47 | running | 12024 | no | 1 second(s) ago
PID of repmgrd:
12024
root@bos1-vcloud-static-161-49 [ ~ ]# su - postgres -c "date; repmgr cluster show; repmgr daemon status; echo 'PID of repmgrd:'; psql -d repmgr -A -t -c 'SELECT repmgr.get_repmgrd_pid();' "
Tue Mar 3 23:20:33 UTC 2020
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
-------+---------------------------+---------+---------------+-----------------------------+----------+----------+----------+----------------------------------------------------------------
5556 | bos1-vcloud-static-161-48 | standby | running | ? bos1-vcloud-static-161-47 | default | 100 | 5 | host=172.18.36.190 user=repmgr dbname=repmgr connect_timeout=2
18346 | bos1-vcloud-static-161-47 | primary | ? unreachable | | default | 100 | ? | host=172.18.36.189 user=repmgr dbname=repmgr connect_timeout=2
22882 | bos1-vcloud-static-161-49 | standby | running | ? bos1-vcloud-static-161-47 | default | 100 | 5 | host=172.18.36.191 user=repmgr dbname=repmgr connect_timeout=2
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-48" (ID: 5556)'s upstream node "bos1-vcloud-static-161-47" (ID: 18346)
- unable to determine if node "bos1-vcloud-static-161-48" (ID: 5556) is attached to its upstream node "bos1-vcloud-static-161-47" (ID: 18346)
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
- node "bos1-vcloud-static-161-47" (ID: 18346) is registered as an active primary but is unreachable
- unable to connect to node "bos1-vcloud-static-161-49" (ID: 22882)'s upstream node "bos1-vcloud-static-161-47" (ID: 18346)
- unable to determine if node "bos1-vcloud-static-161-49" (ID: 22882) is attached to its upstream node "bos1-vcloud-static-161-47" (ID: 18346)
ID | Name | Role | Status | Upstream | repmgrd | PID | Paused? | Upstream last seen
----+---------------------------+---------+---------------+-----------------------------+---------+-------+---------+--------------------
5556 | bos1-vcloud-static-161-48 | standby | running | ? bos1-vcloud-static-161-47 | running | 14380 | no | 3 second(s) ago
18346 | bos1-vcloud-static-161-47 | primary | ? unreachable | | n/a | n/a | n/a | n/a
22882 | bos1-vcloud-static-161-49 | standby | running | ? bos1-vcloud-static-161-47 | running | 12024 | no | 4 second(s) ago
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-48" (ID: 5556)'s upstream node "bos1-vcloud-static-161-47" (ID: 18346)
- unable to determine if node "bos1-vcloud-static-161-48" (ID: 5556) is attached to its upstream node "bos1-vcloud-static-161-47" (ID: 18346)
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
- node "bos1-vcloud-static-161-47" (ID: 18346) is registered as an active primary but is unreachable
- unable to connect to node "bos1-vcloud-static-161-49" (ID: 22882)'s upstream node "bos1-vcloud-static-161-47" (ID: 18346)
- unable to determine if node "bos1-vcloud-static-161-49" (ID: 22882) is attached to its upstream node "bos1-vcloud-static-161-47" (ID: 18346)
PID of repmgrd:
12024
root@bos1-vcloud-static-161-49 [ ~ ]# su - postgres -c "date; repmgr cluster show; repmgr daemon status; echo 'PID of repmgrd:'; psql -d repmgr -A -t -c 'SELECT repmgr.get_repmgrd_pid();' "
Tue Mar 3 23:20:34 UTC 2020
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
-------+---------------------------+---------+---------------+-----------------------------+----------+----------+----------+----------------------------------------------------------------
5556 | bos1-vcloud-static-161-48 | standby | running | ? bos1-vcloud-static-161-47 | default | 100 | 5 | host=172.18.36.190 user=repmgr dbname=repmgr connect_timeout=2
18346 | bos1-vcloud-static-161-47 | primary | ? unreachable | | default | 100 | ? | host=172.18.36.189 user=repmgr dbname=repmgr connect_timeout=2
22882 | bos1-vcloud-static-161-49 | standby | running | ? bos1-vcloud-static-161-47 | default | 100 | 5 | host=172.18.36.191 user=repmgr dbname=repmgr connect_timeout=2
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-48" (ID: 5556)'s upstream node "bos1-vcloud-static-161-47" (ID: 18346)
- unable to determine if node "bos1-vcloud-static-161-48" (ID: 5556) is attached to its upstream node "bos1-vcloud-static-161-47" (ID: 18346)
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
- node "bos1-vcloud-static-161-47" (ID: 18346) is registered as an active primary but is unreachable
- unable to connect to node "bos1-vcloud-static-161-49" (ID: 22882)'s upstream node "bos1-vcloud-static-161-47" (ID: 18346)
- unable to determine if node "bos1-vcloud-static-161-49" (ID: 22882) is attached to its upstream node "bos1-vcloud-static-161-47" (ID: 18346)
ID | Name | Role | Status | Upstream | repmgrd | PID | Paused? | Upstream last seen
----+---------------------------+---------+---------------+-----------------------------+---------+-------+---------+--------------------
5556 | bos1-vcloud-static-161-48 | standby | running | ? bos1-vcloud-static-161-47 | running | 14380 | no | 4 second(s) ago
18346 | bos1-vcloud-static-161-47 | primary | ? unreachable | | n/a | n/a | n/a | n/a
22882 | bos1-vcloud-static-161-49 | standby | running | ? bos1-vcloud-static-161-47 | running | 12024 | no | 5 second(s) ago
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-48" (ID: 5556)'s upstream node "bos1-vcloud-static-161-47" (ID: 18346)
- unable to determine if node "bos1-vcloud-static-161-48" (ID: 5556) is attached to its upstream node "bos1-vcloud-static-161-47" (ID: 18346)
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
- node "bos1-vcloud-static-161-47" (ID: 18346) is registered as an active primary but is unreachable
- unable to connect to node "bos1-vcloud-static-161-49" (ID: 22882)'s upstream node "bos1-vcloud-static-161-47" (ID: 18346)
- unable to determine if node "bos1-vcloud-static-161-49" (ID: 22882) is attached to its upstream node "bos1-vcloud-static-161-47" (ID: 18346)
PID of repmgrd:
12024
root@bos1-vcloud-static-161-49 [ ~ ]# su - postgres -c "date; repmgr cluster show; repmgr daemon status; echo 'PID of repmgrd:'; psql -d repmgr -A -t -c 'SELECT repmgr.get_repmgrd_pid();' "
Tue Mar 3 23:20:35 UTC 2020
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
-------+---------------------------+---------+---------------+-----------------------------+----------+----------+----------+----------------------------------------------------------------
5556 | bos1-vcloud-static-161-48 | standby | running | ? bos1-vcloud-static-161-47 | default | 100 | 5 | host=172.18.36.190 user=repmgr dbname=repmgr connect_timeout=2
18346 | bos1-vcloud-static-161-47 | primary | ? unreachable | | default | 100 | ? | host=172.18.36.189 user=repmgr dbname=repmgr connect_timeout=2
22882 | bos1-vcloud-static-161-49 | standby | running | ? bos1-vcloud-static-161-47 | default | 100 | 5 | host=172.18.36.191 user=repmgr dbname=repmgr connect_timeout=2
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-48" (ID: 5556)'s upstream node "bos1-vcloud-static-161-47" (ID: 18346)
- unable to determine if node "bos1-vcloud-static-161-48" (ID: 5556) is attached to its upstream node "bos1-vcloud-static-161-47" (ID: 18346)
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
- node "bos1-vcloud-static-161-47" (ID: 18346) is registered as an active primary but is unreachable
- unable to connect to node "bos1-vcloud-static-161-49" (ID: 22882)'s upstream node "bos1-vcloud-static-161-47" (ID: 18346)
- unable to determine if node "bos1-vcloud-static-161-49" (ID: 22882) is attached to its upstream node "bos1-vcloud-static-161-47" (ID: 18346)
ID | Name | Role | Status | Upstream | repmgrd | PID | Paused? | Upstream last seen
----+---------------------------+---------+---------------+-----------------------------+---------+-------+---------+--------------------
5556 | bos1-vcloud-static-161-48 | standby | running | ? bos1-vcloud-static-161-47 | running | 14380 | no | 5 second(s) ago
18346 | bos1-vcloud-static-161-47 | primary | ? unreachable | | n/a | n/a | n/a | n/a
22882 | bos1-vcloud-static-161-49 | standby | running | ? bos1-vcloud-static-161-47 | running | 12024 | no | 6 second(s) ago
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-48" (ID: 5556)'s upstream node "bos1-vcloud-static-161-47" (ID: 18346)
- unable to determine if node "bos1-vcloud-static-161-48" (ID: 5556) is attached to its upstream node "bos1-vcloud-static-161-47" (ID: 18346)
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
- node "bos1-vcloud-static-161-47" (ID: 18346) is registered as an active primary but is unreachable
- unable to connect to node "bos1-vcloud-static-161-49" (ID: 22882)'s upstream node "bos1-vcloud-static-161-47" (ID: 18346)
- unable to determine if node "bos1-vcloud-static-161-49" (ID: 22882) is attached to its upstream node "bos1-vcloud-static-161-47" (ID: 18346)
PID of repmgrd:
12024
root@bos1-vcloud-static-161-49 [ ~ ]# su - postgres -c "date; repmgr cluster show; repmgr daemon status; echo 'PID of repmgrd:'; psql -d repmgr -A -t -c 'SELECT repmgr.get_repmgrd_pid();' "
Tue Mar 3 23:20:37 UTC 2020
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
-------+---------------------------+---------+---------------+-----------------------------+----------+----------+----------+----------------------------------------------------------------
5556 | bos1-vcloud-static-161-48 | standby | running | ? bos1-vcloud-static-161-47 | default | 100 | 5 | host=172.18.36.190 user=repmgr dbname=repmgr connect_timeout=2
18346 | bos1-vcloud-static-161-47 | primary | ? unreachable | | default | 100 | ? | host=172.18.36.189 user=repmgr dbname=repmgr connect_timeout=2
22882 | bos1-vcloud-static-161-49 | standby | running | ? bos1-vcloud-static-161-47 | default | 100 | 5 | host=172.18.36.191 user=repmgr dbname=repmgr connect_timeout=2
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-48" (ID: 5556)'s upstream node "bos1-vcloud-static-161-47" (ID: 18346)
- unable to determine if node "bos1-vcloud-static-161-48" (ID: 5556) is attached to its upstream node "bos1-vcloud-static-161-47" (ID: 18346)
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
- node "bos1-vcloud-static-161-47" (ID: 18346) is registered as an active primary but is unreachable
- unable to connect to node "bos1-vcloud-static-161-49" (ID: 22882)'s upstream node "bos1-vcloud-static-161-47" (ID: 18346)
- unable to determine if node "bos1-vcloud-static-161-49" (ID: 22882) is attached to its upstream node "bos1-vcloud-static-161-47" (ID: 18346)
ID | Name | Role | Status | Upstream | repmgrd | PID | Paused? | Upstream last seen
----+---------------------------+---------+---------------+-----------------------------+---------+-------+---------+--------------------
5556 | bos1-vcloud-static-161-48 | standby | running | ? bos1-vcloud-static-161-47 | running | 14380 | no | 7 second(s) ago
18346 | bos1-vcloud-static-161-47 | primary | ? unreachable | | n/a | n/a | n/a | n/a
22882 | bos1-vcloud-static-161-49 | standby | running | ? bos1-vcloud-static-161-47 | running | 12024 | no | 7 second(s) ago
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-48" (ID: 5556)'s upstream node "bos1-vcloud-static-161-47" (ID: 18346)
- unable to determine if node "bos1-vcloud-static-161-48" (ID: 5556) is attached to its upstream node "bos1-vcloud-static-161-47" (ID: 18346)
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
- node "bos1-vcloud-static-161-47" (ID: 18346) is registered as an active primary but is unreachable
- unable to connect to node "bos1-vcloud-static-161-49" (ID: 22882)'s upstream node "bos1-vcloud-static-161-47" (ID: 18346)
- unable to determine if node "bos1-vcloud-static-161-49" (ID: 22882) is attached to its upstream node "bos1-vcloud-static-161-47" (ID: 18346)
PID of repmgrd:
12024
root@bos1-vcloud-static-161-49 [ ~ ]# su - postgres -c "date; repmgr cluster show; repmgr daemon status; echo 'PID of repmgrd:'; psql -d repmgr -A -t -c 'SELECT repmgr.get_repmgrd_pid();' "
Tue Mar 3 23:20:38 UTC 2020
ERROR: connection to database failed
DETAIL:
SSL connection has been closed unexpectedly
DETAIL: attempted to connect using:
user=repmgr connect_timeout=2 dbname=repmgr host=172.18.36.191 fallback_application_name=repmgr
ERROR: connection to database failed
DETAIL:
FATAL: the database system is shutting down
FATAL: the database system is shutting down
DETAIL: attempted to connect using:
user=repmgr connect_timeout=2 dbname=repmgr host=172.18.36.191 fallback_application_name=repmgr
PID of repmgrd:
psql.bin: FATAL: the database system is shutting down
root@bos1-vcloud-static-161-49 [ ~ ]# su - postgres -c "date; repmgr cluster show; repmgr daemon status; echo 'PID of repmgrd:'; psql -d repmgr -A -t -c 'SELECT repmgr.get_repmgrd_pid();' "
Tue Mar 3 23:20:39 UTC 2020
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
-------+---------------------------+---------+-----------+---------------------------+----------+----------+----------+----------------------------------------------------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | default | 100 | 5 | host=172.18.36.190 user=repmgr dbname=repmgr connect_timeout=2
18346 | bos1-vcloud-static-161-47 | primary | - failed | | default | 100 | ? | host=172.18.36.189 user=repmgr dbname=repmgr connect_timeout=2
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | default | 100 | 5 | host=172.18.36.191 user=repmgr dbname=repmgr connect_timeout=2
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
ID | Name | Role | Status | Upstream | repmgrd | PID | Paused? | Upstream last seen
----+---------------------------+---------+-----------+---------------------------+-------------+-------+---------+--------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | running | 14380 | no | n/a
18346 | bos1-vcloud-static-161-47 | primary | - failed | | n/a | n/a | n/a | n/a
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | not running | n/a | n/a | n/a
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
PID of repmgrd:
-1
root@bos1-vcloud-static-161-49 [ ~ ]# su - postgres -c "date; repmgr cluster show; repmgr daemon status; echo 'PID of repmgrd:'; psql -d repmgr -A -t -c 'SELECT repmgr.get_repmgrd_pid();' "
Tue Mar 3 23:20:40 UTC 2020
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
-------+---------------------------+---------+-----------+---------------------------+----------+----------+----------+----------------------------------------------------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | default | 100 | 5 | host=172.18.36.190 user=repmgr dbname=repmgr connect_timeout=2
18346 | bos1-vcloud-static-161-47 | primary | - failed | | default | 100 | ? | host=172.18.36.189 user=repmgr dbname=repmgr connect_timeout=2
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | default | 100 | 5 | host=172.18.36.191 user=repmgr dbname=repmgr connect_timeout=2
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
ID | Name | Role | Status | Upstream | repmgrd | PID | Paused? | Upstream last seen
----+---------------------------+---------+-----------+---------------------------+-------------+-------+---------+--------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | running | 14380 | no | n/a
18346 | bos1-vcloud-static-161-47 | primary | - failed | | n/a | n/a | n/a | n/a
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | not running | n/a | n/a | n/a
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
PID of repmgrd:
-1
root@bos1-vcloud-static-161-49 [ ~ ]# su - postgres -c "date; repmgr cluster show; repmgr daemon status; echo 'PID of repmgrd:'; psql -d repmgr -A -t -c 'SELECT repmgr.get_repmgrd_pid();' "
Tue Mar 3 23:20:42 UTC 2020
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
-------+---------------------------+---------+-----------+---------------------------+----------+----------+----------+----------------------------------------------------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | default | 100 | 5 | host=172.18.36.190 user=repmgr dbname=repmgr connect_timeout=2
18346 | bos1-vcloud-static-161-47 | primary | - failed | | default | 100 | ? | host=172.18.36.189 user=repmgr dbname=repmgr connect_timeout=2
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | default | 100 | 5 | host=172.18.36.191 user=repmgr dbname=repmgr connect_timeout=2
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
ID | Name | Role | Status | Upstream | repmgrd | PID | Paused? | Upstream last seen
----+---------------------------+---------+-----------+---------------------------+-------------+-------+---------+--------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | running | 14380 | no | n/a
18346 | bos1-vcloud-static-161-47 | primary | - failed | | n/a | n/a | n/a | n/a
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | not running | n/a | n/a | n/a
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
PID of repmgrd:
-1
root@bos1-vcloud-static-161-49 [ ~ ]# su - postgres -c "date; repmgr cluster show; repmgr daemon status; echo 'PID of repmgrd:'; psql -d repmgr -A -t -c 'SELECT repmgr.get_repmgrd_pid();' "
Tue Mar 3 23:20:43 UTC 2020
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
-------+---------------------------+---------+-----------+---------------------------+----------+----------+----------+----------------------------------------------------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | default | 100 | 5 | host=172.18.36.190 user=repmgr dbname=repmgr connect_timeout=2
18346 | bos1-vcloud-static-161-47 | primary | - failed | | default | 100 | ? | host=172.18.36.189 user=repmgr dbname=repmgr connect_timeout=2
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | default | 100 | 5 | host=172.18.36.191 user=repmgr dbname=repmgr connect_timeout=2
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
ID | Name | Role | Status | Upstream | repmgrd | PID | Paused? | Upstream last seen
----+---------------------------+---------+-----------+---------------------------+-------------+-------+---------+--------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | running | 14380 | no | n/a
18346 | bos1-vcloud-static-161-47 | primary | - failed | | n/a | n/a | n/a | n/a
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | not running | n/a | n/a | n/a
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
PID of repmgrd:
-1
root@bos1-vcloud-static-161-49 [ ~ ]# su - postgres -c "date; repmgr cluster show; repmgr daemon status; echo 'PID of repmgrd:'; psql -d repmgr -A -t -c 'SELECT repmgr.get_repmgrd_pid();' "
Tue Mar 3 23:20:44 UTC 2020
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
-------+---------------------------+---------+-----------+---------------------------+----------+----------+----------+----------------------------------------------------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | default | 100 | 5 | host=172.18.36.190 user=repmgr dbname=repmgr connect_timeout=2
18346 | bos1-vcloud-static-161-47 | primary | - failed | | default | 100 | ? | host=172.18.36.189 user=repmgr dbname=repmgr connect_timeout=2
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | default | 100 | 5 | host=172.18.36.191 user=repmgr dbname=repmgr connect_timeout=2
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
ID | Name | Role | Status | Upstream | repmgrd | PID | Paused? | Upstream last seen
----+---------------------------+---------+-----------+---------------------------+-------------+-------+---------+--------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | running | 14380 | no | n/a
18346 | bos1-vcloud-static-161-47 | primary | - failed | | n/a | n/a | n/a | n/a
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | not running | n/a | n/a | n/a
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
PID of repmgrd:
-1
root@bos1-vcloud-static-161-49 [ ~ ]# su - postgres -c "date; repmgr cluster show; repmgr daemon status; echo 'PID of repmgrd:'; psql -d repmgr -A -t -c 'SELECT repmgr.get_repmgrd_pid();' "
Tue Mar 3 23:20:45 UTC 2020
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
-------+---------------------------+---------+-----------+---------------------------+----------+----------+----------+----------------------------------------------------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | default | 100 | 5 | host=172.18.36.190 user=repmgr dbname=repmgr connect_timeout=2
18346 | bos1-vcloud-static-161-47 | primary | - failed | | default | 100 | ? | host=172.18.36.189 user=repmgr dbname=repmgr connect_timeout=2
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | default | 100 | 5 | host=172.18.36.191 user=repmgr dbname=repmgr connect_timeout=2
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
ID | Name | Role | Status | Upstream | repmgrd | PID | Paused? | Upstream last seen
----+---------------------------+---------+-----------+---------------------------+-------------+-------+---------+--------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | running | 14380 | no | n/a
18346 | bos1-vcloud-static-161-47 | primary | - failed | | n/a | n/a | n/a | n/a
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | not running | n/a | n/a | n/a
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
PID of repmgrd:
-1
root@bos1-vcloud-static-161-49 [ ~ ]# su - postgres -c "date; repmgr cluster show; repmgr daemon status; echo 'PID of repmgrd:'; psql -d repmgr -A -t -c 'SELECT repmgr.get_repmgrd_pid();' "
Tue Mar 3 23:20:47 UTC 2020
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
-------+---------------------------+---------+-----------+---------------------------+----------+----------+----------+----------------------------------------------------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | default | 100 | 5 | host=172.18.36.190 user=repmgr dbname=repmgr connect_timeout=2
18346 | bos1-vcloud-static-161-47 | primary | - failed | | default | 100 | ? | host=172.18.36.189 user=repmgr dbname=repmgr connect_timeout=2
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | default | 100 | 5 | host=172.18.36.191 user=repmgr dbname=repmgr connect_timeout=2
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
ID | Name | Role | Status | Upstream | repmgrd | PID | Paused? | Upstream last seen
----+---------------------------+---------+-----------+---------------------------+-------------+-------+---------+--------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | running | 14380 | no | n/a
18346 | bos1-vcloud-static-161-47 | primary | - failed | | n/a | n/a | n/a | n/a
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | not running | n/a | n/a | n/a
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
PID of repmgrd:
-1
root@bos1-vcloud-static-161-49 [ ~ ]# su - postgres -c "date; repmgr cluster show; repmgr daemon status; echo 'PID of repmgrd:'; psql -d repmgr -A -t -c 'SELECT repmgr.get_repmgrd_pid();' "
Tue Mar 3 23:20:48 UTC 2020
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
-------+---------------------------+---------+-----------+---------------------------+----------+----------+----------+----------------------------------------------------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | default | 100 | 5 | host=172.18.36.190 user=repmgr dbname=repmgr connect_timeout=2
18346 | bos1-vcloud-static-161-47 | primary | - failed | | default | 100 | ? | host=172.18.36.189 user=repmgr dbname=repmgr connect_timeout=2
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | default | 100 | 5 | host=172.18.36.191 user=repmgr dbname=repmgr connect_timeout=2
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
ID | Name | Role | Status | Upstream | repmgrd | PID | Paused? | Upstream last seen
----+---------------------------+---------+-----------+---------------------------+-------------+-------+---------+--------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | running | 14380 | no | n/a
18346 | bos1-vcloud-static-161-47 | primary | - failed | | n/a | n/a | n/a | n/a
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | not running | n/a | n/a | n/a
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
PID of repmgrd:
-1
root@bos1-vcloud-static-161-49 [ ~ ]# su - postgres -c "date; repmgr cluster show; repmgr daemon status; echo 'PID of repmgrd:'; psql -d repmgr -A -t -c 'SELECT repmgr.get_repmgrd_pid();' "
Tue Mar 3 23:20:49 UTC 2020
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
-------+---------------------------+---------+-----------+---------------------------+----------+----------+----------+----------------------------------------------------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | default | 100 | 5 | host=172.18.36.190 user=repmgr dbname=repmgr connect_timeout=2
18346 | bos1-vcloud-static-161-47 | primary | - failed | | default | 100 | ? | host=172.18.36.189 user=repmgr dbname=repmgr connect_timeout=2
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | default | 100 | 5 | host=172.18.36.191 user=repmgr dbname=repmgr connect_timeout=2
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
ID | Name | Role | Status | Upstream | repmgrd | PID | Paused? | Upstream last seen
----+---------------------------+---------+-----------+---------------------------+-------------+-------+---------+--------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | running | 14380 | no | n/a
18346 | bos1-vcloud-static-161-47 | primary | - failed | | n/a | n/a | n/a | n/a
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | not running | n/a | n/a | n/a
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
PID of repmgrd:
-1
root@bos1-vcloud-static-161-49 [ ~ ]# su - postgres -c "date; repmgr cluster show; repmgr daemon status; echo 'PID of repmgrd:'; psql -d repmgr -A -t -c 'SELECT repmgr.get_repmgrd_pid();' "
Tue Mar 3 23:20:50 UTC 2020
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
-------+---------------------------+---------+-----------+---------------------------+----------+----------+----------+----------------------------------------------------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | default | 100 | 5 | host=172.18.36.190 user=repmgr dbname=repmgr connect_timeout=2
18346 | bos1-vcloud-static-161-47 | primary | - failed | | default | 100 | ? | host=172.18.36.189 user=repmgr dbname=repmgr connect_timeout=2
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | default | 100 | 5 | host=172.18.36.191 user=repmgr dbname=repmgr connect_timeout=2
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
ID | Name | Role | Status | Upstream | repmgrd | PID | Paused? | Upstream last seen
----+---------------------------+---------+-----------+---------------------------+-------------+-------+---------+--------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | running | 14380 | no | n/a
18346 | bos1-vcloud-static-161-47 | primary | - failed | | n/a | n/a | n/a | n/a
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | not running | n/a | n/a | n/a
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
PID of repmgrd:
-1
root@bos1-vcloud-static-161-49 [ ~ ]# su - postgres -c "date; repmgr cluster show; repmgr daemon status; echo 'PID of repmgrd:'; psql -d repmgr -A -t -c 'SELECT repmgr.get_repmgrd_pid();' "
Tue Mar 3 23:20:51 UTC 2020
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
-------+---------------------------+---------+-----------+---------------------------+----------+----------+----------+----------------------------------------------------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | default | 100 | 5 | host=172.18.36.190 user=repmgr dbname=repmgr connect_timeout=2
18346 | bos1-vcloud-static-161-47 | primary | - failed | | default | 100 | ? | host=172.18.36.189 user=repmgr dbname=repmgr connect_timeout=2
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | default | 100 | 5 | host=172.18.36.191 user=repmgr dbname=repmgr connect_timeout=2
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
ID | Name | Role | Status | Upstream | repmgrd | PID | Paused? | Upstream last seen
----+---------------------------+---------+-----------+---------------------------+-------------+-------+---------+--------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | running | 14380 | no | n/a
18346 | bos1-vcloud-static-161-47 | primary | - failed | | n/a | n/a | n/a | n/a
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | not running | n/a | n/a | n/a
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
PID of repmgrd:
-1
root@bos1-vcloud-static-161-49 [ ~ ]# su - postgres -c "date; repmgr cluster show; repmgr daemon status; echo 'PID of repmgrd:'; psql -d repmgr -A -t -c 'SELECT repmgr.get_repmgrd_pid();' "
Tue Mar 3 23:20:52 UTC 2020
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
-------+---------------------------+---------+-----------+---------------------------+----------+----------+----------+----------------------------------------------------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | default | 100 | 5 | host=172.18.36.190 user=repmgr dbname=repmgr connect_timeout=2
18346 | bos1-vcloud-static-161-47 | primary | - failed | | default | 100 | ? | host=172.18.36.189 user=repmgr dbname=repmgr connect_timeout=2
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | default | 100 | 5 | host=172.18.36.191 user=repmgr dbname=repmgr connect_timeout=2
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
ID | Name | Role | Status | Upstream | repmgrd | PID | Paused? | Upstream last seen
----+---------------------------+---------+-----------+---------------------------+-------------+-------+---------+--------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | running | 14380 | no | n/a
18346 | bos1-vcloud-static-161-47 | primary | - failed | | n/a | n/a | n/a | n/a
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | not running | n/a | n/a | n/a
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
PID of repmgrd:
-1
root@bos1-vcloud-static-161-49 [ ~ ]# su - postgres -c "date; repmgr cluster show; repmgr daemon status; echo 'PID of repmgrd:'; psql -d repmgr -A -t -c 'SELECT repmgr.get_repmgrd_pid();' "
Tue Mar 3 23:20:53 UTC 2020
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
-------+---------------------------+---------+-----------+---------------------------+----------+----------+----------+----------------------------------------------------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | default | 100 | 5 | host=172.18.36.190 user=repmgr dbname=repmgr connect_timeout=2
18346 | bos1-vcloud-static-161-47 | primary | - failed | | default | 100 | ? | host=172.18.36.189 user=repmgr dbname=repmgr connect_timeout=2
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | default | 100 | 5 | host=172.18.36.191 user=repmgr dbname=repmgr connect_timeout=2
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
ID | Name | Role | Status | Upstream | repmgrd | PID | Paused? | Upstream last seen
----+---------------------------+---------+-----------+---------------------------+-------------+-------+---------+--------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | running | 14380 | no | n/a
18346 | bos1-vcloud-static-161-47 | primary | - failed | | n/a | n/a | n/a | n/a
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | not running | n/a | n/a | n/a
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
PID of repmgrd:
-1
root@bos1-vcloud-static-161-49 [ ~ ]# su - postgres -c "date; repmgr cluster show; repmgr daemon status; echo 'PID of repmgrd:'; psql -d repmgr -A -t -c 'SELECT repmgr.get_repmgrd_pid();' "
Tue Mar 3 23:20:54 UTC 2020
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
-------+---------------------------+---------+-----------+---------------------------+----------+----------+----------+----------------------------------------------------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | default | 100 | 5 | host=172.18.36.190 user=repmgr dbname=repmgr connect_timeout=2
18346 | bos1-vcloud-static-161-47 | primary | - failed | | default | 100 | ? | host=172.18.36.189 user=repmgr dbname=repmgr connect_timeout=2
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | default | 100 | 5 | host=172.18.36.191 user=repmgr dbname=repmgr connect_timeout=2
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
ID | Name | Role | Status | Upstream | repmgrd | PID | Paused? | Upstream last seen
----+---------------------------+---------+-----------+---------------------------+-------------+-------+---------+--------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | running | 14380 | no | n/a
18346 | bos1-vcloud-static-161-47 | primary | - failed | | n/a | n/a | n/a | n/a
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | not running | n/a | n/a | n/a
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
PID of repmgrd:
-1
root@bos1-vcloud-static-161-49 [ ~ ]# su - postgres -c "date; repmgr cluster show; repmgr daemon status; echo 'PID of repmgrd:'; psql -d repmgr -A -t -c 'SELECT repmgr.get_repmgrd_pid();' "
Tue Mar 3 23:20:55 UTC 2020
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
-------+---------------------------+---------+-----------+---------------------------+----------+----------+----------+----------------------------------------------------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | default | 100 | 5 | host=172.18.36.190 user=repmgr dbname=repmgr connect_timeout=2
18346 | bos1-vcloud-static-161-47 | primary | - failed | | default | 100 | ? | host=172.18.36.189 user=repmgr dbname=repmgr connect_timeout=2
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | default | 100 | 5 | host=172.18.36.191 user=repmgr dbname=repmgr connect_timeout=2
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
ID | Name | Role | Status | Upstream | repmgrd | PID | Paused? | Upstream last seen
----+---------------------------+---------+-----------+---------------------------+-------------+-------+---------+--------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | running | 14380 | no | n/a
18346 | bos1-vcloud-static-161-47 | primary | - failed | | n/a | n/a | n/a | n/a
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | not running | n/a | n/a | n/a
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
PID of repmgrd:
-1
root@bos1-vcloud-static-161-49 [ ~ ]# su - postgres -c "date; repmgr cluster show; repmgr daemon status; echo 'PID of repmgrd:'; psql -d repmgr -A -t -c 'SELECT repmgr.get_repmgrd_pid();' "
Tue Mar 3 23:20:56 UTC 2020
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
-------+---------------------------+---------+-----------+---------------------------+----------+----------+----------+----------------------------------------------------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | default | 100 | 5 | host=172.18.36.190 user=repmgr dbname=repmgr connect_timeout=2
18346 | bos1-vcloud-static-161-47 | primary | - failed | | default | 100 | ? | host=172.18.36.189 user=repmgr dbname=repmgr connect_timeout=2
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | default | 100 | 5 | host=172.18.36.191 user=repmgr dbname=repmgr connect_timeout=2
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
ID | Name | Role | Status | Upstream | repmgrd | PID | Paused? | Upstream last seen
----+---------------------------+---------+-----------+---------------------------+-------------+-------+---------+--------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | running | 14380 | no | n/a
18346 | bos1-vcloud-static-161-47 | primary | - failed | | n/a | n/a | n/a | n/a
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | not running | n/a | n/a | n/a
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
PID of repmgrd:
-1
root@bos1-vcloud-static-161-49 [ ~ ]# su - postgres -c "date; repmgr cluster show; repmgr daemon status; echo 'PID of repmgrd:'; psql -d repmgr -A -t -c 'SELECT repmgr.get_repmgrd_pid();' "
Tue Mar 3 23:20:57 UTC 2020
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
-------+---------------------------+---------+-----------+---------------------------+----------+----------+----------+----------------------------------------------------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | default | 100 | 5 | host=172.18.36.190 user=repmgr dbname=repmgr connect_timeout=2
18346 | bos1-vcloud-static-161-47 | primary | - failed | | default | 100 | ? | host=172.18.36.189 user=repmgr dbname=repmgr connect_timeout=2
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | default | 100 | 5 | host=172.18.36.191 user=repmgr dbname=repmgr connect_timeout=2
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
ID | Name | Role | Status | Upstream | repmgrd | PID | Paused? | Upstream last seen
----+---------------------------+---------+-----------+---------------------------+-------------+-------+---------+--------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | running | 14380 | no | n/a
18346 | bos1-vcloud-static-161-47 | primary | - failed | | n/a | n/a | n/a | n/a
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | not running | n/a | n/a | n/a
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
PID of repmgrd:
-1
root@bos1-vcloud-static-161-49 [ ~ ]# su - postgres -c "date; repmgr cluster show; repmgr daemon status; echo 'PID of repmgrd:'; psql -d repmgr -A -t -c 'SELECT repmgr.get_repmgrd_pid();' "
Tue Mar 3 23:20:58 UTC 2020
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
-------+---------------------------+---------+-----------+---------------------------+----------+----------+----------+----------------------------------------------------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | default | 100 | 5 | host=172.18.36.190 user=repmgr dbname=repmgr connect_timeout=2
18346 | bos1-vcloud-static-161-47 | primary | - failed | | default | 100 | ? | host=172.18.36.189 user=repmgr dbname=repmgr connect_timeout=2
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | default | 100 | 5 | host=172.18.36.191 user=repmgr dbname=repmgr connect_timeout=2
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
ID | Name | Role | Status | Upstream | repmgrd | PID | Paused? | Upstream last seen
----+---------------------------+---------+-----------+---------------------------+-------------+-------+---------+--------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | running | 14380 | no | n/a
18346 | bos1-vcloud-static-161-47 | primary | - failed | | n/a | n/a | n/a | n/a
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | not running | n/a | n/a | n/a
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
PID of repmgrd:
-1
root@bos1-vcloud-static-161-49 [ ~ ]# su - postgres -c "date; repmgr cluster show; repmgr daemon status; echo 'PID of repmgrd:'; psql -d repmgr -A -t -c 'SELECT repmgr.get_repmgrd_pid();' "
Tue Mar 3 23:20:59 UTC 2020
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
-------+---------------------------+---------+-----------+---------------------------+----------+----------+----------+----------------------------------------------------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | default | 100 | 5 | host=172.18.36.190 user=repmgr dbname=repmgr connect_timeout=2
18346 | bos1-vcloud-static-161-47 | primary | - failed | | default | 100 | ? | host=172.18.36.189 user=repmgr dbname=repmgr connect_timeout=2
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | default | 100 | 5 | host=172.18.36.191 user=repmgr dbname=repmgr connect_timeout=2
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
ID | Name | Role | Status | Upstream | repmgrd | PID | Paused? | Upstream last seen
----+---------------------------+---------+-----------+---------------------------+---------+-------+---------+--------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | running | 14380 | no | n/a
18346 | bos1-vcloud-static-161-47 | primary | - failed | | n/a | n/a | n/a | n/a
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | running | 12024 | no | 0 second(s) ago
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
PID of repmgrd:
12024
root@bos1-vcloud-static-161-49 [ ~ ]# su - postgres -c "date; repmgr cluster show; repmgr daemon status; echo 'PID of repmgrd:'; psql -d repmgr -A -t -c 'SELECT repmgr.get_repmgrd_pid();' "
Tue Mar 3 23:21:00 UTC 2020
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
-------+---------------------------+---------+-----------+---------------------------+----------+----------+----------+----------------------------------------------------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | default | 100 | 5 | host=172.18.36.190 user=repmgr dbname=repmgr connect_timeout=2
18346 | bos1-vcloud-static-161-47 | primary | - failed | | default | 100 | ? | host=172.18.36.189 user=repmgr dbname=repmgr connect_timeout=2
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | default | 100 | 5 | host=172.18.36.191 user=repmgr dbname=repmgr connect_timeout=2
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
ID | Name | Role | Status | Upstream | repmgrd | PID | Paused? | Upstream last seen
----+---------------------------+---------+-----------+---------------------------+---------+-------+---------+--------------------
5556 | bos1-vcloud-static-161-48 | primary | * running | | running | 14380 | no | n/a
18346 | bos1-vcloud-static-161-47 | primary | - failed | | n/a | n/a | n/a | n/a
22882 | bos1-vcloud-static-161-49 | standby | running | bos1-vcloud-static-161-48 | running | 12024 | no | 1 second(s) ago
WARNING: following issues were detected
- unable to connect to node "bos1-vcloud-static-161-47" (ID: 18346)
PID of repmgrd:
12024
repmgr log file on the surviving standby. Note the 20 sec delay for our standby follow script between 23:20:39 and 23:20:58:
[2020-03-03 23:18:06] [INFO] node "bos1-vcloud-static-161-49" (ID: 22882) monitoring upstream node "bos1-vcloud-static-161-47" (ID: 18346) in normal state
[2020-03-03 23:20:31] [WARNING] unable to ping "host=172.18.36.189 user=repmgr dbname=repmgr connect_timeout=2"
[2020-03-03 23:20:31] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2020-03-03 23:20:31] [WARNING] unable to connect to upstream node "bos1-vcloud-static-161-47" (ID: 18346)
[2020-03-03 23:20:31] [INFO] checking state of node 18346, 1 of 6 attempts
[2020-03-03 23:20:31] [WARNING] unable to ping "user=repmgr connect_timeout=2 dbname=repmgr host=172.18.36.189 fallback_application_name=repmgr"
[2020-03-03 23:20:31] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2020-03-03 23:20:31] [INFO] sleeping 1 seconds until next reconnection attempt
[2020-03-03 23:20:32] [INFO] checking state of node 18346, 2 of 6 attempts
[2020-03-03 23:20:32] [WARNING] unable to ping "user=repmgr connect_timeout=2 dbname=repmgr host=172.18.36.189 fallback_application_name=repmgr"
[2020-03-03 23:20:32] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2020-03-03 23:20:32] [INFO] sleeping 1 seconds until next reconnection attempt
[2020-03-03 23:20:33] [INFO] checking state of node 18346, 3 of 6 attempts
[2020-03-03 23:20:33] [WARNING] unable to ping "user=repmgr connect_timeout=2 dbname=repmgr host=172.18.36.189 fallback_application_name=repmgr"
[2020-03-03 23:20:33] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2020-03-03 23:20:33] [INFO] sleeping 1 seconds until next reconnection attempt
[2020-03-03 23:20:34] [INFO] checking state of node 18346, 4 of 6 attempts
[2020-03-03 23:20:34] [WARNING] unable to ping "user=repmgr connect_timeout=2 dbname=repmgr host=172.18.36.189 fallback_application_name=repmgr"
[2020-03-03 23:20:34] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2020-03-03 23:20:34] [INFO] sleeping 1 seconds until next reconnection attempt
[2020-03-03 23:20:35] [INFO] checking state of node 18346, 5 of 6 attempts
[2020-03-03 23:20:35] [WARNING] unable to ping "user=repmgr connect_timeout=2 dbname=repmgr host=172.18.36.189 fallback_application_name=repmgr"
[2020-03-03 23:20:35] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2020-03-03 23:20:35] [INFO] sleeping 1 seconds until next reconnection attempt
[2020-03-03 23:20:36] [INFO] checking state of node 18346, 6 of 6 attempts
[2020-03-03 23:20:36] [WARNING] unable to ping "user=repmgr connect_timeout=2 dbname=repmgr host=172.18.36.189 fallback_application_name=repmgr"
[2020-03-03 23:20:36] [DETAIL] PQping() returned "PQPING_NO_RESPONSE"
[2020-03-03 23:20:36] [WARNING] unable to reconnect to node 18346 after 6 attempts
[2020-03-03 23:20:36] [INFO] 1 active sibling nodes registered
[2020-03-03 23:20:36] [INFO] primary and this node have the same location ("default")
[2020-03-03 23:20:36] [INFO] local node's last receive lsn: 0/E92B3228
[2020-03-03 23:20:36] [INFO] checking state of sibling node "bos1-vcloud-static-161-48" (ID: 5556)
[2020-03-03 23:20:36] [INFO] node "bos1-vcloud-static-161-48" (ID: 5556) reports its upstream is node 18346, last seen 6 second(s) ago
[2020-03-03 23:20:36] [INFO] node 5556 last saw primary node 6 second(s) ago
[2020-03-03 23:20:36] [INFO] last receive LSN for sibling node "bos1-vcloud-static-161-48" (ID: 5556) is: 0/E92B3228
[2020-03-03 23:20:36] [INFO] node "bos1-vcloud-static-161-48" (ID: 5556) has same LSN as current candidate "bos1-vcloud-static-161-49" (ID: 22882)
[2020-03-03 23:20:36] [INFO] node "bos1-vcloud-static-161-48" (ID: 5556) has same priority but lower node_id than current candidate "bos1-vcloud-static-161-49" (ID: 22882)
[2020-03-03 23:20:36] [INFO] visible nodes: 2; total nodes: 2; no nodes have seen the primary within the last 4 seconds
[2020-03-03 23:20:36] [NOTICE] promotion candidate is "bos1-vcloud-static-161-48" (ID: 5556)
[2020-03-03 23:20:36] [INFO] follower node awaiting notification from a candidate node
[2020-03-03 23:20:38] [INFO] local node 22882 can attach to follow target node 5556
[2020-03-03 23:20:38] [DETAIL] local node's recovery point: 0/E92B3228; follow target node's fork point: 0/E92B3228
[2020-03-03 23:20:38] [NOTICE] setting node 22882's upstream to node 5556
[2020-03-03 23:20:38] [NOTICE] restarting server using "sudo /usr/bin/systemctl restart vpostgres"
[2020-03-03 23:20:39] [NOTICE] STANDBY FOLLOW successful
[2020-03-03 23:20:39] [DETAIL] standby attached to upstream node "bos1-vcloud-static-161-48" (ID: 5556)
INFO: set_repmgrd_pid(): provided pidfile is
[2020-03-03 23:20:58] [NOTICE] node 22882 now following new upstream node 5556
[2020-03-03 23:20:58] [INFO] resuming standby monitoring mode
[2020-03-03 23:20:58] [DETAIL] following new primary "bos1-vcloud-static-161-48" (ID: 5556)
[2020-03-03 23:25:59] [INFO] node "bos1-vcloud-static-161-49" (ID: 22882) monitoring upstream node "bos1-vcloud-static-161-48" (ID: 5556) in normal state