sometimes peers are deleted after master change
kmlebedev opened this issue · 0 comments
kmlebedev commented
sometimes peers are deleted(13:51) after master change (12:39)
raft v1.3.9
in the logs i see:
May 20, 2022 @ 12:39:21.877 2022-05-20T07:39:21.877Z [WARN] raft: failed to contact: server-id=slow-master-0.dc2:9333 time=317.196538ms slow-master-1(node9-dc1)
May 20, 2022 @ 12:39:21.999 2022-05-20T07:39:21.999Z [WARN] raft: heartbeat timeout reached, starting election: last-leader-addr=slow-master-1.dc1:9333 last-leader-id= slow-master-0(node9-dc2)
May 20, 2022 @ 12:39:22.001 2022-05-20T07:39:22.001Z [WARN] raft: rejecting vote request since we have a leader: from=slow-master-0.dc2:9333 leader=slow-master-1.dc1:9333 leader-id="" slow-master-0(node3-dc2)
May 20, 2022 @ 12:39:22.002 2022-05-20T07:39:22.000Z [WARN] raft: rejecting vote request since we have a leader: from=slow-master-0.dc2:9333 leader=slow-master-1.dc1:9333 leader-id=slow-master-1.dc1:9333 slow-master-1(node9-dc1)
May 20, 2022 @ 12:39:22.164 2022-05-20T07:39:22.163Z [WARN] raft: failed to contact: server-id=slow-master-0.dc2:9333 time=602.853871ms slow-master-1(node9-dc1)
May 20, 2022 @ 12:39:22.199 2022-05-20T07:39:22.198Z [ERROR] raft: peer has newer term, stopping replication: peer="{Voter slow-master-0.dc2:9333 slow-master-0.dc2:19333}" slow-master-1(node9-dc1)
May 20, 2022 @ 12:39:22.754 2022-05-20T07:39:22.754Z [WARN] raft: heartbeat timeout reached, starting election: last-leader-addr=slow-master-1.dc1:9333 last-leader-id= slow-master-0(node3-dc1)
May 20, 2022 @ 13:51:23.826 2022-05-20T08:51:23.825Z [WARN] raft: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election slow-master-1(node9-dc1)
May 20, 2022 @ 13:51:24.259 2022-05-20T08:51:24.257Z [WARN] raft: not part of stable configuration, aborting election slow-master-0(node9-dc2)
How can this be avoided?