openshift/cluster-etcd-operator

reboot etcd node cause etcd member list lose

lance5890 opened this issue · 0 comments

What happened?
when I reboot one etcd node,i found this node is removed from the member list sometimes

What did you expect to happen?
because this happens occasionally, i wonder when i shutdown this node , is it ok to remove this node form member list

How can we reproduce it (as minimally and precisely as possible)?
1 we have 3 nodes
[root@zn-node5 /]# etcdctl member list
298c08557f0f5c94, started, node7, https://10.255.136.107:2380/, https://10.255.136.107:2379/, false
795d385bdf226ec9, started, node6, https://10.255.136.106:2380/, https://10.255.136.106:2379/, false
f33aaabc2a13d637, started, node5, https://10.255.136.105:2380/, https://10.255.136.105:2379/, false

2 then i shutdown node6,and i found this node is removed from the member list
[root@zn-node5 /]# etcdctl member list
298c08557f0f5c94, started, node7, https://10.255.136.107:2380/, https://10.255.136.107:2379/, false
f33aaabc2a13d637, started, node5, https://10.255.136.105:2380/, https://10.255.136.105:2379/, false

3 Log

{"level":"debug","ts":"2022-07-13T03:27:18.121Z","caller":"etcdserver/server.go:2107","msg":"Applying entries","num-entries":1}
{"level":"debug","ts":"2022-07-13T03:27:18.121Z","caller":"etcdserver/server.go:2110","msg":"Applying entry","index":3898929,"term":191,"type":"EntryConfChange"}
{"level":"info","ts":"2022-07-13T03:27:18.121Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"f33aaabc2a13d637 switched to configuration voters=(2993777015591492756 175265082223159)"}
{"level":"info","ts":"2022-07-13T03:27:18.121Z","caller":"membership/cluster.go:472","msg":"removed member","cluster-id":"29732bb804dd83e8","local-member-id":"f33aaabc2a13d637","removed-r-peer-id":"795d385bdf226ec9","removed-remote-peer-urls":["https://10.255.136.106:2380"]}
{"level":"info","ts":"2022-07-13T03:27:18.121Z","caller":"rafthttp/peer.go:330","msg":"stopping remote peer","remote-peer-id":"795d385bdf226ec9"}
{"level":"info","ts":"2022-07-13T03:27:18.121Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","remote-id":"795d385bdf226ec9"}
{"level":"info","ts":"2022-07-13T03:27:18.121Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"stream Message","remote-pd":"795d385bdf226ec9"}
{"level":"info","ts":"2022-07-13T03:27:18.121Z","caller":"rafthttp/pipeline.go:85","msg":"stopped HTTP pipelining with remote peer","local-member-id":"f33aaabc2a13d637","remote-peer-id":"85bdf226ec9"}
{"level":"info","ts":"2022-07-13T03:27:18.121Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":aabc2a13d637","remote-peer-id":"795d385bdf226ec9"}
{"level":"info","ts":"2022-07-13T03:27:18.121Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"fbc2a13d637","remote-peer-id":"795d385bdf226ec9"}
{"level":"info","ts":"2022-07-13T03:27:18.121Z","caller":"rafthttp/peer.go:335","msg":"stopped remote peer","remote-peer-id":"795d385bdf226ec9"}
{"level":"info","ts":"2022-07-13T03:27:18.121Z","caller":"rafthttp/transport.go:355","msg":"removed remote peer","local-member-id":"f33aaabc2a13d637","removed-remote-peer-id":"795d385bdf2"}
{"level":"debug","ts":"2022-07-13T03:27:18.124Z","caller":"auth/store.go:1133","msg":"found command name","common-name":"etcd","user-name":"etcd","revision":1}