contiv-experimental/cluster

Services are not running when bringing up node after shutdown in B-series cluster

pradvara opened this issue · 0 comments

pradvara commented

Steps to reproduce:

Add nodes to cluster

[stack@contiv-b1 ~]$ clusterctl nodes get | grep " status"
contiv-b1-FCH1702J22M: status: Allocated
contiv-b2-FCH1701J2KV: status: Allocated
contiv-b3-FCH1828KBGQ: status: Allocated
contiv-b4-FCH1811JLXV: status: Allocated
contiv-b5-FCH1834JF2M: status: Allocated
contiv-b6-FCH1811JD9C: status: Allocated

Shutdown a node:

[stack@contiv-b1 ~]$ clusterctl node get contiv-b5-FCH1834JF2M
contiv-b5-FCH1834JF2M: Inventory State
contiv-b5-FCH1834JF2M: name: contiv-b5-FCH1834JF2M
contiv-b5-FCH1834JF2M: prev_state: Discovered
contiv-b5-FCH1834JF2M: prev_status: Allocated
contiv-b5-FCH1834JF2M: state: Disappeared
contiv-b5-FCH1834JF2M: status: Allocated
contiv-b5-FCH1834JF2M: Monitoring State
contiv-b5-FCH1834JF2M: label: contiv-b5
contiv-b5-FCH1834JF2M: management_address: 10.106.240.109
contiv-b5-FCH1834JF2M: serial_number: FCH1834JF2M
contiv-b5-FCH1834JF2M: Configuration State
contiv-b5-FCH1834JF2M: host_group: service-worker
contiv-b5-FCH1834JF2M: inventory_name: contiv-b5-FCH1834JF2M
contiv-b5-FCH1834JF2M: inventory_vars:
contiv-b5-FCH1834JF2M: node_addr: 10.106.240.109
contiv-b5-FCH1834JF2M: node_name: contiv-b5-FCH1834JF2M
contiv-b5-FCH1834JF2M: ssh_address: 10.106.240.109

The state of the node moves to "Disappeared"

Bring up the node again

[stack@contiv-b1 ~]$ clusterctl node get contiv-b5-FCH1834JF2M
contiv-b5-FCH1834JF2M: Inventory State
contiv-b5-FCH1834JF2M: name: contiv-b5-FCH1834JF2M
contiv-b5-FCH1834JF2M: prev_state: Disappeared
contiv-b5-FCH1834JF2M: prev_status: Allocated
contiv-b5-FCH1834JF2M: state: Discovered
contiv-b5-FCH1834JF2M: status: Allocated
contiv-b5-FCH1834JF2M: Monitoring State
contiv-b5-FCH1834JF2M: label: contiv-b5
contiv-b5-FCH1834JF2M: management_address: 10.106.240.109
contiv-b5-FCH1834JF2M: serial_number: FCH1834JF2M
contiv-b5-FCH1834JF2M: Configuration State
contiv-b5-FCH1834JF2M: host_group: service-worker
contiv-b5-FCH1834JF2M: inventory_name: contiv-b5-FCH1834JF2M
contiv-b5-FCH1834JF2M: inventory_vars:
contiv-b5-FCH1834JF2M: node_addr: 10.106.240.109
contiv-b5-FCH1834JF2M: node_name: contiv-b5-FCH1834JF2M
contiv-b5-FCH1834JF2M: ssh_address: 10.106.240.109

The state is Discovered, and status is allocated.

Check wheather services are running in node

[stack@contiv-b5 ~]$ ps -ef | grep netplugin
stack 2992 2785 0 15:50 pts/0 00:00:00 grep --color=auto netplugin

[stack@contiv-b5 ~]$ systemctl status docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/docker.service.d
└─env.conf
Active: inactive (dead)
Docs: https://docs.docker.com

[stack@contiv-b5 ~]$ sudo systemctl status etcd
● etcd.service - Etcd
Loaded: loaded (/etc/systemd/system/etcd.service; static; vendor preset: disabled)
Active: inactive (dead)

Services are not running, but the status is allocated