portworx/px-dev

Cluster offline node can‘t be deleted

AlvinLv00 opened this issue · 1 comments

The cluster node is offline. Delete the following error:

#pxctl c p
NODE NODE STATUS POOL POOL STATUS IO_PRIORITY SIZE AVAILABLE USED PROVISIONED RESERVEFACTOR ZONE REGION RACK
5769741b-0e67-4abc-a8f2-123422ad078f Offline 0 Offline HIGH 1.6 TiB 1.6 TiB 62 GiB 50 GiB 0 default default default
64120b54-7801-4ca7-89b5-5bee630096ad Online 0 Online HIGH 1.6 TiB 1.6 TiB 31 GiB 882 GiB 0 default default default
6fb83078-cbce-42be-9761-a23e5fc26e41 Offline 0 Online HIGH 1.6 TiB 1.6 TiB 60 GiB 1.1 TiB 0 default default default
91f92a69-d844-4e99-8f42-8c4896854295 Online 0 Online HIGH 1.6 TiB 1.6 TiB 41 GiB 1.1 TiB 0 default default default

#pxctl c delete 6fb83078-cbce-42be-9761-a23e5fc26e41
delete node: Node Remove: Cannot remove node ID 6fb83078-cbce-42be-9761-a23e5fc26e41: Node add in progress for 6fb83078-cbce-42be-9761-a23e5fc26e41.

# pxctl c delete -f 6fb83078-cbce-42be-9761-a23e5fc26e41
This is a distruptive operation.
Volumes which reside only on this node or which have latest data only on this node will be irrevocably deleted.
Are you sure you want to proceed ? (Y/N): Y
delete node: Node Remove: Cannot remove node ID 6fb83078-cbce-42be-9761-a23e5fc26e41: Node add in progress for 6fb83078-cbce-42be-9761-a23e5fc26e41.

The node information is as follows:
pxctl c i 6fb83078-cbce-42be-9761-a23e5fc26e41
ID : 6fb83078-cbce-42be-9761-a23e5fc26e41
Mgmt IP : 10.x.x.x
Data IP : 10.x.x.x
CPU : 0
Mem Total : 0
Mem Used : 0
Volumes :

Status : Offline

And the related mount volume is detached and delete,still can't delete,and development Edition restricts clusters to only three nodes,cause new nodes to fail to join。

@AlvinLv00 if you can collect the journal logs for portworx, we can see what is going on. It is likely these nodes have replicas for other volumes in them.

Can you also attach pxctl status