Drive add start failed. Failed to get spec. Make sure PX is up and running (pxctl status).

[rlewkowicz]

Command

root@ip-172-31-28-166:/home/ubuntu# pxctl service maintenance --enter
This is a disruptive operation, PX will restart in maintenance mode.
Are you sure you want to proceed ? (Y/N): Y
Maintenance: Local storage spec not initialized

root@ip-172-31-28-166:/home/ubuntu# pxctl service drive add --drive /dev/xvda1 --operation start
Drive add start failed. Failed to get spec. Make sure PX is up and running (pxctl status).

I have 3 nodes spun up as part of an auto scaling group.

Status: PX Cluster is not in quorum because there are no storage devices allocated to any of the nodes.
	Please add storage device to atleast one node and restart PX.
	Please take a look at: https://docs.portworx.com/#install for instructions on how to provide storage.
License: PX-Developer
Node ID: 412de949-2ecc-4243-9a2d-c1d2a42a086a
	IP: 172.31.28.166
 	Local Storage Pool: 0 pool
	POOL	IO_PRIORITY	RAID_LEVEL	USABLE	USED	STATUS	ZONE	REGION
	No storage pool
	Local Storage Devices: 0 device
	Device	Path	Media Type	Size		Last-Scan
	No storage device
	total		-	0 B
Cluster Summary
	Cluster ID: px-cluster-nomadv8
	Cluster UUID: 0443b80c-c5ff-4bd3-b632-d077a0e64b78
	Scheduler: none
	Nodes: 3 node(s) without storage
	IP		ID					SchedulerNodeName	StorageNode	Used		Capacity	Status		StorageStatus		Version		Kernel		OS
	172.31.47.234	e37e8a7e-5582-477a-a8ee-abe2ab14b947	N/A			No		Unavailable	Unavailable	Not Available	No Storage		2.1.3.0-651d5d4	4.15.0-1044-awsUbuntu 18.04.2 LTS
	172.31.32.74	70d86852-2b9b-4cdd-892f-fc65e5a3ba05	N/A			No		Unavailable	Unavailable	Not Available	No Storage		2.1.3.0-651d5d4	4.15.0-1044-awsUbuntu 18.04.2 LTS
	172.31.28.166	412de949-2ecc-4243-9a2d-c1d2a42a086a	N/A			No		Unavailable	Unavailable	Not in Quorum	No Storage (This node)	2.1.3.0-651d5d4	4.15.0-1044-awsUbuntu 18.04.2 LTS
Global Storage Pool
	Total Used    	:  0 B
	Total Capacity	:  0 B

The disk is the root volume

root@ip-172-31-28-166:/home/ubuntu# fdisk -l
...
...
...
Disk /dev/xvda: 200 GiB, 214748364800 bytes, 419430400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x7dcfac84

Device     Boot Start       End   Sectors  Size Id Type
/dev/xvda1 *     2048 419430366 419428319  200G 83 Linux

Interestingly enough spinning up a standard instance using aws_instance in terraform worked just fine.

[kmunson1973]

You will need to have disks available when installing Portworx on the initial 3 nodes. Once that is completed you can then add drives individually. At this point, the easiest thing will be to uninstall and reinstall Potworx. You can follow the doc here https://docs.portworx.com/portworx-install-with-kubernetes/operate-and-maintain-on-kubernetes/uninstall/uninstall/. Portworx also supports a feature called cloud drives that will automatically create disks in an ASG environment. Please let us know if you need additional information.