Volume will not unmount
Closed this issue · 5 comments
$ pxctl status
Status: PX is operational
License: Trial (expires in 22 days)
Node ID: 5de69d6c-1cff-4ae1-8b37-e7f4e89c7f71
IP: 192.168.10.68
Local Storage Pool: 1 pool
POOL IO_PRIORITY RAID_LEVEL USABLE USED STATUS ZONE REGION
0 HIGH raid0 100 GiB 2.9 GiB Online default default
Local Storage Devices: 1 device
Device Path Media Type Size Last-Scan
0:1 /dev/sdb STORAGE_MEDIUM_MAGNETIC 100 GiB 20 Jul 17 15:47 UTC
total - 100 GiB
Cluster Summary
Cluster ID: devops-cluster-01
Cluster UUID: 5238855239480218275
IP ID Used Capacity Status
192.168.10.68 5de69d6c-1cff-4ae1-8b37-e7f4e89c7f71 2.9 GiB 100 GiB Online (This node)
192.168.10.138 1a47ccf0-95d8-4964-a4e1-3f0adbe6e707 2.7 GiB 100 GiB Online
Global Storage Pool
Total Used : 5.6 GiB
Total Capacity : 200 GiB
~$ pxctl v l
ID NAME SIZE HA SHARED ENCRYPTED IO_PRIORITY SCALE STATUS
105555218929839946 jenkins-home 10 GiB 1 no no LOW 1 up - attached on 192.168.10.68
1081000382262830454 jenkins-slave-ssh 1 GiB 2 yes no LOW 1 up - attached on 192.168.10.138
1013395067276956434 m2-repository 4 GiB 2 yes no LOW 1 up - attached on 192.168.10.68
$ pxctl v d 1013395067276956434
delete: Volume ID 1013395067276956434 is attached on IP 192.168.10.68 (5de69d6c-1cff-4ae1-8b37-e7f4e89c7f71). Please stop the Docker container using the volume first.
==== second host =====
$ pxctl host attach 1013395067276956434
Volume successfully attached at: /dev/pxd/pxd1013395067276956434
abienkow@adrian-vm:~$ pxctl v l
ID NAME SIZE HA SHARED ENCRYPTED IO_PRIORITY SCALE STATUS
105555218929839946 jenkins-home 10 GiB 1 no no LOW 1up - attached on 192.168.10.68
1081000382262830454 jenkins-slave-ssh 1 GiB 2 yes no LOW 1up - attached on 192.168.10.138
1013395067276956434 m2-repository 4 GiB 2 yes no LOW 1up - attached on 192.168.10.68
==== back to first host ===
$ pxctl v d 1013395067276956434
delete: Volume 1013395067276956434 is mounted at 1 location(s): /var/lib/osd/pxns/1013395067276956434
$ pxctl host unmount 1013395067276956434 --path /var/lib/osd/pxns/1013395067276956434
Volume 1013395067276956434 successfully unmounted at /var/lib/osd/pxns/1013395067276956434
$ pxctl v d 1013395067276956434 delete: Volume 1013395067276956434 is mounted at 1 location(s): /var/lib/osd/pxns/1013395067276956434
Seems like you are trying to delete shared volume. shared volumes can be mounted at multiple nodes. All of them must be unmounted before the delete. Could you paste the out of following command?
"pxctl -j v i 1013395067276956434"
$ pxctl -j v i 1013395067276956434
[{
"id": "1013395067276956434",
"readonly": false,
"locator": {
"name": "m2-repository"
},
"ctime": "2017-07-19T12:11:10Z",
"spec": {
"ephemeral": false,
"size": "4294967296",
"format": "ext4",
"block_size": "32768",
"ha_level": "2",
"cos": "low",
"io_profile": "sequential",
"dedupe": false,
"snapshot_interval": 0,
"shared": true,
"replica_set": {
},
"aggregation_level": 1,
"encrypted": false,
"passphrase": "",
"snapshot_schedule": "",
"scale": 1,
"sticky": false,
"group_enforced": false
},
"usage": "549191680",
"last_scan": "2017-07-19T12:11:10Z",
"format": "ext4",
"status": "up",
"state": "attached",
"attached_on": "5de69d6c-1cff-4ae1-8b37-e7f4e89c7f71",
"device_path": "/dev/pxd/pxd1013395067276956434",
"attach_path": [
".138",
".68"
],
"replica_sets": [
{
"nodes": [
"1a47ccf0-95d8-4964-a4e1-3f0adbe6e707",
"5de69d6c-1cff-4ae1-8b37-e7f4e89c7f71"
]
}
],
"error": "",
"runtime_state": [
{
"runtime_state": {
"FullResyncBlocks": "[{0 0} {3 0} {-1 0} {-1 0} {-1 0}]",
"ID": "0",
"ReadQuorum": "1",
"ReadSet": "[0 3]",
"ReplicaSetCurr": "[0 3]",
"ReplicaSetNext": "[0 3]",
"ResyncBlocks": "[{0 0} {3 0} {-1 0} {-1 0} {-1 0}]",
"RuntimeState": "clean",
"TimestampBlocksPerNode": "[0 0 0 0 0]",
"TimestampBlocksTotal": "0",
"WriteQuorum": "2",
"WriteSet": "[0 3]"
}
}
],
"secure_device_path": "",
"background_processing": false
}]
Seems like volume is mounted on 2 nodes.
"attach_path": [
".138",
".68"
],
if you issue "mount | grep px", that will show the mount paths. these paths on respective nodes need to be unmounted using the following command
pxctl host unmount 1013395067276956434
The above command must be executed on both .138 and .68 for the respective mount paths
Then you can detach and delete the volume.
pxctl host unmounts requires --path. Using mount to show all mounted device on the host does not show the expected volume to be mounted on any path. Using the device_path from the inspect command has no affect. also looking in /dev/pxd/ does not show the specific volume id on either host.
That is correct. For shared volumes it is fuse mount and not the block device mount. Mount output will show pxfs mounts. That would be the path input for pxctl unmount command. For example in my system mount output is as below.
mount | grep px
/dev/pxd/pxd393534584233243386 on /var/lib/osd/pxns/393534584233243386 type ext4 (rw,relatime,discard,data=ordered)
pxfs on /var/lib/osd/mounts/svol type fuse.pxfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
Here the path var/lib/osd/mounts/svol where the pxfs is mounted is the path parameter to be provided as input to unmount command.