StorageClass not respecting the labels on the drives
zeph opened this issue · 24 comments
zeph commented
Describe the bug
I associated the disktype=hdd label to 25GiB drives and the disktype=ssd on others.
On creation of a tenant, MinIO's directpv just used the 1st available ones disregarding the label.
To Reproduce
Got the 2 set of drives labeled...
% kubectl directpv label drives disktype=ssd --drives=sd{b,c,d,e}% kubectl directpv label drives disktype=hdd --drives=sd{f,g,h,i}
getting % kubectl directpv list drives --output wide --show-labels
┌──────────────────────────┬──────┬─────────────────────┬────────┬─────────┬─────────┬────────┬──────────────────────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ NODE │ NAME │ MAKE │ SIZE │ FREE │ VOLUMES │ STATUS │ DRIVE ID │ LABELS │
├──────────────────────────┼──────┼─────────────────────┼────────┼─────────┼─────────┼────────┼──────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ mbbf-extern-k8s-worker-4 │ sdf │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ ebd5313d-ef0a-493e-bd50-b7087cf49b53 │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdf,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdg │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ 9f437a30-2431-4549-997d-1eb126d7f63b │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdg,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdh │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ 18388cbc-f291-40df-b4a4-5f21f658f38d │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdh,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdi │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ 84a81ed3-5e2e-45cc-a23f-aab6471e6e4e │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdi,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdb │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ c640b9bb-cdab-4582-8fd3-a12fe7ade9ea │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdb,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdc │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ dc956005-c68c-454c-82bc-219bf0430e83 │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdc,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdd │ VMware Virtual_disk │ 25 GiB │ 25 GiB │ - │ Ready │ 4306c5f5-22d2-457a-bddf-1015942c0dbe │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdd,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sde │ VMware Virtual_disk │ 25 GiB │ 5.0 GiB │ 2 │ Ready │ 7df9e86e-9176-4468-8300-d6fd66e69945 │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sde,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdf │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ 7a83fcba-e04a-453a-9e3e-5f0f50e415cb │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdf,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdg │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ a8adab8f-ef59-4f29-87e6-1a93c9920ee1 │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdg,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdh │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ da347aad-c8d5-4af9-9ea4-aa5440b8679b │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdh,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdi │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ ea9466da-3faa-4ba2-bfdc-1ba66393ed6a │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdi,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdb │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ 86c2a0fa-a5dd-4b08-a8ec-af0ce89a27a8 │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdb,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdc │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ cd571970-ff8b-47b0-a1f7-ee974b20efc4 │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdc,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdd │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ dce8daf3-4681-450e-bd71-eb9147825754 │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdd,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sde │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ 81c52ff1-2eed-4d4d-8563-1bc8ecf4446c │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sde,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdf │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ 7f9bd6b8-a318-4e22-8cc6-949b8d4d67d4 │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdf,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdg │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ 84db6f9c-4180-4eb1-9a45-b9304c5be44c │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdg,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdh │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ aeaed335-2b8b-4c08-a168-702de57c9f39 │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdh,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdi │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ 3a771c42-95fe-470a-9d53-204cee918dfc │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdi,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdd │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ 04d14895-1424-460b-a0e1-ed1089fc27eb │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdd,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sde │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ 1a8911b9-01ae-49ce-bcea-f9ca360cf95e │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sde,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdb │ VMware Virtual_disk │ 25 GiB │ 25 GiB │ - │ Ready │ aeb58bcb-a5cc-40a1-aa3b-02014faa430d │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdb,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdc │ VMware Virtual_disk │ 25 GiB │ 5.0 GiB │ 2 │ Ready │ 6d59f5fe-28ce-4689-aaf8-7782cd366deb │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdc,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
└──────────────────────────┴──────┴─────────────────────┴────────┴─────────┴─────────┴────────┴──────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
which works just fine...
% directpv.stage list drives --labels disktype=ssd
┌──────────────────────────┬──────┬─────────────────────┬────────┬────────┬─────────┬────────┐
│ NODE │ NAME │ MAKE │ SIZE │ FREE │ VOLUMES │ STATUS │
├──────────────────────────┼──────┼─────────────────────┼────────┼────────┼─────────┼────────┤
│ mbbf-extern-k8s-worker-4 │ sdf │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │
│ mbbf-extern-k8s-worker-4 │ sdg │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │
│ mbbf-extern-k8s-worker-4 │ sdh │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │
│ mbbf-extern-k8s-worker-4 │ sdi │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │
│ mbbf-extern-k8s-worker-5 │ sdf │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │
│ mbbf-extern-k8s-worker-5 │ sdg │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │
│ mbbf-extern-k8s-worker-5 │ sdh │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │
│ mbbf-extern-k8s-worker-5 │ sdi │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │
│ mbbf-extern-k8s-worker-6 │ sdf │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │
│ mbbf-extern-k8s-worker-6 │ sdg │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │
│ mbbf-extern-k8s-worker-6 │ sdh │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │
│ mbbf-extern-k8s-worker-6 │ sdi │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │
└──────────────────────────┴──────┴─────────────────────┴────────┴────────┴─────────┴────────┘
./create-storage-class.sh ssd-tier-storage 'disktype: ssd'
% k8s describe StorageClass ssd-tier-storage
Name: ssd-tier-storage
IsDefaultClass: No
Annotations: kubectl.kubernetes.io/last-applied-configuration={"allowVolumeExpansion":true,"allowedTopologies":[{"matchLabelExpressions":[{"key":"directpv.min.io/identity","values":["directpv-min-io"]}]}],"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"finalizers":["foregroundDeletion"],"labels":{"application-name":"directpv.min.io","application-type":"CSIDriver","directpv.min.io/created-by":"kubectl-directpv","directpv.min.io/version":"v1beta1"},"name":"ssd-tier-storage"},"parameters":{"disktype":"ssd","fstype":"xfs"},"provisioner":"directpv-min-io","reclaimPolicy":"Delete","volumeBindingMode":"WaitForFirstConsumer"}
Provisioner: directpv-min-io
Parameters: disktype=ssd,fstype=xfs
AllowVolumeExpansion: True
MountOptions: <none>
ReclaimPolicy: Delete
VolumeBindingMode: WaitForFirstConsumer
AllowedTopologies:
Term 0: directpv.min.io/identity in [directpv-min-io]
Events: <none>
and here is the Claim for the tenant... one of the claims
% k8s describe pvc data0-stage-pool-0-0
Name: data0-stage-pool-0-0
Namespace: tenant-stage
StorageClass: ssd-tier-storage
Status: Bound
Volume: pvc-d09e2143-74ad-4a43-a4b8-568cf402b37f
Labels: v1.min.io/console=stage-console
v1.min.io/pool=pool-0
v1.min.io/tenant=stage
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: directpv-min-io
volume.kubernetes.io/selected-node: mbbf-extern-k8s-worker-6
volume.kubernetes.io/storage-provisioner: directpv-min-io
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 10728470391
Access Modes: RWO
VolumeMode: Filesystem
Used By: stage-pool-0-0
Events: <none>
well, the PVs end up on the wrong label...
% directpv.stage list drives --labels disktype=hdd
┌──────────────────────────┬──────┬─────────────────────┬────────┬─────────┬─────────┬────────┐
│ NODE │ NAME │ MAKE │ SIZE │ FREE │ VOLUMES │ STATUS │
├──────────────────────────┼──────┼─────────────────────┼────────┼─────────┼─────────┼────────┤
│ mbbf-extern-k8s-worker-4 │ sdb │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │
│ mbbf-extern-k8s-worker-4 │ sdc │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │
│ mbbf-extern-k8s-worker-4 │ sdd │ VMware Virtual_disk │ 25 GiB │ 25 GiB │ - │ Ready │
│ mbbf-extern-k8s-worker-4 │ sde │ VMware Virtual_disk │ 25 GiB │ 5.0 GiB │ 2 │ Ready │
│ mbbf-extern-k8s-worker-5 │ sdb │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │
│ mbbf-extern-k8s-worker-5 │ sdc │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │
│ mbbf-extern-k8s-worker-5 │ sdd │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │
│ mbbf-extern-k8s-worker-5 │ sde │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │
│ mbbf-extern-k8s-worker-6 │ sdd │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │
│ mbbf-extern-k8s-worker-6 │ sde │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │
│ mbbf-extern-k8s-worker-6 │ sdb │ VMware Virtual_disk │ 25 GiB │ 25 GiB │ - │ Ready │
│ mbbf-extern-k8s-worker-6 │ sdc │ VMware Virtual_disk │ 25 GiB │ 5.0 GiB │ 2 │ Ready │
Expected behavior
I'd love to see the 10 GiB PVCs being fulfilled by the 10GiB drives labeled as disktype=ssd
Screenshots and logs
(just let me know and I will add anything necessary)
Deployment information (please complete the following information):
- DirectPV version: v4.0.6
- Kubernetes Version: v1.24.15
- OS info: Ubuntu 20.04.3 LTS
- Kernel version: 5.4.0-144-generic
Additional context
This is the VMWare based test setup as discussed in this ticket too #857
I'm trying to have a setup with 2 tenants to test the multi-tiering
zeph commented
adding a bit of extra context... as now, for this stage environment, I'm forcing 1st a tenant-stage-warm creation on the hdd drives, that in theory shall allocate only on the bigger ones (also labeled as disktype=hdd) and it did... now I'm trying to allocate the other tenant tenant-stage with the 10 GiB drives... and I was expecting it to work out of the box, not
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForFirstConsumer 28s persistentvolume-controller waiting for first consumer to be created before binding
Normal Provisioning 13s (x5 over 28s) directpv-min-io_controller-54f6fd599d-6stbd_8eb9e7a2-976a-43f2-b972-fcf99dbc3076 External provisioner is provisioning volume for claim "tenant-stage/data0-stage-pool-0-0"
Warning ProvisioningFailed 13s (x5 over 28s) directpv-min-io_controller-54f6fd599d-6stbd_8eb9e7a2-976a-43f2-b972-fcf99dbc3076 failed to provision volume with StorageClass "ssd-tier-storage": error generating accessibility requirements: no topology key found on CSINode mbbf-extern-k8s-worker-3
Normal ExternalProvisioning 3s (x3 over 28s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "directpv-min-io" or manually created by system administrator
no topology key found on CSINode mbbf-extern-k8s-worker-3 which remember I had to fiddle with the nodes marking the role on workers 4,5,6 as the 1,2,3 did not have a drive... therefore I conducted the directpv installation using: kubectl directpv install --node-selector node-role.kubernetes.io/minio=true
balamurugana commented
-
In the description, I see, you labeled drives by
% kubectl directpv label drives disktype=hdd --drives=sd{f,g,h,i}, but% directpv.stage list drives --labels disktype=ssdshowedhddlabeled drives asssd. What isdirectpv.stage? -
Please share
kubectl directpv list volumes --all -o wideoutput to know each volume is on what drive.
zeph commented
directpv.stage is just an alias for kubectl directpv as
I have to use a 1.22 build to talk to a rancher cluster
that said:
- showed
hddlabelled drives assdd??? true, I copied from my BASH_HISTORY the wrong attempts:
- all 10 GiB drives are to be considered
ssd-> {f,g,h,i} - all 25 GiB drives are to be considered
hdd-> {b,c,d,e}
...exactly opposite of what I wrote when opening the ticket, my bad
- that is already provided, and listed above... anyhow, I will add
the current one (after I created 1st the volumes with 25GiB to workaround the issue)
% directpv.stage list drives --output wide --show-labels
┌──────────────────────────┬──────┬─────────────────────┬────────┬─────────┬─────────┬────────┬──────────────────────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ NODE │ NAME │ MAKE │ SIZE │ FREE │ VOLUMES │ STATUS │ DRIVE ID │ LABELS │
├──────────────────────────┼──────┼─────────────────────┼────────┼─────────┼─────────┼────────┼──────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ mbbf-extern-k8s-worker-4 │ sdf │ VMware Virtual_disk │ 10 GiB │ 6.9 MiB │ 1 │ Ready │ ebd5313d-ef0a-493e-bd50-b7087cf49b53 │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdf,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdg │ VMware Virtual_disk │ 10 GiB │ 6.9 MiB │ 1 │ Ready │ 9f437a30-2431-4549-997d-1eb126d7f63b │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdg,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdh │ VMware Virtual_disk │ 10 GiB │ 6.9 MiB │ 1 │ Ready │ 18388cbc-f291-40df-b4a4-5f21f658f38d │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdh,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdi │ VMware Virtual_disk │ 10 GiB │ 6.9 MiB │ 1 │ Ready │ 84a81ed3-5e2e-45cc-a23f-aab6471e6e4e │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdi,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdb │ VMware Virtual_disk │ 25 GiB │ 4.4 MiB │ 1 │ Ready │ c640b9bb-cdab-4582-8fd3-a12fe7ade9ea │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdb,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdc │ VMware Virtual_disk │ 25 GiB │ 4.4 MiB │ 1 │ Ready │ dc956005-c68c-454c-82bc-219bf0430e83 │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdc,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdd │ VMware Virtual_disk │ 25 GiB │ 4.4 MiB │ 1 │ Ready │ 4306c5f5-22d2-457a-bddf-1015942c0dbe │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdd,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sde │ VMware Virtual_disk │ 25 GiB │ 4.4 MiB │ 1 │ Ready │ 7df9e86e-9176-4468-8300-d6fd66e69945 │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sde,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdf │ VMware Virtual_disk │ 10 GiB │ 6.9 MiB │ 1 │ Ready │ 7a83fcba-e04a-453a-9e3e-5f0f50e415cb │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdf,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdg │ VMware Virtual_disk │ 10 GiB │ 6.9 MiB │ 1 │ Ready │ a8adab8f-ef59-4f29-87e6-1a93c9920ee1 │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdg,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdh │ VMware Virtual_disk │ 10 GiB │ 6.9 MiB │ 1 │ Ready │ da347aad-c8d5-4af9-9ea4-aa5440b8679b │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdh,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdi │ VMware Virtual_disk │ 10 GiB │ 6.9 MiB │ 1 │ Ready │ ea9466da-3faa-4ba2-bfdc-1ba66393ed6a │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdi,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdb │ VMware Virtual_disk │ 25 GiB │ 4.4 MiB │ 1 │ Ready │ 86c2a0fa-a5dd-4b08-a8ec-af0ce89a27a8 │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdb,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdc │ VMware Virtual_disk │ 25 GiB │ 4.4 MiB │ 1 │ Ready │ cd571970-ff8b-47b0-a1f7-ee974b20efc4 │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdc,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdd │ VMware Virtual_disk │ 25 GiB │ 4.4 MiB │ 1 │ Ready │ dce8daf3-4681-450e-bd71-eb9147825754 │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdd,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sde │ VMware Virtual_disk │ 25 GiB │ 4.4 MiB │ 1 │ Ready │ 81c52ff1-2eed-4d4d-8563-1bc8ecf4446c │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sde,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdf │ VMware Virtual_disk │ 10 GiB │ 6.9 MiB │ 1 │ Ready │ 7f9bd6b8-a318-4e22-8cc6-949b8d4d67d4 │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdf,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdg │ VMware Virtual_disk │ 10 GiB │ 6.9 MiB │ 1 │ Ready │ 84db6f9c-4180-4eb1-9a45-b9304c5be44c │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdg,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdh │ VMware Virtual_disk │ 10 GiB │ 6.9 MiB │ 1 │ Ready │ aeaed335-2b8b-4c08-a168-702de57c9f39 │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdh,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdi │ VMware Virtual_disk │ 10 GiB │ 6.9 MiB │ 1 │ Ready │ 3a771c42-95fe-470a-9d53-204cee918dfc │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdi,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdb │ VMware Virtual_disk │ 25 GiB │ 4.4 MiB │ 1 │ Ready │ aeb58bcb-a5cc-40a1-aa3b-02014faa430d │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdb,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdc │ VMware Virtual_disk │ 25 GiB │ 4.4 MiB │ 1 │ Ready │ 6d59f5fe-28ce-4689-aaf8-7782cd366deb │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdc,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdd │ VMware Virtual_disk │ 25 GiB │ 4.4 MiB │ 1 │ Ready │ 04d14895-1424-460b-a0e1-ed1089fc27eb │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdd,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sde │ VMware Virtual_disk │ 25 GiB │ 4.4 MiB │ 1 │ Ready │ 1a8911b9-01ae-49ce-bcea-f9ca360cf95e │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sde,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
└──────────────────────────┴──────┴─────────────────────┴────────┴─────────┴─────────┴────────┴──────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
balamurugana commented
@zeph You would need to provide kubectl directpv list volumes --all -o wide output, not list drives.
zeph commented
% directpv.stage list volumes --all -o wide
┌──────────────────────────────────────────┬──────────┬──────────────────────────┬───────┬─────────────────────┬───────────────────┬─────────┬──────────────────────────────────────┐
│ VOLUME │ CAPACITY │ NODE │ DRIVE │ PODNAME │ PODNAMESPACE │ STATUS │ DRIVE ID │
├──────────────────────────────────────────┼──────────┼──────────────────────────┼───────┼─────────────────────┼───────────────────┼─────────┼──────────────────────────────────────┤
│ pvc-c5373554-83f1-42e5-8778-71d0e7857d47 │ 10 GiB │ mbbf-extern-k8s-worker-4 │ sdf │ stage-pool-0-2 │ tenant-stage │ Bounded │ ebd5313d-ef0a-493e-bd50-b7087cf49b53 │
│ pvc-fba3367b-675e-4b35-837b-334976a60d4f │ 10 GiB │ mbbf-extern-k8s-worker-4 │ sdg │ stage-pool-0-2 │ tenant-stage │ Bounded │ 9f437a30-2431-4549-997d-1eb126d7f63b │
│ pvc-208af0e4-866b-4096-abf8-816d41cf2717 │ 10 GiB │ mbbf-extern-k8s-worker-4 │ sdh │ stage-pool-0-2 │ tenant-stage │ Bounded │ 18388cbc-f291-40df-b4a4-5f21f658f38d │
│ pvc-7cc28727-c0c0-4579-857a-3f24c8e28d98 │ 10 GiB │ mbbf-extern-k8s-worker-4 │ sdi │ stage-pool-0-2 │ tenant-stage │ Bounded │ 84a81ed3-5e2e-45cc-a23f-aab6471e6e4e │
│ pvc-255f4d49-09a4-4dc6-b4c6-9c8da5dd139a │ 10 GiB │ mbbf-extern-k8s-worker-5 │ sdf │ stage-pool-0-1 │ tenant-stage │ Bounded │ 7a83fcba-e04a-453a-9e3e-5f0f50e415cb │
│ pvc-752e9a33-8ae9-4033-867b-00b75fb9264f │ 10 GiB │ mbbf-extern-k8s-worker-5 │ sdg │ stage-pool-0-1 │ tenant-stage │ Bounded │ a8adab8f-ef59-4f29-87e6-1a93c9920ee1 │
│ pvc-34bbe5c8-6d6b-4aa1-8fdf-10fe30dc3408 │ 10 GiB │ mbbf-extern-k8s-worker-5 │ sdh │ stage-pool-0-1 │ tenant-stage │ Bounded │ da347aad-c8d5-4af9-9ea4-aa5440b8679b │
│ pvc-ae8a3ff4-9030-4e31-af0b-e89d105ae654 │ 10 GiB │ mbbf-extern-k8s-worker-5 │ sdi │ stage-pool-0-1 │ tenant-stage │ Bounded │ ea9466da-3faa-4ba2-bfdc-1ba66393ed6a │
│ pvc-85b46d95-7de9-4613-9bf5-3312bf1f93f1 │ 10 GiB │ mbbf-extern-k8s-worker-6 │ sdf │ stage-pool-0-0 │ tenant-stage │ Bounded │ 7f9bd6b8-a318-4e22-8cc6-949b8d4d67d4 │
│ pvc-aa746f40-8dd4-44b7-9c06-da87d53c2832 │ 10 GiB │ mbbf-extern-k8s-worker-6 │ sdg │ stage-pool-0-0 │ tenant-stage │ Bounded │ 84db6f9c-4180-4eb1-9a45-b9304c5be44c │
│ pvc-0d4132ea-13cb-4f69-a5a9-983338022648 │ 10 GiB │ mbbf-extern-k8s-worker-6 │ sdh │ stage-pool-0-0 │ tenant-stage │ Bounded │ aeaed335-2b8b-4c08-a168-702de57c9f39 │
│ pvc-0815bc62-0abb-4a4e-a779-63232e723f13 │ 10 GiB │ mbbf-extern-k8s-worker-6 │ sdi │ stage-pool-0-0 │ tenant-stage │ Bounded │ 3a771c42-95fe-470a-9d53-204cee918dfc │
│ pvc-85b9c64c-1f17-423e-b2da-bcc56e874a87 │ 25 GiB │ mbbf-extern-k8s-worker-4 │ sdb │ stage-warm-pool-0-2 │ tenant-stage-warm │ Bounded │ c640b9bb-cdab-4582-8fd3-a12fe7ade9ea │
│ pvc-22186978-58ec-4beb-b9c3-2ddc6b0a8828 │ 25 GiB │ mbbf-extern-k8s-worker-4 │ sdc │ stage-warm-pool-0-2 │ tenant-stage-warm │ Bounded │ dc956005-c68c-454c-82bc-219bf0430e83 │
│ pvc-875127fe-298e-407a-bf58-fb1c388cfb5c │ 25 GiB │ mbbf-extern-k8s-worker-4 │ sdd │ stage-warm-pool-0-2 │ tenant-stage-warm │ Bounded │ 4306c5f5-22d2-457a-bddf-1015942c0dbe │
│ pvc-acc13fc3-7345-4784-b83b-ff98512ed225 │ 25 GiB │ mbbf-extern-k8s-worker-4 │ sde │ stage-warm-pool-0-2 │ tenant-stage-warm │ Bounded │ 7df9e86e-9176-4468-8300-d6fd66e69945 │
│ pvc-dbcc529f-3420-4e57-b562-85066eface80 │ 25 GiB │ mbbf-extern-k8s-worker-5 │ sdb │ stage-warm-pool-0-1 │ tenant-stage-warm │ Bounded │ 86c2a0fa-a5dd-4b08-a8ec-af0ce89a27a8 │
│ pvc-9c831643-3ce4-4c4e-a620-51c5620d027c │ 25 GiB │ mbbf-extern-k8s-worker-5 │ sdc │ stage-warm-pool-0-1 │ tenant-stage-warm │ Bounded │ cd571970-ff8b-47b0-a1f7-ee974b20efc4 │
│ pvc-bc7f89dd-ec2d-4a21-9a3a-5bb31249c7fd │ 25 GiB │ mbbf-extern-k8s-worker-5 │ sdd │ stage-warm-pool-0-1 │ tenant-stage-warm │ Bounded │ dce8daf3-4681-450e-bd71-eb9147825754 │
│ pvc-c2282426-fece-4623-96a8-67fbd963ad3e │ 25 GiB │ mbbf-extern-k8s-worker-5 │ sde │ stage-warm-pool-0-1 │ tenant-stage-warm │ Bounded │ 81c52ff1-2eed-4d4d-8563-1bc8ecf4446c │
│ pvc-bc621689-52a3-43d0-9919-d59a413aab88 │ 25 GiB │ mbbf-extern-k8s-worker-6 │ sdb │ stage-warm-pool-0-0 │ tenant-stage-warm │ Bounded │ aeb58bcb-a5cc-40a1-aa3b-02014faa430d │
│ pvc-30de2b4e-2178-4261-b274-8c11397ad790 │ 25 GiB │ mbbf-extern-k8s-worker-6 │ sdc │ stage-warm-pool-0-0 │ tenant-stage-warm │ Bounded │ 6d59f5fe-28ce-4689-aaf8-7782cd366deb │
│ pvc-9a827628-abc1-4ae5-8fc4-378646b7a629 │ 25 GiB │ mbbf-extern-k8s-worker-6 │ sdd │ stage-warm-pool-0-0 │ tenant-stage-warm │ Bounded │ 04d14895-1424-460b-a0e1-ed1089fc27eb │
│ pvc-f7fe817e-7530-4fb4-83ef-1f9b6bfe316a │ 25 GiB │ mbbf-extern-k8s-worker-6 │ sde │ stage-warm-pool-0-0 │ tenant-stage-warm │ Bounded │ 1a8911b9-01ae-49ce-bcea-f9ca360cf95e │
└──────────────────────────────────────────┴──────────┴──────────────────────────┴───────┴─────────────────────┴───────────────────┴─────────┴──────────────────────────────────────┘
p.s. this is the "expected to be" that I obtained by 1st provisioning the 25 GiB drives, then the 2nd tenancy with the 10GiB drives
zeph commented
shall I go back and allocate using the labels? as is, it will be doing the 10 GiB, even 2 times (since they fit) in the 25 GiB drives... completely ignoring the label constraints I did put on the StorageClass...
balamurugana commented
I do not see pvc-d09e2143-74ad-4a43-a4b8-568cf402b37f of PVC data0-stage-pool-0-0 you mentioned in the issue description in kubectl directpv list volumes --all -o wide output.
I am closing the issue with inappropriate information. However provide all relevant/complete/required information here to continue further.
zeph commented
gms@fluffy tools % directpv.stage list volumes --all -o wide
┌──────────────────────────────────────────┬──────────┬──────────────────────────┬───────┬────────────────┬──────────────┬─────────┬──────────────────────────────────────┐
│ VOLUME │ CAPACITY │ NODE │ DRIVE │ PODNAME │ PODNAMESPACE │ STATUS │ DRIVE ID │
├──────────────────────────────────────────┼──────────┼──────────────────────────┼───────┼────────────────┼──────────────┼─────────┼──────────────────────────────────────┤
│ pvc-28731122-9235-4402-9377-edf185afc494 │ 9.9 GiB │ mbbf-extern-k8s-worker-4 │ sdb │ stage-pool-0-2 │ tenant-stage │ Bounded │ c640b9bb-cdab-4582-8fd3-a12fe7ade9ea │
│ pvc-587320a9-f473-46ff-bf49-f1f595002b02 │ 9.9 GiB │ mbbf-extern-k8s-worker-4 │ sdc │ stage-pool-0-2 │ tenant-stage │ Bounded │ dc956005-c68c-454c-82bc-219bf0430e83 │
│ pvc-2aa03b6a-13c6-4c74-aa52-e08dce61df39 │ 9.9 GiB │ mbbf-extern-k8s-worker-4 │ sdd │ stage-pool-0-2 │ tenant-stage │ Bounded │ 4306c5f5-22d2-457a-bddf-1015942c0dbe │
│ pvc-94b48a95-9de5-43e1-bc15-8b7cbbe6d17b │ 9.9 GiB │ mbbf-extern-k8s-worker-4 │ sde │ stage-pool-0-2 │ tenant-stage │ Bounded │ 7df9e86e-9176-4468-8300-d6fd66e69945 │
│ pvc-a8161a3a-086b-49ab-9bdd-d0fc0f8bad6a │ 9.9 GiB │ mbbf-extern-k8s-worker-5 │ sdb │ stage-pool-0-1 │ tenant-stage │ Bounded │ 86c2a0fa-a5dd-4b08-a8ec-af0ce89a27a8 │
│ pvc-b9b55c4a-fc51-461a-8af3-453546dc42b0 │ 9.9 GiB │ mbbf-extern-k8s-worker-5 │ sdc │ stage-pool-0-1 │ tenant-stage │ Bounded │ cd571970-ff8b-47b0-a1f7-ee974b20efc4 │
│ pvc-a4ea79c7-cfa2-4394-8003-0ca770267daf │ 9.9 GiB │ mbbf-extern-k8s-worker-5 │ sdd │ stage-pool-0-1 │ tenant-stage │ Bounded │ dce8daf3-4681-450e-bd71-eb9147825754 │
│ pvc-e7ed5ac9-8707-447b-a571-efad47fba932 │ 9.9 GiB │ mbbf-extern-k8s-worker-5 │ sde │ stage-pool-0-1 │ tenant-stage │ Bounded │ 81c52ff1-2eed-4d4d-8563-1bc8ecf4446c │
│ pvc-b0fc48a2-1f96-4009-9ea5-39d5c50c5ff3 │ 9.9 GiB │ mbbf-extern-k8s-worker-6 │ sdb │ stage-pool-0-0 │ tenant-stage │ Bounded │ aeb58bcb-a5cc-40a1-aa3b-02014faa430d │
│ pvc-920f7a09-e5b9-4ef9-a639-c55ff1da6e06 │ 9.9 GiB │ mbbf-extern-k8s-worker-6 │ sdc │ stage-pool-0-0 │ tenant-stage │ Bounded │ 6d59f5fe-28ce-4689-aaf8-7782cd366deb │
│ pvc-66b3a4fc-fef7-4592-b1d6-52d150afc288 │ 9.9 GiB │ mbbf-extern-k8s-worker-6 │ sdd │ stage-pool-0-0 │ tenant-stage │ Bounded │ 04d14895-1424-460b-a0e1-ed1089fc27eb │
│ pvc-ccab942a-79d8-46cb-b8f3-3005c1f73e13 │ 9.9 GiB │ mbbf-extern-k8s-worker-6 │ sde │ stage-pool-0-0 │ tenant-stage │ Bounded │ 1a8911b9-01ae-49ce-bcea-f9ca360cf95e │
└──────────────────────────────────────────┴──────────┴──────────────────────────┴───────┴────────────────┴──────────────┴─────────┴──────────────────────────────────────┘
gms@fluffy tools % directpv.stage list drives --output wide --show-labels
┌──────────────────────────┬──────┬─────────────────────┬────────┬────────┬─────────┬────────┬──────────────────────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ NODE │ NAME │ MAKE │ SIZE │ FREE │ VOLUMES │ STATUS │ DRIVE ID │ LABELS │
├──────────────────────────┼──────┼─────────────────────┼────────┼────────┼─────────┼────────┼──────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ mbbf-extern-k8s-worker-4 │ sdf │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ ebd5313d-ef0a-493e-bd50-b7087cf49b53 │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdf,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdg │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ 9f437a30-2431-4549-997d-1eb126d7f63b │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdg,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdh │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ 18388cbc-f291-40df-b4a4-5f21f658f38d │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdh,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdi │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ 84a81ed3-5e2e-45cc-a23f-aab6471e6e4e │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdi,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdb │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ c640b9bb-cdab-4582-8fd3-a12fe7ade9ea │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdb,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdc │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ dc956005-c68c-454c-82bc-219bf0430e83 │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdc,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdd │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ 4306c5f5-22d2-457a-bddf-1015942c0dbe │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdd,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sde │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ 7df9e86e-9176-4468-8300-d6fd66e69945 │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sde,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdf │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ 7a83fcba-e04a-453a-9e3e-5f0f50e415cb │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdf,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdg │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ a8adab8f-ef59-4f29-87e6-1a93c9920ee1 │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdg,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdh │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ da347aad-c8d5-4af9-9ea4-aa5440b8679b │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdh,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdi │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ ea9466da-3faa-4ba2-bfdc-1ba66393ed6a │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdi,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdb │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ 86c2a0fa-a5dd-4b08-a8ec-af0ce89a27a8 │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdb,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdc │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ cd571970-ff8b-47b0-a1f7-ee974b20efc4 │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdc,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdd │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ dce8daf3-4681-450e-bd71-eb9147825754 │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdd,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sde │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ 81c52ff1-2eed-4d4d-8563-1bc8ecf4446c │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sde,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdf │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ 7f9bd6b8-a318-4e22-8cc6-949b8d4d67d4 │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdf,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdg │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ 84db6f9c-4180-4eb1-9a45-b9304c5be44c │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdg,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdh │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ aeaed335-2b8b-4c08-a168-702de57c9f39 │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdh,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdi │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ 3a771c42-95fe-470a-9d53-204cee918dfc │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdi,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdb │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ aeb58bcb-a5cc-40a1-aa3b-02014faa430d │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdb,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdc │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ 6d59f5fe-28ce-4689-aaf8-7782cd366deb │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdc,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdd │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ 04d14895-1424-460b-a0e1-ed1089fc27eb │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdd,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sde │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ 1a8911b9-01ae-49ce-bcea-f9ca360cf95e │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sde,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
└──────────────────────────┴──────┴─────────────────────┴────────┴────────┴─────────┴────────┴──────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
@balamurugana please reopen, I hope the above is enough information to improve the product... I'm happy to open support ticket and follow up even if I did find a workaround suitable for myself... but I can't keep and environment dedicated to it... I need to move forward in my work. The issue is there (unless you prove me wrong, happy to have done/mistaken something), and closing with inappropriate information kind of seems inappropriate
balamurugana commented
Please share kubectl get pvc --all -o yaml output.
zeph commented
[deleted, recreated... here u go]
% directpv.stage list volumes --all -o wide
┌──────────────────────────────────────────┬──────────┬──────────────────────────┬───────┬────────────────┬──────────────┬─────────┬──────────────────────────────────────┐
│ VOLUME │ CAPACITY │ NODE │ DRIVE │ PODNAME │ PODNAMESPACE │ STATUS │ DRIVE ID │
├──────────────────────────────────────────┼──────────┼──────────────────────────┼───────┼────────────────┼──────────────┼─────────┼──────────────────────────────────────┤
│ pvc-925032db-2391-4322-ad47-4eec78f42e7b │ 10 GiB │ mbbf-extern-k8s-worker-4 │ sdb │ stage-pool-0-2 │ tenant-stage │ Bounded │ c640b9bb-cdab-4582-8fd3-a12fe7ade9ea │
│ pvc-a4641a83-8596-4259-9d17-17b0d923b4cd │ 10 GiB │ mbbf-extern-k8s-worker-4 │ sdc │ stage-pool-0-2 │ tenant-stage │ Bounded │ dc956005-c68c-454c-82bc-219bf0430e83 │
│ pvc-baab636b-61ca-4492-8dc8-65720270d188 │ 10 GiB │ mbbf-extern-k8s-worker-4 │ sdd │ stage-pool-0-2 │ tenant-stage │ Bounded │ 4306c5f5-22d2-457a-bddf-1015942c0dbe │
│ pvc-973554cf-30e6-4343-bbfc-42c879f134db │ 10 GiB │ mbbf-extern-k8s-worker-4 │ sde │ stage-pool-0-2 │ tenant-stage │ Bounded │ 7df9e86e-9176-4468-8300-d6fd66e69945 │
│ pvc-c13beaa3-ee7f-4b5f-92ad-e018df6eb318 │ 10 GiB │ mbbf-extern-k8s-worker-5 │ sdb │ stage-pool-0-1 │ tenant-stage │ Bounded │ 86c2a0fa-a5dd-4b08-a8ec-af0ce89a27a8 │
│ pvc-fdd6a8af-d8b4-4c6a-8c84-52d93ec8f20a │ 10 GiB │ mbbf-extern-k8s-worker-5 │ sdc │ stage-pool-0-1 │ tenant-stage │ Bounded │ cd571970-ff8b-47b0-a1f7-ee974b20efc4 │
│ pvc-8b4b450e-c1a3-4c89-91b9-82967f5de394 │ 10 GiB │ mbbf-extern-k8s-worker-5 │ sdd │ stage-pool-0-1 │ tenant-stage │ Bounded │ dce8daf3-4681-450e-bd71-eb9147825754 │
│ pvc-08b4f77c-426a-49dc-92d8-1874bad88ccb │ 10 GiB │ mbbf-extern-k8s-worker-5 │ sde │ stage-pool-0-1 │ tenant-stage │ Bounded │ 81c52ff1-2eed-4d4d-8563-1bc8ecf4446c │
│ pvc-16066e37-b2cb-4c4e-aae3-2b87583c5251 │ 10 GiB │ mbbf-extern-k8s-worker-6 │ sdb │ stage-pool-0-0 │ tenant-stage │ Bounded │ aeb58bcb-a5cc-40a1-aa3b-02014faa430d │
│ pvc-dd2d52e7-f9eb-4eb6-93a6-4ff0187c8ce3 │ 10 GiB │ mbbf-extern-k8s-worker-6 │ sdc │ stage-pool-0-0 │ tenant-stage │ Bounded │ 6d59f5fe-28ce-4689-aaf8-7782cd366deb │
│ pvc-ffc7ce0c-3170-45fa-a2a3-86fe228a74ff │ 10 GiB │ mbbf-extern-k8s-worker-6 │ sdd │ stage-pool-0-0 │ tenant-stage │ Bounded │ 04d14895-1424-460b-a0e1-ed1089fc27eb │
│ pvc-2c424bce-9809-4bc1-b3e2-5a5715b45931 │ 10 GiB │ mbbf-extern-k8s-worker-6 │ sde │ stage-pool-0-0 │ tenant-stage │ Bounded │ 1a8911b9-01ae-49ce-bcea-f9ca360cf95e │
└──────────────────────────────────────────┴──────────┴──────────────────────────┴───────┴────────────────┴──────────────┴─────────┴──────────────────────────────────────┘
% directpv.stage list drives --output wide --show-labels
┌──────────────────────────┬──────┬─────────────────────┬────────┬────────┬─────────┬────────┬──────────────────────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ NODE │ NAME │ MAKE │ SIZE │ FREE │ VOLUMES │ STATUS │ DRIVE ID │ LABELS │
├──────────────────────────┼──────┼─────────────────────┼────────┼────────┼─────────┼────────┼──────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ mbbf-extern-k8s-worker-4 │ sdf │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ ebd5313d-ef0a-493e-bd50-b7087cf49b53 │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdf,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdg │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ 9f437a30-2431-4549-997d-1eb126d7f63b │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdg,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdh │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ 18388cbc-f291-40df-b4a4-5f21f658f38d │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdh,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdi │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ 84a81ed3-5e2e-45cc-a23f-aab6471e6e4e │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdi,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdb │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ c640b9bb-cdab-4582-8fd3-a12fe7ade9ea │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdb,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdc │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ dc956005-c68c-454c-82bc-219bf0430e83 │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdc,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdd │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ 4306c5f5-22d2-457a-bddf-1015942c0dbe │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdd,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sde │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ 7df9e86e-9176-4468-8300-d6fd66e69945 │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sde,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdf │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ 7a83fcba-e04a-453a-9e3e-5f0f50e415cb │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdf,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdg │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ a8adab8f-ef59-4f29-87e6-1a93c9920ee1 │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdg,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdh │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ da347aad-c8d5-4af9-9ea4-aa5440b8679b │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdh,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdi │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ ea9466da-3faa-4ba2-bfdc-1ba66393ed6a │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdi,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdb │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ 86c2a0fa-a5dd-4b08-a8ec-af0ce89a27a8 │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdb,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdc │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ cd571970-ff8b-47b0-a1f7-ee974b20efc4 │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdc,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdd │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ dce8daf3-4681-450e-bd71-eb9147825754 │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdd,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sde │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ 81c52ff1-2eed-4d4d-8563-1bc8ecf4446c │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sde,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdf │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ 7f9bd6b8-a318-4e22-8cc6-949b8d4d67d4 │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdf,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdg │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ 84db6f9c-4180-4eb1-9a45-b9304c5be44c │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdg,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdh │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ aeaed335-2b8b-4c08-a168-702de57c9f39 │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdh,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdi │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ 3a771c42-95fe-470a-9d53-204cee918dfc │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdi,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdb │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ aeb58bcb-a5cc-40a1-aa3b-02014faa430d │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdb,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdc │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ 6d59f5fe-28ce-4689-aaf8-7782cd366deb │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdc,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdd │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ 04d14895-1424-460b-a0e1-ed1089fc27eb │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdd,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sde │ VMware Virtual_disk │ 25 GiB │ 15 GiB │ 1 │ Ready │ 1a8911b9-01ae-49ce-bcea-f9ca360cf95e │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sde,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
└──────────────────────────┴──────┴─────────────────────┴────────┴────────┴─────────┴────────┴──────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
% kubectl get pvc -o yaml
apiVersion: v1
items:
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: directpv-min-io
volume.kubernetes.io/selected-node: mbbf-extern-k8s-worker-6
volume.kubernetes.io/storage-provisioner: directpv-min-io
creationTimestamp: "2023-09-29T15:00:52Z"
finalizers:
- kubernetes.io/pvc-protection
labels:
v1.min.io/console: stage-console
v1.min.io/pool: pool-0
v1.min.io/tenant: stage
name: data0-stage-pool-0-0
namespace: tenant-stage
resourceVersion: "211277743"
uid: dd2d52e7-f9eb-4eb6-93a6-4ff0187c8ce3
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "10719522542"
storageClassName: ssd-tier-storage
volumeMode: Filesystem
volumeName: pvc-dd2d52e7-f9eb-4eb6-93a6-4ff0187c8ce3
status:
accessModes:
- ReadWriteOnce
capacity:
storage: "10719522542"
phase: Bound
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: directpv-min-io
volume.kubernetes.io/selected-node: mbbf-extern-k8s-worker-5
volume.kubernetes.io/storage-provisioner: directpv-min-io
creationTimestamp: "2023-09-29T15:00:52Z"
finalizers:
- kubernetes.io/pvc-protection
labels:
v1.min.io/console: stage-console
v1.min.io/pool: pool-0
v1.min.io/tenant: stage
name: data0-stage-pool-0-1
namespace: tenant-stage
resourceVersion: "211277802"
uid: fdd6a8af-d8b4-4c6a-8c84-52d93ec8f20a
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "10719522542"
storageClassName: ssd-tier-storage
volumeMode: Filesystem
volumeName: pvc-fdd6a8af-d8b4-4c6a-8c84-52d93ec8f20a
status:
accessModes:
- ReadWriteOnce
capacity:
storage: "10719522542"
phase: Bound
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: directpv-min-io
volume.kubernetes.io/selected-node: mbbf-extern-k8s-worker-4
volume.kubernetes.io/storage-provisioner: directpv-min-io
creationTimestamp: "2023-09-29T15:00:52Z"
finalizers:
- kubernetes.io/pvc-protection
labels:
v1.min.io/console: stage-console
v1.min.io/pool: pool-0
v1.min.io/tenant: stage
name: data0-stage-pool-0-2
namespace: tenant-stage
resourceVersion: "211277837"
uid: baab636b-61ca-4492-8dc8-65720270d188
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "10719522542"
storageClassName: ssd-tier-storage
volumeMode: Filesystem
volumeName: pvc-baab636b-61ca-4492-8dc8-65720270d188
status:
accessModes:
- ReadWriteOnce
capacity:
storage: "10719522542"
phase: Bound
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: directpv-min-io
volume.kubernetes.io/selected-node: mbbf-extern-k8s-worker-6
volume.kubernetes.io/storage-provisioner: directpv-min-io
creationTimestamp: "2023-09-29T15:00:52Z"
finalizers:
- kubernetes.io/pvc-protection
labels:
v1.min.io/console: stage-console
v1.min.io/pool: pool-0
v1.min.io/tenant: stage
name: data1-stage-pool-0-0
namespace: tenant-stage
resourceVersion: "211277729"
uid: 16066e37-b2cb-4c4e-aae3-2b87583c5251
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "10719522542"
storageClassName: ssd-tier-storage
volumeMode: Filesystem
volumeName: pvc-16066e37-b2cb-4c4e-aae3-2b87583c5251
status:
accessModes:
- ReadWriteOnce
capacity:
storage: "10719522542"
phase: Bound
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: directpv-min-io
volume.kubernetes.io/selected-node: mbbf-extern-k8s-worker-5
volume.kubernetes.io/storage-provisioner: directpv-min-io
creationTimestamp: "2023-09-29T15:00:52Z"
finalizers:
- kubernetes.io/pvc-protection
labels:
v1.min.io/console: stage-console
v1.min.io/pool: pool-0
v1.min.io/tenant: stage
name: data1-stage-pool-0-1
namespace: tenant-stage
resourceVersion: "211277816"
uid: 08b4f77c-426a-49dc-92d8-1874bad88ccb
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "10719522542"
storageClassName: ssd-tier-storage
volumeMode: Filesystem
volumeName: pvc-08b4f77c-426a-49dc-92d8-1874bad88ccb
status:
accessModes:
- ReadWriteOnce
capacity:
storage: "10719522542"
phase: Bound
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: directpv-min-io
volume.kubernetes.io/selected-node: mbbf-extern-k8s-worker-4
volume.kubernetes.io/storage-provisioner: directpv-min-io
creationTimestamp: "2023-09-29T15:00:52Z"
finalizers:
- kubernetes.io/pvc-protection
labels:
v1.min.io/console: stage-console
v1.min.io/pool: pool-0
v1.min.io/tenant: stage
name: data1-stage-pool-0-2
namespace: tenant-stage
resourceVersion: "211277850"
uid: 925032db-2391-4322-ad47-4eec78f42e7b
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "10719522542"
storageClassName: ssd-tier-storage
volumeMode: Filesystem
volumeName: pvc-925032db-2391-4322-ad47-4eec78f42e7b
status:
accessModes:
- ReadWriteOnce
capacity:
storage: "10719522542"
phase: Bound
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: directpv-min-io
volume.kubernetes.io/selected-node: mbbf-extern-k8s-worker-6
volume.kubernetes.io/storage-provisioner: directpv-min-io
creationTimestamp: "2023-09-29T15:00:52Z"
finalizers:
- kubernetes.io/pvc-protection
labels:
v1.min.io/console: stage-console
v1.min.io/pool: pool-0
v1.min.io/tenant: stage
name: data2-stage-pool-0-0
namespace: tenant-stage
resourceVersion: "211277753"
uid: 2c424bce-9809-4bc1-b3e2-5a5715b45931
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "10719522542"
storageClassName: ssd-tier-storage
volumeMode: Filesystem
volumeName: pvc-2c424bce-9809-4bc1-b3e2-5a5715b45931
status:
accessModes:
- ReadWriteOnce
capacity:
storage: "10719522542"
phase: Bound
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: directpv-min-io
volume.kubernetes.io/selected-node: mbbf-extern-k8s-worker-5
volume.kubernetes.io/storage-provisioner: directpv-min-io
creationTimestamp: "2023-09-29T15:00:52Z"
finalizers:
- kubernetes.io/pvc-protection
labels:
v1.min.io/console: stage-console
v1.min.io/pool: pool-0
v1.min.io/tenant: stage
name: data2-stage-pool-0-1
namespace: tenant-stage
resourceVersion: "211277825"
uid: 8b4b450e-c1a3-4c89-91b9-82967f5de394
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "10719522542"
storageClassName: ssd-tier-storage
volumeMode: Filesystem
volumeName: pvc-8b4b450e-c1a3-4c89-91b9-82967f5de394
status:
accessModes:
- ReadWriteOnce
capacity:
storage: "10719522542"
phase: Bound
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: directpv-min-io
volume.kubernetes.io/selected-node: mbbf-extern-k8s-worker-4
volume.kubernetes.io/storage-provisioner: directpv-min-io
creationTimestamp: "2023-09-29T15:00:52Z"
finalizers:
- kubernetes.io/pvc-protection
labels:
v1.min.io/console: stage-console
v1.min.io/pool: pool-0
v1.min.io/tenant: stage
name: data2-stage-pool-0-2
namespace: tenant-stage
resourceVersion: "211277857"
uid: 973554cf-30e6-4343-bbfc-42c879f134db
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "10719522542"
storageClassName: ssd-tier-storage
volumeMode: Filesystem
volumeName: pvc-973554cf-30e6-4343-bbfc-42c879f134db
status:
accessModes:
- ReadWriteOnce
capacity:
storage: "10719522542"
phase: Bound
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: directpv-min-io
volume.kubernetes.io/selected-node: mbbf-extern-k8s-worker-6
volume.kubernetes.io/storage-provisioner: directpv-min-io
creationTimestamp: "2023-09-29T15:00:52Z"
finalizers:
- kubernetes.io/pvc-protection
labels:
v1.min.io/console: stage-console
v1.min.io/pool: pool-0
v1.min.io/tenant: stage
name: data3-stage-pool-0-0
namespace: tenant-stage
resourceVersion: "211277774"
uid: ffc7ce0c-3170-45fa-a2a3-86fe228a74ff
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "10719522542"
storageClassName: ssd-tier-storage
volumeMode: Filesystem
volumeName: pvc-ffc7ce0c-3170-45fa-a2a3-86fe228a74ff
status:
accessModes:
- ReadWriteOnce
capacity:
storage: "10719522542"
phase: Bound
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: directpv-min-io
volume.kubernetes.io/selected-node: mbbf-extern-k8s-worker-5
volume.kubernetes.io/storage-provisioner: directpv-min-io
creationTimestamp: "2023-09-29T15:00:52Z"
finalizers:
- kubernetes.io/pvc-protection
labels:
v1.min.io/console: stage-console
v1.min.io/pool: pool-0
v1.min.io/tenant: stage
name: data3-stage-pool-0-1
namespace: tenant-stage
resourceVersion: "211277843"
uid: c13beaa3-ee7f-4b5f-92ad-e018df6eb318
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "10719522542"
storageClassName: ssd-tier-storage
volumeMode: Filesystem
volumeName: pvc-c13beaa3-ee7f-4b5f-92ad-e018df6eb318
status:
accessModes:
- ReadWriteOnce
capacity:
storage: "10719522542"
phase: Bound
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: directpv-min-io
volume.kubernetes.io/selected-node: mbbf-extern-k8s-worker-4
volume.kubernetes.io/storage-provisioner: directpv-min-io
creationTimestamp: "2023-09-29T15:00:52Z"
finalizers:
- kubernetes.io/pvc-protection
labels:
v1.min.io/console: stage-console
v1.min.io/pool: pool-0
v1.min.io/tenant: stage
name: data3-stage-pool-0-2
namespace: tenant-stage
resourceVersion: "211277861"
uid: a4641a83-8596-4259-9d17-17b0d923b4cd
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "10719522542"
storageClassName: ssd-tier-storage
volumeMode: Filesystem
volumeName: pvc-a4641a83-8596-4259-9d17-17b0d923b4cd
status:
accessModes:
- ReadWriteOnce
capacity:
storage: "10719522542"
phase: Bound
kind: List
metadata:
resourceVersion: ""
I skipped the --all (as it won't be compatible with the verb get for kubectl cli, unless u meant a -A)
zeph commented
% kubectl get StorageClass -o yaml|grep name:
name: directpv-min-io
name: hdd-tier-storage
name: nfs-ssd
name: ssd-tier-storage
% kubectl get StorageClass -o yaml ssd-tier-storage
allowVolumeExpansion: true
allowedTopologies:
- matchLabelExpressions:
- key: directpv.min.io/identity
values:
- directpv-min-io
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"allowVolumeExpansion":true,"allowedTopologies":[{"matchLabelExpressions":[{"key":"directpv.min.io/identity","values":["directpv-min-io"]}]}],"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"finalizers":["foregroundDeletion"],"labels":{"application-name":"directpv.min.io","application-type":"CSIDriver","directpv.min.io/created-by":"kubectl-directpv","directpv.min.io/version":"v1beta1"},"name":"ssd-tier-storage"},"parameters":{"disktype":"ssd","fstype":"xfs"},"provisioner":"directpv-min-io","reclaimPolicy":"Delete","volumeBindingMode":"WaitForFirstConsumer"}
creationTimestamp: "2023-09-27T13:01:21Z"
finalizers:
- foregroundDeletion
labels:
application-name: directpv.min.io
application-type: CSIDriver
directpv.min.io/created-by: kubectl-directpv
directpv.min.io/version: v1beta1
name: ssd-tier-storage
resourceVersion: "210201929"
uid: 054c3890-bfc0-4ba3-a56a-b75b7abf4cc1
parameters:
disktype: ssd
fstype: xfs
provisioner: directpv-min-io
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumerbalamurugana commented
The output doesn't say anything. My guess is previously created volumes with storage class hdd-tier-storage are reused because of same request is made second time. The point is if you change drive labels it won't affect previously created volumes and if you request same PVC (e.g. data3-stage-pool-0-2) it will serve previously created volume.
When you delete your tenants, you have to delete respective volumes as well for this experiment. The right way to test the behavior is using below YAML with appropriate storage class
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: sleep-pvc
spec:
volumeMode: Filesystem
storageClassName: directpv-min-io
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Mi
---
apiVersion: v1
kind: Pod
metadata:
name: sleep-pod
spec:
volumes:
- name: sleep-volume
persistentVolumeClaim:
claimName: sleep-pvc
containers:
- name: sleep-container
image: example.org/test/sleep:v0.0.1
volumeMounts:
- mountPath: "/mnt"
name: sleep-volume
---Note that if you change storageClassName second time, either delete sleep-pvc PVC or use different PVC name.
zeph commented
uhmm... the thing is... I have no control on those PVCs... they are deleted with the tenant, and recreated the moment I do create a tenant... it is all done by the operator itself... is there anything keeping the metadata between one erase and the other? the PVCs are controlled by the STS of the Tenant. As far as I tested so far, if I delete the tenant on the operator, all hierarchies are gone... and if I create the 25GiB tenant (hdd), the directpv has no other choice than to utilize the bigger drives, afterwards I can then create the 10GiB tenant (ssd) and things go smoothly... seems that the label selecting tech is being completely ignored here
I can make an attempt with the sleep-pod sample above, brb
zeph commented
% directpv.stage list volumes --all -o wide
┌──────────────────────────────────────────┬──────────┬──────────────────────────┬───────┬───────────┬──────────────┬─────────┬──────────────────────────────────────┐
│ VOLUME │ CAPACITY │ NODE │ DRIVE │ PODNAME │ PODNAMESPACE │ STATUS │ DRIVE ID │
├──────────────────────────────────────────┼──────────┼──────────────────────────┼───────┼───────────┼──────────────┼─────────┼──────────────────────────────────────┤
│ pvc-76aa1b5e-fa05-4741-96c0-3b3f72721fdc │ 8.0 MiB │ mbbf-extern-k8s-worker-6 │ sdd │ sleep-pod │ tenant-stage │ Bounded │ 04d14895-1424-460b-a0e1-ed1089fc27eb │
└──────────────────────────────────────────┴──────────┴──────────────────────────┴───────┴───────────┴──────────────┴─────────┴──────────────────────────────────────┘
gms@fluffy MaxPlanck % directpv.stage list drives --show-labels
┌──────────────────────────┬──────┬─────────────────────┬────────┬────────┬─────────┬────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ NODE │ NAME │ MAKE │ SIZE │ FREE │ VOLUMES │ STATUS │ LABELS │
├──────────────────────────┼──────┼─────────────────────┼────────┼────────┼─────────┼────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ mbbf-extern-k8s-worker-4 │ sdf │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdf,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdg │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdg,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdh │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdh,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdi │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdi,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdb │ VMware Virtual_disk │ 25 GiB │ 25 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdb,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdc │ VMware Virtual_disk │ 25 GiB │ 25 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdc,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdd │ VMware Virtual_disk │ 25 GiB │ 25 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdd,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sde │ VMware Virtual_disk │ 25 GiB │ 25 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sde,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdf │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdf,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdg │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdg,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdh │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdh,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdi │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdi,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdb │ VMware Virtual_disk │ 25 GiB │ 25 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdb,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdc │ VMware Virtual_disk │ 25 GiB │ 25 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdc,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdd │ VMware Virtual_disk │ 25 GiB │ 25 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdd,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sde │ VMware Virtual_disk │ 25 GiB │ 25 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sde,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdf │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdf,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdg │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdg,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdh │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdh,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdi │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdi,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdb │ VMware Virtual_disk │ 25 GiB │ 25 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdb,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdc │ VMware Virtual_disk │ 25 GiB │ 25 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdc,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdd │ VMware Virtual_disk │ 25 GiB │ 25 GiB │ 1 │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdd,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sde │ VMware Virtual_disk │ 25 GiB │ 25 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sde,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
└──────────────────────────┴──────┴─────────────────────┴────────┴────────┴─────────┴────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
and I used an ssd-tier-storage
...completely ignored as it ended up on mbbf-extern-k8s-worker-6 │ sdd | disktype=hdd
% cat sleep-hdd.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: sleep-pvc
spec:
volumeMode: Filesystem
storageClassName: ssd-tier-storage
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Mi
---
apiVersion: v1
kind: Pod
metadata:
name: sleep-pod
spec:
volumes:
- name: sleep-volume
persistentVolumeClaim:
claimName: sleep-pvc
containers:
- name: sleep-container
image: example.org/test/sleep:v0.0.1
volumeMounts:
- mountPath: "/mnt"
name: sleep-volume
---
zeph commented
I also changed the pvc name as u suggested... same (different drive, same machine)
% directpv.stage list drives --show-labels
┌──────────────────────────┬──────┬─────────────────────┬────────┬────────┬─────────┬────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ NODE │ NAME │ MAKE │ SIZE │ FREE │ VOLUMES │ STATUS │ LABELS │
├──────────────────────────┼──────┼─────────────────────┼────────┼────────┼─────────┼────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ mbbf-extern-k8s-worker-4 │ sdf │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdf,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdg │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdg,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdh │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdh,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdi │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdi,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdb │ VMware Virtual_disk │ 25 GiB │ 25 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdb,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdc │ VMware Virtual_disk │ 25 GiB │ 25 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdc,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sdd │ VMware Virtual_disk │ 25 GiB │ 25 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdd,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-4 │ sde │ VMware Virtual_disk │ 25 GiB │ 25 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sde,node=mbbf-extern-k8s-worker-4,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdf │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdf,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdg │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdg,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdh │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdh,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdi │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdi,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdb │ VMware Virtual_disk │ 25 GiB │ 25 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdb,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdc │ VMware Virtual_disk │ 25 GiB │ 25 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdc,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sdd │ VMware Virtual_disk │ 25 GiB │ 25 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdd,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-5 │ sde │ VMware Virtual_disk │ 25 GiB │ 25 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sde,node=mbbf-extern-k8s-worker-5,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdf │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdf,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdg │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdg,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdh │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdh,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdi │ VMware Virtual_disk │ 10 GiB │ 10 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=ssd,drive-name=sdi,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdb │ VMware Virtual_disk │ 25 GiB │ 25 GiB │ 1 │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdb,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdc │ VMware Virtual_disk │ 25 GiB │ 25 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdc,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sdd │ VMware Virtual_disk │ 25 GiB │ 25 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sdd,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
│ mbbf-extern-k8s-worker-6 │ sde │ VMware Virtual_disk │ 25 GiB │ 25 GiB │ - │ Ready │ access-tier=Default,created-by=directpv-driver,disktype=hdd,drive-name=sde,node=mbbf-extern-k8s-worker-6,version=v1beta1 │
└──────────────────────────┴──────┴─────────────────────┴────────┴────────┴─────────┴────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
gms@fluffy MaxPlanck % directpv.stage list volumes --all -o wide
┌──────────────────────────────────────────┬──────────┬──────────────────────────┬───────┬───────────┬──────────────┬─────────┬──────────────────────────────────────┐
│ VOLUME │ CAPACITY │ NODE │ DRIVE │ PODNAME │ PODNAMESPACE │ STATUS │ DRIVE ID │
├──────────────────────────────────────────┼──────────┼──────────────────────────┼───────┼───────────┼──────────────┼─────────┼──────────────────────────────────────┤
│ pvc-9b9309f9-9401-41ce-9ad8-420be620266d │ 8.0 MiB │ mbbf-extern-k8s-worker-6 │ sdb │ sleep-pod │ tenant-stage │ Bounded │ aeb58bcb-a5cc-40a1-aa3b-02014faa430d │
└──────────────────────────────────────────┴──────────┴──────────────────────────┴───────┴───────────┴──────────────┴─────────┴──────────────────────────────────────┘
zeph commented
I'm looking at the code... it doesn't look like it is considering all
keyNames (of Labels) that a user might decide... or am I missing something?
all examples are related to a keyName: tier
I start to feel that an arbitrary key like disktype is not handled
zeph commented
got it 🤦 ...label keyNames have to be prefixed with directpv.min.io/ or it won't work
% ./create-storage-class.sh prefixed-ssd-tier-storage 'directpv.min.io/disktype: ssd'
balamurugana commented
I strongly recommend to go through DirectPV documentation
zeph commented
the documentation states (not explicitly pointing that out) it actually, my bad I would say
https://github.com/minio/directpv/blob/master/docs/volume-scheduling.md#customizing-drive-selection
zeph commented
@balamurugana I did... it is somehow given for granted by the example provided
zeph commented
thanks for your support, at the end also this ticket will be documentation for the next poor fella xD
balamurugana commented
Feel free to send a PR
zeph commented
done :)
zeph commented
tnx for this failsafe @balamurugana #863