jparklab/synology-csi

Failed to mount existing volumes

jparklab opened this issue · 5 comments

jparklab commented

Since GetDiskFormat function in k8s.io/util is not compatible with blkid binary of alphine linux(
kubernetes/utils#122). synology-csi-node plugin fails to mount existing volumes when a pod is recreated.

One workaround would be to use ubuntu image instead of alphine image until k8s.io/util is updated.

joelrebel commented

@jparklab I'll share my findings here, its possible I'm incorrect,

The alpine linux blkid executable is provided by busybox which is a fairly limited executable,
https://github.com/brgl/busybox/blob/master/util-linux/blkid.c

The util-linux package in the alpine linux is fairly new, although it only provides the shared library object for blkid

util-linux-2.34-r1 license:
GPL-2.0 GPL-2.0-or-later LGPL-2.0-or-later BSD Public-Domain

/ # apk info -a util-linux | grep blkid
so:libblkid.so.1
jparklab commented

@joelrebel Thanks. I didn't know alpine's util-linux package does not include the blkid binary.
blkid included in the busybox is not compatible with the official blkid .
Leo(https://gitlab.alpinelinux.org/alpine/aports/issues/11404#note_81719) suggested installing blkid package, and it produces output as expected.

  • 👍1
ahuffman commented

Hit this today as well:

Error: "MountVolume.SetUp failed for volume \"pvc-d9b10eda-9055-4a8d-be3a-481a03928b91\" (UniqueName: \"kubernetes.io/csi/csi.synology.com^1.1\") pod \"pihole-7859d799b6-c877s\" (UID: \"7aa58ebb-2c7f-4c6c-85b5-a568e33ff536\") : rpc error: code = Internal desc = Failed to mount /dev/disk/by-path/ip-192.168.1.65:3260-iscsi-iqn.2000-01.com.synology:kube-csi-pvc-d9b10eda-9055-4a8d-be3a-481a03928b91-lun-1 to /var/lib/kubelet/pods/7aa58ebb-2c7f-4c6c-85b5-a568e33ff536/volumes/kubernetes.io~csi/pvc-d9b10eda-9055-4a8d-be3a-481a03928b91/mount(fstype: ext4, options: [rw]): failed to get disk format of disk /dev/disk/by-path/ip-192.168.1.65:3260-iscsi-iqn.2000-01.com.synology:kube-csi-pvc-d9b10eda-9055-4a8d-be3a-481a03928b91-lun-1: blkid returns invalid output: /dev/disk/by-path/ip-192.168.1.65:3260-iscsi-iqn.2000-01.com.synology:kube-csi-pvc-d9b10eda-9055-4a8d-be3a-481a03928b91-lun-1: UUID=\"d7848198-c1c1-451b-9a4d-0291fe0243b4\" TYPE=\"ext4\"\n"
benwbooth commented

I had the same issue with blkid errors. Using @ahuffman's Dockerfile fixed the problem for me.

benwbooth commented

Any idea when this fix will be integrated? It's a one line change.

Thanks!