Failure in mounting PVC for Azure Blob Storage when scaling out to more than 1 pods in AKS
Closed this issue · 9 comments
Which version of the blobfuse was used?
blobfuse 1.2.3
Which OS (please include version) are you using?
NAME="Ubuntu"
VERSION="16.04.6 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.6 LTS"
VERSION_ID="16.04"
What problem was encountered?
I am using blobfuse to mount Azure Blob container on a Pod in our AKS cluster. For this i have create Presistant Volume and Presistant Volume Claim with all correct details as per the documentation. The problem i am facing while trying to scale out the deployment for anything more than 1 pod. Please note that the first pod (or replica) starts successfully hence it confirms that the configuration details for container are all correct however the scaleout operation fails to add any more replicas on the AKS cluster. Below the error that i am seeing on all newly created pods that fail to start:
MountVolume.SetUp failed for volume "bv-blobfuse-flexvol" : invalid character 'E' looking for beginning of value
One thing i am not sure about is if we are allowed to use same PVC to be share across multiple pods (may or may not be for same deployment)?
If its not possible, is there any workaround available where we can use same storage container to be mounted on two different pods at same time?
Have you found a mitigation/solution?
No
By default, blobfuse logs errors to syslog. If this is relevant, is there anything in the syslog that might be helpful?
Below are the logs appearing in the log file for blobfuse-driver.log
`Mon Jun 29 00:35:27 UTC 2020 EXEC: mkdir -p /var/lib/kubelet/pods/436241c4-6daa-4c93-903d-55c7b5da0787/volumes/azure~blobfuse/bv-blobfuse-flexvol
Mon Jun 29 00:35:27 UTC 2020 INF: AZURE_STORAGE_ACCESS_KEY is set
Mon Jun 29 00:35:27 UTC 2020 INF: export storage account - export AZURE_STORAGE_ACCOUNT=bvdevstr
Mon Jun 29 00:35:27 UTC 2020 EXEC: blobfuse /var/lib/kubelet/pods/436241c4-6daa-4c93-903d-55c7b5da0787/volumes/azure~blobfuse/test-blobfuse-flexvol --container-name=testisetstore --tmp-path=/tmp/blobfuse -o allow_other --file-cache-timeout-in-seconds=0 --log-level=LOG_DEBUG
Mon Jun 29 00:35:27 UTC 2020 ERROR: { "status": "Failure", "message": "Failed to mount device /dev/ at /var/lib/kubelet/pods/436241c4-6daa-4c93-903d-55c7b5da0787/volumes/azureblobfuse/test-blobfuse-flexvol, accountname:testdevstr, error log:Mon Jun 29 00:35:27 UTC 2020 EXEC: blobfuse /var/lib/kubelet/pods/436241c4-6daa-4c93-903d-55c7b5da0787/volumes/azureblobfuse/test-blobfuse-flexvol --container-name=testisetstore --tmp-path=/tmp/blobfuse -o allow_other --file-cache-timeout-in-seconds=0 --log-level=LOG_DEBUG" }`
If relevant, please share your mount command.
I tried to follow steps documented at Azure/kubernetes-volume-drivers#58 however its still not fixed my issue.
This appears to be a persistant volume expansion issue. what is the fstype ? is it ext4 or XFS?
If this is the case you need to contact the https://github.com/Azure/kubernetes-volume-drivers. They can check the enablle expansion options or driver versions etc.
But, I also see the following issue below.
The output for /var/log/blobfuse-driver.log above shows 2 different names for flexvolume bv-blobfuse-flexvol and then errors on test-blobfuse-flexvol
Do the pods belong to the same node. If not could you mount blobfuse directly on the node of the failing pod?
The fstype is ext4 but please see the details below:
azureuser@aks-nodepool1-41959266-vmss000000:$ mount | grep "/dev"
udev on /dev type devtmpfs (rw,nosuid,relatime,size=28835576k,nr_inodes=7208894,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
/dev/sda1 on / type ext4 (rw,relatime,discard,data=ordered)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
mqueue on /dev/mqueue type mqueue (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
/dev/sda15 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro,discard)
/dev/sdb1 on /mnt type ext4 (rw,relatime,data=ordered)
/dev/sda1 on /var/lib/kubelet type ext4 (rw,relatime,discard,data=ordered)
/dev/sdc on /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/m4185368542 type ext4 (rw,relatime,data=ordered)
/dev/sdc on /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/m4185368542 type ext4 (rw,relatime,data=ordered)
/dev/sdc on /var/lib/kubelet/pods/9e0e3475-c6eb-4950-8180-c5ab90d88124/volumes/kubernetes.io~azure-disk/pvc-f0d0078d-20fe-4b97-892d-be6b882e17c3 type ext4 (rw,relatime,data=ordered)
/dev/sdc on /var/lib/kubelet/pods/9e0e3475-c6eb-4950-8180-c5ab90d88124/volumes/kubernetes.io~azure-disk/pvc-f0d0078d-20fe-4b97-892d-be6b882e17c3 type ext4 (rw,relatime,data=ordered)
/dev/sdc on /var/lib/kubelet/pods/337ab89b-2bff-4a8e-8c66-65b104318426/volumes/kubernetes.io~azure-disk/pvc-f0d0078d-20fe-4b97-892d-be6b882e17c3 type ext4 (rw,relatime,data=ordered)
/dev/sdc on /var/lib/kubelet/pods/337ab89b-2bff-4a8e-8c66-65b104318426/volumes/kubernetes.io~azure-disk/pvc-f0d0078d-20fe-4b97-892d-be6b882e17c3 type ext4 (rw,relatime,data=ordered)
/dev/sdc on /var/lib/kubelet/pods/1066ed68-df0f-423b-a1d6-2694e07550d7/volumes/kubernetes.io~azure-disk/pvc-f0d0078d-20fe-4b97-892d-be6b882e17c3 type ext4 (rw,relatime,data=ordered)
/dev/sdc on /var/lib/kubelet/pods/1066ed68-df0f-423b-a1d6-2694e07550d7/volumes/kubernetes.io~azure-disk/pvc-f0d0078d-20fe-4b97-892d-be6b882e17c3 type ext4 (rw,relatime,data=ordered)
Sorry for the typo around the names. The pods are mounted on same node. Infact in our environment we are currently trying to just use single node for all pods.
Ok then there, there is some pv claim issue or you container does not exist. The issue lies in the volume drivers or the yaml for the pod. You need to create an issue at https://github.com/Azure/kubernetes-volume-drivers/issues and attach your blobfuse-driver.log
Thanks, i will do that. Before i go there, just to add more details so that you can confirm that its surely something to do with volume drivers, below are some more details around the setup.
pvcblobfuse.yaml attached contains the definition for PV and PVC that i am setting up on my cluster node to be used in my reconstruct service pod.
poddeployment.yaml - is the deployment yaml for the pods i am trying to create. Important thing to note about is the replica count which is set to 2 right now and thats where the 2nd pod fails to start. The first pod still works fine.
Also as i mentioned in the inital comment, i am not sure if the blobfuse is suppose to allow two pods using same PVC or not. Can you please confirm that?
The yaml looks correct. Particular versions of the flexvolume driver probably force multiple PVs based on the replica count. I see that you posted in the flexvolume driver github. You should get answers.
Also, Blobfuse does not allow write from multiple instances mounted to the same storage account. Multiple readonly is allowed though.
Thanks for the review. Also for the suggestion about ReadOnly configuration. Infact i did try all 3 possible configurations but none worked so switched back to what was initially configured. I will wait for their response on other repo but will keepe this open for sometime in case there is some other twist into this fact finding.
should be related to issue: Azure/kubernetes-volume-drivers#66 (comment)
@ankurkapoor. Could you please download the latest version 1.3.4 of the flex volume driver. your issue should no longer exits. Please let us know if you are still blocked. I am closing this one for now.