Azure/kubernetes-volume-drivers

Unexpected mount failure

xudifsd opened this issue · 6 comments

xudifsd commented

What happened:
I was trying to use this driver to do azure blob mount, I followed instructions from here, but failed to start nginx-flex-blobfuse in the end. I have checked log from /var/log/blobfuse-driver.log:

Mon Sep 16 03:50:29 UTC 2019 INFO: tmp-path not specified, use default path: /tmp/blobfuse/
Mon Sep 16 03:50:29 UTC 2019 EXEC: mkdir -p /var/lib/kubelet/pods/d6b429ed-96bb-4e9c-b0f7-1933eac38875/volumes/azure~blobfuse/test
Mon Sep 16 03:50:29 UTC 2019 INF: AZURE_STORAGE_ACCESS_KEY is set
Mon Sep 16 03:50:29 UTC 2019 INF: export storage account - export AZURE_STORAGE_ACCOUNT=dixu0test
Mon Sep 16 03:50:29 UTC 2019 EXEC: blobfuse /var/lib/kubelet/pods/d6b429ed-96bb-4e9c-b0f7-1933eac38875/volumes/azure~blobfuse/test --container-name=dixu-test --tmp-path=/tmp/blobfuse/ -o allow_other  --file-cache-timeout-in-seconds=120
Mon Sep 16 03:50:29 UTC 2019 ERROR: Invalid usage. Usage:
Mon Sep 16 03:50:29 UTC 2019 ERROR:     /usr/local/bin/blobfuse init
Mon Sep 16 03:50:29 UTC 2019 ERROR:     /usr/local/bin/blobfuse mount <mount dir> <json params>
Mon Sep 16 03:50:29 UTC 2019 ERROR:     /usr/local/bin/blobfuse unmount <mount dir>
Mon Sep 16 03:50:29 UTC 2019 ERROR: { "status": "Failure", "message": "Failed to mount device /dev/ at /var/lib/kubelet/pods/d6b429ed-96bb-4e9c-b0f7-1933eac38875/volumes/azure~blobfuse/test, accountname:dixu0test, error log:Mon Sep 16 03:50:29 UTC 2019 ERROR: /usr/local/bin/blobfuse unmount <mount dir>" }
Mon Sep 16 03:50:30 UTC 2019 INFO: tmp-path not specified, use default path: /tmp/blobfuse/
Mon Sep 16 03:50:30 UTC 2019 EXEC: mkdir -p /var/lib/kubelet/pods/d6b429ed-96bb-4e9c-b0f7-1933eac38875/volumes/azure~blobfuse/test
Mon Sep 16 03:50:30 UTC 2019 INF: AZURE_STORAGE_ACCESS_KEY is set
Mon Sep 16 03:50:30 UTC 2019 INF: export storage account - export AZURE_STORAGE_ACCOUNT=dixu0test
Mon Sep 16 03:50:30 UTC 2019 EXEC: blobfuse /var/lib/kubelet/pods/d6b429ed-96bb-4e9c-b0f7-1933eac38875/volumes/azure~blobfuse/test --container-name=dixu-test --tmp-path=/tmp/blobfuse/ -o allow_other  --file-cache-timeout-in-seconds=120
Mon Sep 16 03:50:30 UTC 2019 ERROR: Invalid usage. Usage:
Mon Sep 16 03:50:30 UTC 2019 ERROR:     /usr/local/bin/blobfuse init
Mon Sep 16 03:50:30 UTC 2019 ERROR:     /usr/local/bin/blobfuse mount <mount dir> <json params>
Mon Sep 16 03:50:30 UTC 2019 ERROR:     /usr/local/bin/blobfuse unmount <mount dir>
Mon Sep 16 03:50:30 UTC 2019 ERROR: { "status": "Failure", "message": "Failed to mount device /dev/ at /var/lib/kubelet/pods/d6b429ed-96bb-4e9c-b0f7-1933eac38875/volumes/azure~blobfuse/test, accountname:dixu0test, error log:Mon Sep 16 03:50:30 UTC 2019 ERROR: /usr/local/bin/blobfuse unmount <mount dir>" }

blobfuse-flexvolume is:

    Image:          mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.9
    Image ID:       docker-pullable://mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:20704adc7246656251953c88a182767f9e71c829a74fac992b788dfbc1ceaed9

What you expected to happen:
nginx-flex-blobfuse can start and running.

How to reproduce it:
Followed instructions from here.

Anything else we need to know?:
I also tried to mount manually:

mkdir test
export AZURE_STORAGE_ACCOUNT=dixu0test
export AZURE_STORAGE_ACCESS_KEY=xxx
sudo blobfuse mount test --container-name=dixu-test --tmp-path=/tmp/blobfuse -o allow_other --file-cache-timeout-in-seconds=120

but get result:

parse error: Invalid numeric literal at line 2, column 0
parse error: Invalid numeric literal at line 2, column 0
parse error: Invalid numeric literal at line 2, column 0
parse error: Invalid numeric literal at line 2, column 0
parse error: Invalid numeric literal at line 2, column 0
parse error: Invalid numeric literal at line 2, column 0
parse error: Invalid numeric literal at line 2, column 0
parse error: Invalid numeric literal at line 2, column 0
{"status": "Failure", "message": "validation failed, error log:accountname is empty"}

Environment:

  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
  • OS (e.g. from /etc/os-release):
NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
  • Kernel (e.g. uname -a): Linux dixudev-worker-nfrrjw 5.0.0-1018-azure #19~18.04.1-Ubuntu SMP Wed Aug 21 05:13:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools:
  • Others:
andyzhangx commented

could you remove sudo and try again:

mkdir test
export AZURE_STORAGE_ACCOUNT=dixu0test
export AZURE_STORAGE_ACCESS_KEY=xxx
blobfuse mount test --container-name=dixu-test --tmp-path=/tmp/blobfuse -o allow_other --file-cache-timeout-in-seconds=120

One possible reason is that your cluster is running on Ubuntu 18.04, we have not tested on that version.
BTW, how did you set up such k8s cluster? By aks-engine? @xudifsd

xudifsd commented

@andyzhangx Still same output:

$ blobfuse mount test --container-name=dixu-test --tmp-path=/tmp/blobfuse -o allow_other --file-cache-timeout-in-seconds=120
parse error: Invalid numeric literal at line 2, column 0
parse error: Invalid numeric literal at line 2, column 0
parse error: Invalid numeric literal at line 2, column 0
parse error: Invalid numeric literal at line 2, column 0
parse error: Invalid numeric literal at line 2, column 0
parse error: Invalid numeric literal at line 2, column 0
parse error: Invalid numeric literal at line 2, column 0
parse error: Invalid numeric literal at line 2, column 0
/usr/local/bin/blobfuse: 18: /usr/local/bin/blobfuse: cannot create /var/log/blobfuse-driver.log: Permission denied
{"status": "Failure", "message": "validation failed, error log:accountname is empty"}

I used my own script to copy kubelet binary to setup k8s cluster.

I think this may be problem of version not matching? Since log displaying Mon Sep 16 03:50:30 UTC 2019 EXEC: blobfuse /var/lib/kubelet/pods/d6b429ed-96bb-4e9c-b0f7-1933eac38875/volumes/azure~blobfuse/test --container-name=dixu-test --tmp-path=/tmp/blobfuse/ -o allow_other --file-cache-timeout-in-seconds=120 M, seems driver was executing blobfuse <mount point> -o xxx, and output requires init or mount or unmount provided.

andyzhangx commented

that could be the issue, at least your kubelet version should match controller manager version

xudifsd commented

@andyzhangx my kubelet version is also v1.15.2. I meant blobfuse version may not match driver's version. I install blobfuse using kubectl apply -f https://raw.githubusercontent.com/Azure/kubernetes-volume-drivers/master/flexvolume/blobfuse/deployment/blobfuse-flexvol-installer-1.9.yaml .

andyzhangx commented

blobfuse-flexvol-installer-1.9.yaml only install blobfuse flexvol driver, and the blobfuse binary should be installed by yourself. To try blobfuse on your agent node, you could run:

su
mkdir test
export AZURE_STORAGE_ACCOUNT=dixu0test
export AZURE_STORAGE_ACCESS_KEY=xxx
blobfuse mount test --container-name=dixu-test --tmp-path=/tmp/blobfuse -o allow_other --file-cache-timeout-in-seconds=120

And also could you share your exact config, there could be something wrong in your config, e.g. unexpected char.

xudifsd commented

Ah, it's because I did not install blobfuse in agent, but used blobfuse installed in volumeplugins. I thought they are the same...