Azure/kubernetes-volume-drivers

Failed to connect to the Azure China storage container

Abel-Liu opened this issue · 7 comments

Abel-Liu commented

What happened:

blobfuse report error:

Failed to connect to the storage container. There might be something wrong about the storage config, please double check the storage account name, account key and container name. errno = 403

What did I do:

I have installed via kubectl apply -f https://raw.githubusercontent.com/Azure/kubernetes-volume-drivers/master/flexvolume/blobfuse/deployment/blobfuse-flexvol-installer-1.9.yaml

/var/log/blobfuse-driver.log content:

Wed Apr 29 07:39:35 UTC 2020 EXEC: mkdir -p /var/lib/kubelet/pods/2244d38f-c312-4b7e-a404-45e931001c04/volumes/azure~blobfuse/test-blobfuse
Wed Apr 29 07:39:35 UTC 2020 INF: AZURE_STORAGE_ACCESS_KEY is set
Wed Apr 29 07:39:35 UTC 2020 INF: BLOBENDPOINT is set
Wed Apr 29 07:39:35 UTC 2020 INF: export storage account - export AZURE_STORAGE_ACCOUNT=resources
Wed Apr 29 07:39:35 UTC 2020 EXEC: blobfuse /var/lib/kubelet/pods/2244d38f-c312-4b7e-a404-45e931001c04/volumes/azure~blobfuse/test-blobfuse --container-name=k8s --tmp-path=/tmp/blobfuse -o allow_other  --file-cache-timeout-in-seconds=120
Wed Apr 29 07:39:36 UTC 2020 ERROR: { "status": "Failure", "message": "Failed to mount device /dev/ at /var/lib/kubelet/pods/2244d38f-c312-4b7e-a404-45e931001c04/volumes/azure~blobfuse/test-blobfuse, accountname:resources, error log:Wed
Apr 29 07:39:35 UTC 2020 EXEC: blobfuse /var/lib/kubelet/pods/2244d38f-c312-4b7e-a404-45e931001c04/volumes/azure~blobfuse/test-blobfuse --container-name=k8s --tmp-path=/tmp/blobfuse -o allow_other --file-cache-timeout-in-seconds=120" }

Then SSH to AKS vmss and test this:

mkdir /tmp/abc
export AZURE_STORAGE_ACCOUNT=resources
export AZURE_STORAGE_SAS_TOKEN="?sv=2019-02-02&ss=bfqt&srt=sco&sp=rwdlacup&se=2020-04-29T17:02:52Z&st=2020-04-29T09:02:52Z&spr=https,http&sig=..."

/usr/bin/blobfuse /tmp/abc --container-name=k8s --tmp-path=/tmp/blobfuse -o allow_other -o ro --file-cache-timeout-in-seconds=120 --use-https=true --blobEndpoint="https://resources.blob.core.chinacloudapi.cn"

Then report 403 error. I am sure storage account name and sas token are right, and container exists, I also tested use access key and failed too.

Hope you give suggestions, thanks a lot.

Environment:

  • Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.5", GitCommit:"20c265fef0741dd71a66480e35bd69f18351daea", GitTreeState:"clean", BuildDate:"2019-10-15T19:16:51Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.7", GitCommit:"6c143d35bb11d74970e7bc0b6c45b6bfdffc0bd4", GitTreeState:"clean", BuildDate:"2019-12-13T18:46:24Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

andyzhangx commented

add one more export command for azure china:

export AZURE_STORAGE_BLOB_ENDPOINT=resources.blob.core.chinacloudapi.cn
andyzhangx commented

and for flexvolume driver, you should create secret, e.g.

kubectl create secret generic blobfusecreds --from-literal blobendpoint="resources.blob.core.chinacloudapi.cn" --from-literal accountname=ACCOUNT-NAME --from-literal accountsastoken="sastoken" --type="azure/blobfuse"
Abel-Liu commented

I add AZURE_STORAGE_BLOB_ENDPOINT but still not working.

andyzhangx commented

can you run "blobfuse --version", only version > 2.0 would work on Azure China.

Abel-Liu commented

blobfuse version is 1.0.3.

Do we have k8s deploy yaml or docker image that install version above 2.0?

Abel-Liu commented

It works! Thanks a lot.