NVMe disk was not discovered on Standard_L8s_v2 VM SKU

Question

NVMe disk was not discovered on Standard_L8s_v2 VM SKU

sunkararp opened this issue 3 years ago · 21 comments

sunkararp commented 3 years ago

What happened:
I'm using Standard_L8s_v2 VM SKU

The 1X1.92 TM nvme disk was not discovered on
It only discovered 80 GB temp disk

What you expected to happen:
Both 80 GB temp disk and 1.92 NVMe disk be discovered

How to reproduce it:
Create a K8s Cluster with vm scale set with Standard_L8s_v2 VM SKU

Anything else we need to know?:

Followed instructions from this https://github.com/Azure/kubernetes-volume-drivers/tree/master/local
This is working fine with other node pools in our ASK cluster
Screen capture

Environment:

Kubernetes version (use kubectl version): v1.20.7
OS (e.g. from /etc/os-release):
Kernel (e.g. uname -a):
Install tools:
Others:

Answer 1 · 2022-01-20T02:11:33.000Z

have you used https://raw.githubusercontent.com/Azure/kubernetes-volume-drivers/master/local/local-pv-provisioner-nvmedisk.yaml? it would find /dev/nvm* disks on agent node.

Answer 2 · 2022-01-20T02:16:13.000Z

@andyzhangx thanks for the reply.

I did use https://raw.githubusercontent.com/Azure/kubernetes-volume-drivers/master/local/local-pv-provisioner-nvmedisk.yaml?
The new node pool I created today did not find the NVMe disks!
Surprisingly, I have few other node pools that use Standard_L8s_v2 VM SKU and work fine

Answer 3 · 2022-01-20T02:18:23.000Z

@sunkararp could you ssh to the agent node, and check whether there is /dev/nvm* device? if there is, follow guide here to get logs: https://github.com/Azure/kubernetes-volume-drivers/tree/master/local#troubleshooting

Answer 4 · 2022-01-20T02:19:21.000Z

and did you add any taint to prevent that daemonset run on your node pool?

Answer 5 · 2022-01-20T02:26:44.000Z

Attached is the screen capture of command 'kubectl logs local-volume-provisioner-5rzd7 -n kube-system'

Answer 6 · 2022-01-20T02:29:58.000Z

so there is no /dev/nvm* disk under that node?

Answer 7 · 2022-01-20T02:31:45.000Z

and did you add any taint to prevent that daemonset run on your node pool?

No

Answer 8 · 2022-01-20T02:33:45.000Z

then do you know what's the device name of that nvme disk in that node pool? It should be /dev/nvme* device name for all nvme disks. Is it reproable?

Answer 9 · 2022-01-20T02:36:50.000Z

Just noticed...

we were using mcr.microsoft.com/k8s/local-volume-provisioner:v2.3.5-alpha of local-volume-provisioner
New script is using mcr.microsoft.com/k8s/local-volume-provisioner:v2.4.0
Does this matter?

Answer 10 · 2022-01-20T02:38:37.000Z

Just noticed...

we were using mcr.microsoft.com/k8s/local-volume-provisioner:v2.3.5-alpha of local-volume-provisioner

New script is using mcr.microsoft.com/k8s/local-volume-provisioner:v2.4.0

Does this matter?

@sunkararp these 2 versions almost has no difference. I think you need to find out what is the nvme device on that node, it's strange.

Answer 11 · 2022-01-20T02:41:23.000Z

@sunkararp these 2 versions almost has no difference. I think you need to find out what is the nvme device on that node, it's strange.

How do I do that? I cannot connect to the node due to compliance.

I can run kubectl exec -it POD however.

Answer 12 · 2022-01-20T02:45:38.000Z

BTW does app container deployed on this VM need to do anything for nvme disks to be visible?

Answer 13 · 2022-01-20T02:53:45.000Z

try kubectl exec -it local-volume-provisioner-xxxxx -n kube-system -- ls /dev/

Answer 14 · 2022-01-20T02:56:30.000Z

sunkararp commented 3 years ago

Answer 15 · 2022-01-20T03:00:35.000Z

i see nvme disk is there, there is conflict between https://github.com/Azure/kubernetes-volume-drivers/blob/master/local/local-pv-provisioner-tempdisk.yaml and local-pv-provisioner-nvmedisk.yaml, pls only use one yaml config.

Answer 16 · 2022-01-20T03:25:21.000Z

Awesome!
Thanks for your help

Answer 17 · 2022-01-29T15:08:40.000Z

Sorry to reopen this issue.

I changed access mode to ReadWriteMany & it fails to BIND (I recreated the VM)
Below is the screen capture of the logs

Therefore, is ReadWriteMany supported?

Answer 18 · 2022-01-29T15:10:40.000Z

@sunkararp no, local disk only supports ReadWriteOnce

Answer 19 · 2022-01-29T15:11:22.000Z

thanks, for quick reply

Answer 20 · 2022-08-19T17:36:49.000Z

@sunkararp would you be willing to have a chat with the PG team on your use case for this?

Answer 21 · 2022-08-19T20:07:22.000Z

@sunkararp would you be willing to have a chat with the PG team on your use case for this?

Sure.

BTW, we already in contact with MSFT on another issue relating to NVMe disk. TrackingID#2207130010002328