digitalocean/csi-digitalocean

[feature request] More forgiveness in rate limiting

rbren opened this issue · 9 comments

rbren commented

I seem to get rate limited rather quickly when using the DO CSI. Given the nature of the kubernetes control loop, it's not surprising that it makes a number of (probably unnecessary) requests.

I wonder if rate limiting could be relaxed a bit for traffic coming from the CSI?

What did you do? (required. The issue will be closed when not provided.)

I tried to install Budibase via the helm chart

What did you expect to happen?

Everything installs nicely. But instead, I get

  Normal   Provisioning          3m31s (x9 over 7m47s)  dobs.csi.digitalocean.com_master-rbren-cluster_****  External provisioner is provisioning volume for claim "budibase/minio-data"
  Warning  ProvisioningFailed    3m31s (x9 over 7m47s)  dobs.csi.digitalocean.com_master-rbren-cluster_****  failed to provision volume with StorageClass "do-block-storage": rpc error: code = Internal desc = GET https://api.digitalocean.com/v2/volumes?name=pvc-****&region=nyc1: 429 Too many requests
  Normal   ExternalProvisioning  114s (x26 over 7m47s)  persistentvolume-controller                                                          waiting for a volume to be created, either by external provisioner "dobs.csi.digitalocean.com" or manually created by system administrator

Configuration (MUST fill this out):

  • system logs:

Please provide the following logs:


kubectl cluster-info dump > kubernetes-dump.log

I can provide these if need be, but not super comfortable giving all that info out.

This will output everthing from your cluster. Please use a private gist via
https://gist.github.com/ to share this dump with us

  • manifests, such as pvc, deployments, etc.. you used to reproduce:

Please provide the total set of manifests that are needed to reproduce the
issue. Just providing the pvc is not helpful. If you cannot provide it due
privacy concerns, please try creating a reproducible case.

  • CSI Version:
    docker.io/digitalocean/do-csi-plugin:v2.1.2

  • Kubernetes Version:

❯ k version
Client Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.0-rc.2", GitCommit:"27bb2a4a0a5cb8330178d19e57fa61fffa895c98", GitTreeState:"clean", BuildDate:"2020-07-21T17:39:35Z", GoVersion:"go1.14.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.15", GitCommit:"58178e7f7aab455bc8de88d3bdd314b64141e7ee", GitTreeState:"clean", BuildDate:"2021-09-15T19:18:00Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider/framework version, if applicable (such as Rancher): DO
rbren commented

FWIW I deleted the chart, waited 15 minutes 12 hours, and tried again (IIRC the rate limit resets after 10m). Sadly it didn't seem to work - still rate limited. So I'm guessing there's something else going on.

Happy to provide additional logs.

Here are the values I supplied to the budibase chart - that may help to reproduce.

      services:
        couchdb:
          storage: 1Gi
        objectStore:
          storage: 1Gi
        redis:
          storage: 1Gi
      ingress:
        enabled: true
        hosts:
          - host: $BUDIBASE_HOST
            tls: true
            tlsSecret: tls-prod-cert-ghost
            paths:
            - path: /
              pathType: Prefix
              backend:
                service:
                  name: proxy-service
                  port:
                    number: 10000
        annotations:
          kubernetes.io/ingress.class: nginx
          certmanager.k8s.io/cluster-issuer: letsencrypt-prod
          kubernetes.io/tls-acme: "true"
          nginx.ingress.kubernetes.io/proxy-body-size: 5m
timoreimann commented

To my knowledge, the rate limit is not reset entirely at a discrete point in time but rather happens so fluently, with each request having its own timer. See https://docs.digitalocean.com/reference/api/api-reference/#section/Introduction/Rate-Limit for details.

The retry and backoff intervals should be controlled by the sidecars that interact with the DO CSI driver. At this point, I believe all of them should come with reasonable tradeoffs. (I think csi-snapshotter used to have a bug where the retry wasn't honored, but based on your events this should be about provisioning volumes.)

Do I understand correctly that you are using the CSI driver in a self-managed Kubernetes installation (i.e., not DOKS)? If so, could you check the CSI driver logs for how the requests space out? For continuous errors, it should eventually land at a 5 minute waiting period, which should be enough to recover from rate limiting. (Note though that rate limiting applies per API token, so if you reuse the same token then other token consumers may also contribute to hitting the rate limit more easily.)

rbren commented

Just tried again - got 429s within a few seconds of installing.

This is the CSI driver in a managed DOKS cluster.

Here are the last few minutes of logs from csi driver. Nothing really suspicious IMO.

csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:02Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/5b5e44c3-5ac8-4f6e-ad02-f79445d86817/volumes/kubernetes.io~csi/pvc-2f34de0b-b85c-4141-b566-c0995f2b311e/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:02Z" level=info msg="node capacity statistics retrieved" bytes_available=950214656 bytes_total=1023303680 bytes_used=2625536 host_id=275418297 inodes_available=65525 inodes_total=65536 inodes_used=11 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=0468c53e-75ae-11ea-a87f-0a58ac14489e volume_mode=filesystem volume_path="/var/lib/kubelet/pods/5b5e44c3-5ac8-4f6e-ad02-f79445d86817/volumes/kubernetes.io~csi/pvc-2f34de0b-b85c-4141-b566-c0995f2b311e/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:06Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME >  rpc:<type:EXPAND_VOLUME >  rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:06Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=4ced3b86-75c9-11ea-969f-0a58ac144db2 volume_path="/var/lib/kubelet/pods/c5007016-9111-453f-a529-d1ee0513ae93/volumes/kubernetes.io~csi/pvc-e80f4df6-70d1-49d2-bd12-f9653ac62382/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:06Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/c5007016-9111-453f-a529-d1ee0513ae93/volumes/kubernetes.io~csi/pvc-e80f4df6-70d1-49d2-bd12-f9653ac62382/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:06Z" level=info msg="node capacity statistics retrieved" bytes_available=3906445312 bytes_total=4160421888 bytes_used=22454272 host_id=275418297 inodes_available=261974 inodes_total=262144 inodes_used=170 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=4ced3b86-75c9-11ea-969f-0a58ac144db2 volume_mode=filesystem volume_path="/var/lib/kubelet/pods/c5007016-9111-453f-a529-d1ee0513ae93/volumes/kubernetes.io~csi/pvc-e80f4df6-70d1-49d2-bd12-f9653ac62382/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:09Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME >  rpc:<type:EXPAND_VOLUME >  rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:09Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c5c5bbba-1d0b-11eb-a40f-0a58ac14506d volume_path="/var/lib/kubelet/pods/1d1cdb98-329d-41c5-b05e-5040fe216c8d/volumes/kubernetes.io~csi/pvc-8c2ebf45-603f-4e39-89c6-ba58023a9f43/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:09Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/1d1cdb98-329d-41c5-b05e-5040fe216c8d/volumes/kubernetes.io~csi/pvc-8c2ebf45-603f-4e39-89c6-ba58023a9f43/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:09Z" level=info msg="node capacity statistics retrieved" bytes_available=7679172608 bytes_total=8388009984 bytes_used=262565888 host_id=275418297 inodes_available=524093 inodes_total=524288 inodes_used=195 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c5c5bbba-1d0b-11eb-a40f-0a58ac14506d volume_mode=filesystem volume_path="/var/lib/kubelet/pods/1d1cdb98-329d-41c5-b05e-5040fe216c8d/volumes/kubernetes.io~csi/pvc-8c2ebf45-603f-4e39-89c6-ba58023a9f43/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:38Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME >  rpc:<type:EXPAND_VOLUME >  rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:38Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=debe707e-777a-11ea-b3a8-0a58ac144a9e volume_path="/var/lib/kubelet/pods/48faac62-e354-4a20-a4a8-f2f97315a81f/volumes/kubernetes.io~csi/pvc-60b9563e-7b3e-4617-b046-5e04b39f5548/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:38Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/48faac62-e354-4a20-a4a8-f2f97315a81f/volumes/kubernetes.io~csi/pvc-60b9563e-7b3e-4617-b046-5e04b39f5548/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:38Z" level=info msg="node capacity statistics retrieved" bytes_available=25775620096 bytes_total=50462347264 bytes_used=22092972032 host_id=275418297 inodes_available=3138861 inodes_total=3145728 inodes_used=6867 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=debe707e-777a-11ea-b3a8-0a58ac144a9e volume_mode=filesystem volume_path="/var/lib/kubelet/pods/48faac62-e354-4a20-a4a8-f2f97315a81f/volumes/kubernetes.io~csi/pvc-60b9563e-7b3e-4617-b046-5e04b39f5548/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:44Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME >  rpc:<type:EXPAND_VOLUME >  rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:44Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=a3d6a386-60b6-11ea-88c9-0a58ac1446ab volume_path="/var/lib/kubelet/pods/da656c76-3a61-4a85-a7bc-e75590dcf929/volumes/kubernetes.io~csi/pvc-a84b3288-0aca-48e5-bfca-22c3986e8187/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:44Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/da656c76-3a61-4a85-a7bc-e75590dcf929/volumes/kubernetes.io~csi/pvc-a84b3288-0aca-48e5-bfca-22c3986e8187/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:44Z" level=info msg="node capacity statistics retrieved" bytes_available=7581257728 bytes_total=8388009984 bytes_used=360480768 host_id=275418297 inodes_available=523049 inodes_total=524288 inodes_used=1239 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=a3d6a386-60b6-11ea-88c9-0a58ac1446ab volume_mode=filesystem volume_path="/var/lib/kubelet/pods/da656c76-3a61-4a85-a7bc-e75590dcf929/volumes/kubernetes.io~csi/pvc-a84b3288-0aca-48e5-bfca-22c3986e8187/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:46Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME >  rpc:<type:EXPAND_VOLUME >  rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:46Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=8701ff1c-75ad-11ea-b009-0a58ac144514 volume_path="/var/lib/kubelet/pods/c6ac1ad6-5091-4ed3-b95e-69d0f1821d86/volumes/kubernetes.io~csi/pvc-fcd10d7e-7017-4f63-8e87-48d89a6d93ab/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:46Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/c6ac1ad6-5091-4ed3-b95e-69d0f1821d86/volumes/kubernetes.io~csi/pvc-fcd10d7e-7017-4f63-8e87-48d89a6d93ab/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:54:46Z" level=info msg="node capacity statistics retrieved" bytes_available=1541267456 bytes_total=2046640128 bytes_used=381222912 host_id=275418297 inodes_available=129826 inodes_total=131072 inodes_used=1246 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=8701ff1c-75ad-11ea-b009-0a58ac144514 volume_mode=filesystem volume_path="/var/lib/kubelet/pods/c6ac1ad6-5091-4ed3-b95e-69d0f1821d86/volumes/kubernetes.io~csi/pvc-fcd10d7e-7017-4f63-8e87-48d89a6d93ab/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:09Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME >  rpc:<type:EXPAND_VOLUME >  rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:09Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c40c49dd-60b6-11ea-b89e-0a58ac14480f volume_path="/var/lib/kubelet/pods/16a28727-423f-47eb-b172-d9738158e85f/volumes/kubernetes.io~csi/pvc-0141174a-de96-409d-8efd-cea6c528893c/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:09Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/16a28727-423f-47eb-b172-d9738158e85f/volumes/kubernetes.io~csi/pvc-0141174a-de96-409d-8efd-cea6c528893c/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:09Z" level=info msg="node capacity statistics retrieved" bytes_available=1916198912 bytes_total=2046640128 bytes_used=6291456 host_id=275418297 inodes_available=131061 inodes_total=131072 inodes_used=11 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c40c49dd-60b6-11ea-b89e-0a58ac14480f volume_mode=filesystem volume_path="/var/lib/kubelet/pods/16a28727-423f-47eb-b172-d9738158e85f/volumes/kubernetes.io~csi/pvc-0141174a-de96-409d-8efd-cea6c528893c/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:14Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME >  rpc:<type:EXPAND_VOLUME >  rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:14Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c5c5bbba-1d0b-11eb-a40f-0a58ac14506d volume_path="/var/lib/kubelet/pods/1d1cdb98-329d-41c5-b05e-5040fe216c8d/volumes/kubernetes.io~csi/pvc-8c2ebf45-603f-4e39-89c6-ba58023a9f43/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:14Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/1d1cdb98-329d-41c5-b05e-5040fe216c8d/volumes/kubernetes.io~csi/pvc-8c2ebf45-603f-4e39-89c6-ba58023a9f43/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:14Z" level=info msg="node capacity statistics retrieved" bytes_available=7679172608 bytes_total=8388009984 bytes_used=262565888 host_id=275418297 inodes_available=524093 inodes_total=524288 inodes_used=195 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c5c5bbba-1d0b-11eb-a40f-0a58ac14506d volume_mode=filesystem volume_path="/var/lib/kubelet/pods/1d1cdb98-329d-41c5-b05e-5040fe216c8d/volumes/kubernetes.io~csi/pvc-8c2ebf45-603f-4e39-89c6-ba58023a9f43/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:31Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME >  rpc:<type:EXPAND_VOLUME >  rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:31Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=0468c53e-75ae-11ea-a87f-0a58ac14489e volume_path="/var/lib/kubelet/pods/5b5e44c3-5ac8-4f6e-ad02-f79445d86817/volumes/kubernetes.io~csi/pvc-2f34de0b-b85c-4141-b566-c0995f2b311e/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:31Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/5b5e44c3-5ac8-4f6e-ad02-f79445d86817/volumes/kubernetes.io~csi/pvc-2f34de0b-b85c-4141-b566-c0995f2b311e/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:31Z" level=info msg="node capacity statistics retrieved" bytes_available=950214656 bytes_total=1023303680 bytes_used=2625536 host_id=275418297 inodes_available=65525 inodes_total=65536 inodes_used=11 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=0468c53e-75ae-11ea-a87f-0a58ac14489e volume_mode=filesystem volume_path="/var/lib/kubelet/pods/5b5e44c3-5ac8-4f6e-ad02-f79445d86817/volumes/kubernetes.io~csi/pvc-2f34de0b-b85c-4141-b566-c0995f2b311e/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:42Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME >  rpc:<type:EXPAND_VOLUME >  rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:42Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=4ced3b86-75c9-11ea-969f-0a58ac144db2 volume_path="/var/lib/kubelet/pods/c5007016-9111-453f-a529-d1ee0513ae93/volumes/kubernetes.io~csi/pvc-e80f4df6-70d1-49d2-bd12-f9653ac62382/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:42Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/c5007016-9111-453f-a529-d1ee0513ae93/volumes/kubernetes.io~csi/pvc-e80f4df6-70d1-49d2-bd12-f9653ac62382/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:42Z" level=info msg="node capacity statistics retrieved" bytes_available=3906445312 bytes_total=4160421888 bytes_used=22454272 host_id=275418297 inodes_available=261974 inodes_total=262144 inodes_used=170 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=4ced3b86-75c9-11ea-969f-0a58ac144db2 volume_mode=filesystem volume_path="/var/lib/kubelet/pods/c5007016-9111-453f-a529-d1ee0513ae93/volumes/kubernetes.io~csi/pvc-e80f4df6-70d1-49d2-bd12-f9653ac62382/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:57Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME >  rpc:<type:EXPAND_VOLUME >  rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:57Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=8701ff1c-75ad-11ea-b009-0a58ac144514 volume_path="/var/lib/kubelet/pods/c6ac1ad6-5091-4ed3-b95e-69d0f1821d86/volumes/kubernetes.io~csi/pvc-fcd10d7e-7017-4f63-8e87-48d89a6d93ab/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:57Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/c6ac1ad6-5091-4ed3-b95e-69d0f1821d86/volumes/kubernetes.io~csi/pvc-fcd10d7e-7017-4f63-8e87-48d89a6d93ab/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:55:57Z" level=info msg="node capacity statistics retrieved" bytes_available=1541267456 bytes_total=2046640128 bytes_used=381222912 host_id=275418297 inodes_available=129826 inodes_total=131072 inodes_used=1246 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=8701ff1c-75ad-11ea-b009-0a58ac144514 volume_mode=filesystem volume_path="/var/lib/kubelet/pods/c6ac1ad6-5091-4ed3-b95e-69d0f1821d86/volumes/kubernetes.io~csi/pvc-fcd10d7e-7017-4f63-8e87-48d89a6d93ab/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:04Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME >  rpc:<type:EXPAND_VOLUME >  rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:04Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=a3d6a386-60b6-11ea-88c9-0a58ac1446ab volume_path="/var/lib/kubelet/pods/da656c76-3a61-4a85-a7bc-e75590dcf929/volumes/kubernetes.io~csi/pvc-a84b3288-0aca-48e5-bfca-22c3986e8187/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:04Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/da656c76-3a61-4a85-a7bc-e75590dcf929/volumes/kubernetes.io~csi/pvc-a84b3288-0aca-48e5-bfca-22c3986e8187/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:04Z" level=info msg="node capacity statistics retrieved" bytes_available=7581257728 bytes_total=8388009984 bytes_used=360480768 host_id=275418297 inodes_available=523049 inodes_total=524288 inodes_used=1239 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=a3d6a386-60b6-11ea-88c9-0a58ac1446ab volume_mode=filesystem volume_path="/var/lib/kubelet/pods/da656c76-3a61-4a85-a7bc-e75590dcf929/volumes/kubernetes.io~csi/pvc-a84b3288-0aca-48e5-bfca-22c3986e8187/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:30Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME >  rpc:<type:EXPAND_VOLUME >  rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:30Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=debe707e-777a-11ea-b3a8-0a58ac144a9e volume_path="/var/lib/kubelet/pods/48faac62-e354-4a20-a4a8-f2f97315a81f/volumes/kubernetes.io~csi/pvc-60b9563e-7b3e-4617-b046-5e04b39f5548/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:30Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/48faac62-e354-4a20-a4a8-f2f97315a81f/volumes/kubernetes.io~csi/pvc-60b9563e-7b3e-4617-b046-5e04b39f5548/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:30Z" level=info msg="node capacity statistics retrieved" bytes_available=25775620096 bytes_total=50462347264 bytes_used=22092972032 host_id=275418297 inodes_available=3138861 inodes_total=3145728 inodes_used=6867 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=debe707e-777a-11ea-b3a8-0a58ac144a9e volume_mode=filesystem volume_path="/var/lib/kubelet/pods/48faac62-e354-4a20-a4a8-f2f97315a81f/volumes/kubernetes.io~csi/pvc-60b9563e-7b3e-4617-b046-5e04b39f5548/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:34Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME >  rpc:<type:EXPAND_VOLUME >  rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:34Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=0468c53e-75ae-11ea-a87f-0a58ac14489e volume_path="/var/lib/kubelet/pods/5b5e44c3-5ac8-4f6e-ad02-f79445d86817/volumes/kubernetes.io~csi/pvc-2f34de0b-b85c-4141-b566-c0995f2b311e/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:34Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/5b5e44c3-5ac8-4f6e-ad02-f79445d86817/volumes/kubernetes.io~csi/pvc-2f34de0b-b85c-4141-b566-c0995f2b311e/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:34Z" level=info msg="node capacity statistics retrieved" bytes_available=950214656 bytes_total=1023303680 bytes_used=2625536 host_id=275418297 inodes_available=65525 inodes_total=65536 inodes_used=11 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=0468c53e-75ae-11ea-a87f-0a58ac14489e volume_mode=filesystem volume_path="/var/lib/kubelet/pods/5b5e44c3-5ac8-4f6e-ad02-f79445d86817/volumes/kubernetes.io~csi/pvc-2f34de0b-b85c-4141-b566-c0995f2b311e/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:43Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME >  rpc:<type:EXPAND_VOLUME >  rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:43Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c5c5bbba-1d0b-11eb-a40f-0a58ac14506d volume_path="/var/lib/kubelet/pods/1d1cdb98-329d-41c5-b05e-5040fe216c8d/volumes/kubernetes.io~csi/pvc-8c2ebf45-603f-4e39-89c6-ba58023a9f43/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:43Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/1d1cdb98-329d-41c5-b05e-5040fe216c8d/volumes/kubernetes.io~csi/pvc-8c2ebf45-603f-4e39-89c6-ba58023a9f43/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:43Z" level=info msg="node capacity statistics retrieved" bytes_available=7679172608 bytes_total=8388009984 bytes_used=262565888 host_id=275418297 inodes_available=524093 inodes_total=524288 inodes_used=195 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c5c5bbba-1d0b-11eb-a40f-0a58ac14506d volume_mode=filesystem volume_path="/var/lib/kubelet/pods/1d1cdb98-329d-41c5-b05e-5040fe216c8d/volumes/kubernetes.io~csi/pvc-8c2ebf45-603f-4e39-89c6-ba58023a9f43/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:49Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME >  rpc:<type:EXPAND_VOLUME >  rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:49Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=4ced3b86-75c9-11ea-969f-0a58ac144db2 volume_path="/var/lib/kubelet/pods/c5007016-9111-453f-a529-d1ee0513ae93/volumes/kubernetes.io~csi/pvc-e80f4df6-70d1-49d2-bd12-f9653ac62382/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:49Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/c5007016-9111-453f-a529-d1ee0513ae93/volumes/kubernetes.io~csi/pvc-e80f4df6-70d1-49d2-bd12-f9653ac62382/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:49Z" level=info msg="node capacity statistics retrieved" bytes_available=3906445312 bytes_total=4160421888 bytes_used=22454272 host_id=275418297 inodes_available=261974 inodes_total=262144 inodes_used=170 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=4ced3b86-75c9-11ea-969f-0a58ac144db2 volume_mode=filesystem volume_path="/var/lib/kubelet/pods/c5007016-9111-453f-a529-d1ee0513ae93/volumes/kubernetes.io~csi/pvc-e80f4df6-70d1-49d2-bd12-f9653ac62382/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:51Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME >  rpc:<type:EXPAND_VOLUME >  rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:51Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c40c49dd-60b6-11ea-b89e-0a58ac14480f volume_path="/var/lib/kubelet/pods/16a28727-423f-47eb-b172-d9738158e85f/volumes/kubernetes.io~csi/pvc-0141174a-de96-409d-8efd-cea6c528893c/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:51Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/16a28727-423f-47eb-b172-d9738158e85f/volumes/kubernetes.io~csi/pvc-0141174a-de96-409d-8efd-cea6c528893c/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:56:51Z" level=info msg="node capacity statistics retrieved" bytes_available=1916198912 bytes_total=2046640128 bytes_used=6291456 host_id=275418297 inodes_available=131061 inodes_total=131072 inodes_used=11 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c40c49dd-60b6-11ea-b89e-0a58ac14480f volume_mode=filesystem volume_path="/var/lib/kubelet/pods/16a28727-423f-47eb-b172-d9738158e85f/volumes/kubernetes.io~csi/pvc-0141174a-de96-409d-8efd-cea6c528893c/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:57:02Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME >  rpc:<type:EXPAND_VOLUME >  rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:57:02Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=8701ff1c-75ad-11ea-b009-0a58ac144514 volume_path="/var/lib/kubelet/pods/c6ac1ad6-5091-4ed3-b95e-69d0f1821d86/volumes/kubernetes.io~csi/pvc-fcd10d7e-7017-4f63-8e87-48d89a6d93ab/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:57:02Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/c6ac1ad6-5091-4ed3-b95e-69d0f1821d86/volumes/kubernetes.io~csi/pvc-fcd10d7e-7017-4f63-8e87-48d89a6d93ab/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:57:02Z" level=info msg="node capacity statistics retrieved" bytes_available=1541267456 bytes_total=2046640128 bytes_used=381222912 host_id=275418297 inodes_available=129826 inodes_total=131072 inodes_used=1246 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=8701ff1c-75ad-11ea-b009-0a58ac144514 volume_mode=filesystem volume_path="/var/lib/kubelet/pods/c6ac1ad6-5091-4ed3-b95e-69d0f1821d86/volumes/kubernetes.io~csi/pvc-fcd10d7e-7017-4f63-8e87-48d89a6d93ab/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:57:09Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME >  rpc:<type:EXPAND_VOLUME >  rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:57:09Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=a3d6a386-60b6-11ea-88c9-0a58ac1446ab volume_path="/var/lib/kubelet/pods/da656c76-3a61-4a85-a7bc-e75590dcf929/volumes/kubernetes.io~csi/pvc-a84b3288-0aca-48e5-bfca-22c3986e8187/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:57:09Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/da656c76-3a61-4a85-a7bc-e75590dcf929/volumes/kubernetes.io~csi/pvc-a84b3288-0aca-48e5-bfca-22c3986e8187/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:57:09Z" level=info msg="node capacity statistics retrieved" bytes_available=7581257728 bytes_total=8388009984 bytes_used=360480768 host_id=275418297 inodes_available=523049 inodes_total=524288 inodes_used=1239 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=a3d6a386-60b6-11ea-88c9-0a58ac1446ab volume_mode=filesystem volume_path="/var/lib/kubelet/pods/da656c76-3a61-4a85-a7bc-e75590dcf929/volumes/kubernetes.io~csi/pvc-a84b3288-0aca-48e5-bfca-22c3986e8187/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:57:36Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME >  rpc:<type:EXPAND_VOLUME >  rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:57:36Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=debe707e-777a-11ea-b3a8-0a58ac144a9e volume_path="/var/lib/kubelet/pods/48faac62-e354-4a20-a4a8-f2f97315a81f/volumes/kubernetes.io~csi/pvc-60b9563e-7b3e-4617-b046-5e04b39f5548/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:57:36Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/48faac62-e354-4a20-a4a8-f2f97315a81f/volumes/kubernetes.io~csi/pvc-60b9563e-7b3e-4617-b046-5e04b39f5548/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:57:36Z" level=info msg="node capacity statistics retrieved" bytes_available=25775620096 bytes_total=50462347264 bytes_used=22092972032 host_id=275418297 inodes_available=3138861 inodes_total=3145728 inodes_used=6867 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=debe707e-777a-11ea-b3a8-0a58ac144a9e volume_mode=filesystem volume_path="/var/lib/kubelet/pods/48faac62-e354-4a20-a4a8-f2f97315a81f/volumes/kubernetes.io~csi/pvc-60b9563e-7b3e-4617-b046-5e04b39f5548/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:58:02Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME >  rpc:<type:EXPAND_VOLUME >  rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:58:02Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c40c49dd-60b6-11ea-b89e-0a58ac14480f volume_path="/var/lib/kubelet/pods/16a28727-423f-47eb-b172-d9738158e85f/volumes/kubernetes.io~csi/pvc-0141174a-de96-409d-8efd-cea6c528893c/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:58:02Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/16a28727-423f-47eb-b172-d9738158e85f/volumes/kubernetes.io~csi/pvc-0141174a-de96-409d-8efd-cea6c528893c/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:58:02Z" level=info msg="node capacity statistics retrieved" bytes_available=1916198912 bytes_total=2046640128 bytes_used=6291456 host_id=275418297 inodes_available=131061 inodes_total=131072 inodes_used=11 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c40c49dd-60b6-11ea-b89e-0a58ac14480f volume_mode=filesystem volume_path="/var/lib/kubelet/pods/16a28727-423f-47eb-b172-d9738158e85f/volumes/kubernetes.io~csi/pvc-0141174a-de96-409d-8efd-cea6c528893c/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:58:05Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME >  rpc:<type:EXPAND_VOLUME >  rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:58:05Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=8701ff1c-75ad-11ea-b009-0a58ac144514 volume_path="/var/lib/kubelet/pods/c6ac1ad6-5091-4ed3-b95e-69d0f1821d86/volumes/kubernetes.io~csi/pvc-fcd10d7e-7017-4f63-8e87-48d89a6d93ab/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:58:05Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/c6ac1ad6-5091-4ed3-b95e-69d0f1821d86/volumes/kubernetes.io~csi/pvc-fcd10d7e-7017-4f63-8e87-48d89a6d93ab/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:58:05Z" level=info msg="node capacity statistics retrieved" bytes_available=1541267456 bytes_total=2046640128 bytes_used=381222912 host_id=275418297 inodes_available=129826 inodes_total=131072 inodes_used=1246 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=8701ff1c-75ad-11ea-b009-0a58ac144514 volume_mode=filesystem volume_path="/var/lib/kubelet/pods/c6ac1ad6-5091-4ed3-b95e-69d0f1821d86/volumes/kubernetes.io~csi/pvc-fcd10d7e-7017-4f63-8e87-48d89a6d93ab/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:58:18Z" level=info msg="node get capabilities called" host_id=275418297 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME >  rpc:<type:EXPAND_VOLUME >  rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:58:18Z" level=info msg="node get volume stats called" host_id=275418297 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c5c5bbba-1d0b-11eb-a40f-0a58ac14506d volume_path="/var/lib/kubelet/pods/1d1cdb98-329d-41c5-b05e-5040fe216c8d/volumes/kubernetes.io~csi/pvc-8c2ebf45-603f-4e39-89c6-ba58023a9f43/mount"
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:58:18Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/1d1cdb98-329d-41c5-b05e-5040fe216c8d/volumes/kubernetes.io~csi/pvc-8c2ebf45-603f-4e39-89c6-ba58023a9f43/mount -J]" cmd=findmnt host_id=275418297 region=nyc1 version=v2.1.2
csi-do-node-mv4pf csi-do-plugin time="2021-11-27T17:58:18Z" level=info msg="node capacity statistics retrieved" bytes_available=7679172608 bytes_total=8388009984 bytes_used=262565888 host_id=275418297 inodes_available=524093 inodes_total=524288 inodes_used=195 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c5c5bbba-1d0b-11eb-a40f-0a58ac14506d volume_mode=filesystem volume_path="/var/lib/kubelet/pods/1d1cdb98-329d-41c5-b05e-5040fe216c8d/volumes/kubernetes.io~csi/pvc-8c2ebf45-603f-4e39-89c6-ba58023a9f43/mount"
csi-do-node-kkxlb csi-do-plugin time="2021-11-27T17:58:23Z" level=info msg="node get capabilities called" host_id=275421176 method=node_get_capabilities node_capabilities="[rpc:<type:STAGE_UNSTAGE_VOLUME >  rpc:<type:EXPAND_VOLUME >  rpc:<type:GET_VOLUME_STATS > ]" region=nyc1 version=v2.1.2
csi-do-node-kkxlb csi-do-plugin time="2021-11-27T17:58:23Z" level=info msg="node get volume stats called" host_id=275421176 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c5743bb9-1d0b-11eb-84b4-0a58ac144252 volume_path="/var/lib/kubelet/pods/93e941f7-9448-42e0-8e55-a861370eabc9/volumes/kubernetes.io~csi/pvc-95514ac5-945b-4a6c-a322-77864bfc7c50/mount"
csi-do-node-kkxlb csi-do-plugin time="2021-11-27T17:58:23Z" level=info msg="checking if target is mounted" args="[-o TARGET,PROPAGATION,FSTYPE,OPTIONS -M /var/lib/kubelet/pods/93e941f7-9448-42e0-8e55-a861370eabc9/volumes/kubernetes.io~csi/pvc-95514ac5-945b-4a6c-a322-77864bfc7c50/mount -J]" cmd=findmnt host_id=275421176 region=nyc1 version=v2.1.2
csi-do-node-kkxlb csi-do-plugin time="2021-11-27T17:58:23Z" level=info msg="node capacity statistics retrieved" bytes_available=85047205888 bytes_total=100924641280 bytes_used=10706698240 host_id=275421176 inodes_available=6276744 inodes_total=6291456 inodes_used=14712 method=node_get_volume_stats region=nyc1 version=v2.1.2 volume_id=c5743bb9-1d0b-11eb-84b4-0a58ac144252 volume_mode=filesystem volume_path="/var/lib/kubelet/pods/93e941f7-9448-42e0-8e55-a861370eabc9/volumes/kubernetes.io~csi/pvc-95514ac5-945b-4a6c-a322-77864bfc7c50/mount"
rbren commented

Could the issue possibly have to do with the fact that it's trying to provision two volumes at the same time?

timoreimann commented

@rbren I took a look at your cluster, and I think I found the problem.

Turns out you are indeed affected by the lack of backoff in csi-snapshotter. Although the feature has been released by now, it is only available in v4 of the csi-snapshotter sidecar while we still have to run v3 in DOKS. (We yet have to migrate from snapshots v1beta1 to v1 which is a requirement for using csi-snapshotter v4.)

The reason why your copy of csi-snapshotter hits the API repeatedly is because you are trying to have more than 25 snapshots per volume which is beyond the limit, so the driver keeps attempting to create snapshots but fails perpetually. Could you delete at least enough snapshots to drop below the 25 snapshots limit (and stop/adjust any processes you may have in place that periodically create new snapshots)? Note that the deletes won't go through right now since I have temporarily disabled your cluster's CSI driver so that your API quota can recover. I will re-enable the driver once you have confirmed that the snapshots deletes were issued. As soon as that happens, the snapshot deletes should complete, and your CSI driver should operate normally again.

Sorry for the inconveniences. Once we're able to upgrade to csi-snapshotter v4 in DOKS, exceeding snapshot limits should not lead to API quota exhaustion anymore. Additionally, there is also no clear Kubernetes event emitted by csi-snapshotter right now to inform about the nature of the problem; if that's still not the case with v4, then we can surely drive an improvement on the error reporting front.

rbren commented

@timoreimann sorry for the lack of response!

Just to confirm - that's 25 snapshots per volume, right? I've deleted out a bunch of old snapshots, hopefully that helps. (side note: bulk deletion in the UI would be a very helpful feature)

I don't really want to disable my auto-snapshotter, as that's my backup strategy 😬

Any word on v4?

rbren commented

Deleting the old snapshots seemed to work! Hopefully I won't regret it 😅

timoreimann commented

My bad, the 25 limit is per volume of course. Glad you figured it out. 👍

v4 support is still on the road map. We are currently sorting out a few details on how to best integrate the validating web hook that's needed to prevent invalid snapshots from becoming inaccessible as customers move from snapshots API beta to v1/GA. Stay tuned. 🙂

timoreimann commented

Going to close out the ticket as I believe we've wrapped up the discussion / case.

A new release should make it through once we do support snapshot v4 with the improved retry logic.