error resizing volume: volume/snapshot capacity limit exceeded
sjdrc opened this issue · 1 comments
sjdrc commented
What did you do? (required. The issue will be closed when not provided.)
I updated the persistence.size value for the minio helm chart to be 900Gi from 256Gi.
What did you expect to happen?
The volume would be expanded
What did happen?
$ kubectl describe pvc minio -n portal
Name: minio
Namespace: <redacted>
StorageClass: do-block-storage
Status: Bound
Volume: pvc-9d39fce7-8781-4c11-bb56-e3b0b0cebd50
Labels: app=minio
app.kubernetes.io/managed-by=Helm
chart=minio-5.0.4
heritage=Helm
release=minio
Annotations: meta.helm.sh/release-name: minio
meta.helm.sh/release-namespace: <redacted>
pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: dobs.csi.digitalocean.com
volume.kubernetes.io/storage-provisioner: dobs.csi.digitalocean.com
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 256Gi
Access Modes: RWO
VolumeMode: Filesystem
Used By: minio-656b9664f8-twlnw
Conditions:
Type Status LastProbeTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
Resizing True Mon, 01 Jan 0001 00:00:00 +0000 Thu, 02 Feb 2023 14:47:48 +1100
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ExternalExpanding 10m volume_expand Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC.
Warning VolumeResizeFailed 10m external-resizer dobs.csi.digitalocean.com resize volume "pvc-9d39fce7-8781-4c11-bb56-e3b0b0cebd50" by resizer "dobs.csi.digitalocean.com" failed: rpc error: code = Internal desc = cannot resize volume 3de6b6d4-6e67-11ed-9fa9-0a58ac12a07e: POST https://api.digitalocean.com/v2/volumes/3de6b6d4-6e67-11ed-9fa9-0a58ac12a07e/actions: 403 (request "b27cf377-9508-4813-b390-9ab3aa5cc727") error resizing volume: volume/snapshot capacity limit exceeded
Warning VolumeResizeFailed 10m external-resizer dobs.csi.digitalocean.com resize volume "pvc-9d39fce7-8781-4c11-bb56-e3b0b0cebd50" by resizer "dobs.csi.digitalocean.com" failed: rpc error: code = Internal desc = cannot resize volume 3de6b6d4-6e67-11ed-9fa9-0a58ac12a07e: POST https://api.digitalocean.com/v2/volumes/3de6b6d4-6e67-11ed-9fa9-0a58ac12a07e/actions: 403 (request "9520fb4b-0068-4d2c-9840-3b8eaf4670d7") error resizing volume: volume/snapshot capacity limit exceeded
Warning VolumeResizeFailed 10m external-resizer dobs.csi.digitalocean.com resize volume "pvc-9d39fce7-8781-4c11-bb56-e3b0b0cebd50" by resizer "dobs.csi.digitalocean.com" failed: rpc error: code = Internal desc = cannot resize volume 3de6b6d4-6e67-11ed-9fa9-0a58ac12a07e: POST https://api.digitalocean.com/v2/volumes/3de6b6d4-6e67-11ed-9fa9-0a58ac12a07e/actions: 403 (request "2fa91fdc-7f4b-4ceb-ac4a-b52e6586079a") error resizing volume: volume/snapshot capacity limit exceeded
Warning VolumeResizeFailed 10m external-resizer dobs.csi.digitalocean.com resize volume "pvc-9d39fce7-8781-4c11-bb56-e3b0b0cebd50" by resizer "dobs.csi.digitalocean.com" failed: rpc error: code = Internal desc = cannot resize volume 3de6b6d4-6e67-11ed-9fa9-0a58ac12a07e: POST https://api.digitalocean.com/v2/volumes/3de6b6d4-6e67-11ed-9fa9-0a58ac12a07e/actions: 403 (request "f3b7614c-10ca-45e1-b4f0-a83c41971711") error resizing volume: volume/snapshot capacity limit exceeded
Warning VolumeResizeFailed 10m external-resizer dobs.csi.digitalocean.com resize volume "pvc-9d39fce7-8781-4c11-bb56-e3b0b0cebd50" by resizer "dobs.csi.digitalocean.com" failed: rpc error: code = Internal desc = cannot resize volume 3de6b6d4-6e67-11ed-9fa9-0a58ac12a07e: POST https://api.digitalocean.com/v2/volumes/3de6b6d4-6e67-11ed-9fa9-0a58ac12a07e/actions: 403 (request "3978b71d-db87-419e-9b1d-012bf9cfcf62") error resizing volume: volume/snapshot capacity limit exceeded
Warning VolumeResizeFailed 10m external-resizer dobs.csi.digitalocean.com resize volume "pvc-9d39fce7-8781-4c11-bb56-e3b0b0cebd50" by resizer "dobs.csi.digitalocean.com" failed: rpc error: code = Internal desc = cannot resize volume 3de6b6d4-6e67-11ed-9fa9-0a58ac12a07e: POST https://api.digitalocean.com/v2/volumes/3de6b6d4-6e67-11ed-9fa9-0a58ac12a07e/actions: 403 (request "e39ebc33-bc7a-44e0-b39e-6726ff4845bd") error resizing volume: volume/snapshot capacity limit exceeded
Warning VolumeResizeFailed 10m external-resizer dobs.csi.digitalocean.com resize volume "pvc-9d39fce7-8781-4c11-bb56-e3b0b0cebd50" by resizer "dobs.csi.digitalocean.com" failed: rpc error: code = Internal desc = cannot resize volume 3de6b6d4-6e67-11ed-9fa9-0a58ac12a07e: POST https://api.digitalocean.com/v2/volumes/3de6b6d4-6e67-11ed-9fa9-0a58ac12a07e/actions: 403 (request "8b52674b-8568-46cc-a7db-c8b3a914a404") error resizing volume: volume/snapshot capacity limit exceeded
Warning VolumeResizeFailed 10m external-resizer dobs.csi.digitalocean.com resize volume "pvc-9d39fce7-8781-4c11-bb56-e3b0b0cebd50" by resizer "dobs.csi.digitalocean.com" failed: rpc error: code = Internal desc = cannot resize volume 3de6b6d4-6e67-11ed-9fa9-0a58ac12a07e: POST https://api.digitalocean.com/v2/volumes/3de6b6d4-6e67-11ed-9fa9-0a58ac12a07e/actions: 403 (request "202c107f-9f0d-498b-b4b3-7a6c8f21a5de") error resizing volume: volume/snapshot capacity limit exceeded
Warning VolumeResizeFailed 10m external-resizer dobs.csi.digitalocean.com resize volume "pvc-9d39fce7-8781-4c11-bb56-e3b0b0cebd50" by resizer "dobs.csi.digitalocean.com" failed: rpc error: code = Internal desc = cannot resize volume 3de6b6d4-6e67-11ed-9fa9-0a58ac12a07e: POST https://api.digitalocean.com/v2/volumes/3de6b6d4-6e67-11ed-9fa9-0a58ac12a07e/actions: 403 (request "4abcf459-79f2-4dd3-9aa5-5844479679ac") error resizing volume: volume/snapshot capacity limit exceeded
Warning VolumeResizeFailed 5m31s (x18 over 9m57s) external-resizer dobs.csi.digitalocean.com (combined from similar events): resize volume "pvc-9d39fce7-8781-4c11-bb56-e3b0b0cebd50" by resizer "dobs.csi.digitalocean.com" failed: rpc error: code = Internal desc = cannot resize volume 3de6b6d4-6e67-11ed-9fa9-0a58ac12a07e: POST https://api.digitalocean.com/v2/volumes/3de6b6d4-6e67-11ed-9fa9-0a58ac12a07e/actions: 403 (request "72dbd51f-b067-41bf-bd1d-ead42cc1de33") error resizing volume: volume/snapshot capacity limit exceeded
Normal Resizing 34s (x31 over 15d) external-resizer dobs.csi.digitalocean.com External resizer is resizing volume pvc-9d39fce7-8781-4c11-bb56-e3b0b0cebd50
Configuration (MUST fill this out):
- system logs:
Please provide the following logs:
kubectl cluster-info dump > kubernetes-dump.log
I can provide the full logs if required, I will need to redact them.
This will output everthing from your cluster. Please use a private gist via
https://gist.github.com/ to share this dump with us
- manifests, such as pvc, deployments, etc.. you used to reproduce:
Please provide the total set of manifests that are needed to reproduce the
issue. Just providing the pvc is not helpful. If you cannot provide it due
privacy concerns, please try creating a reproducible case.
helm install -f values.yaml --create-namespace -n minio minio minio/minio
ingress:
annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: "true"
enabled: true
hosts:
- <redacted>
labels: {}
path: /
tls:
memory: 4Gi
rootPassword: <redacted>
rootUser: <redacted>
-
CSI Version: default installed
-
Kubernetes Version: 1.24.4
-
Cloud provider/framework version, if applicable (such as Rancher):
sjdrc commented
I opened a support ticket and they increased the backend limits on my account.