minio/console

Can't remove empty bucket via console

Closed this issue · 7 comments

AlexZIX commented

I can't remove the empty bucket via web console. When tries it returned following error:

Resource requested is unwritable, please reduce your request rate.

This is trace output:

2024-03-04T12:39:16.448 [503 Service Unavailable] s3.DeleteBucket minio-cold-1.bazissoft.ru:9001/videos/ 192.168.158.6 239.429ms ↑ 114 B ↓ 344 B

At the same time my minio haven't high load and other requests like GetObject or PutObject works perfectly:

2024-03-04T12:42:36.338 [200 OK] s3.PutObject minio-cold-1.bazissoft.ru:9001/zix/Video MP4_Bike - testfile.org.mp4 192.168.158.6 19.762484s ↑ 228 MiB ↓ 0 B

I've tried to push the delete button many times and always get 503 error. So it seems that issue aren't related to cluster load,

Your Environment

  • MinIO version used (minio --version): minio version RELEASE.2024-02-24T17-11-14Z (commit-id=c2b54d92f66ea02642be4ede26f2b7ecac1f2056)
  • Server setup and configuration: 4 nodes with 4 disks on each
  • Operating System and version (uname -a): Linux minio-cold-1 5.15.0-69-generic #76-Ubuntu SMP Fri Mar 17 17:19:29 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
prakashsvmx commented

I tried with latest version. it is working fine.

how is minio deployed ? how many nodes/drives?

is the bucket versioning enabled ?

are the server logs/ dmsg checked on nodes/drives?

AlexZIX commented

@prakashsvmx After some testing I can say that this problem is only with few buckets. Some buckets removes correctly and few returns such error. It's not too old buckets - was created 2-3 weeks ago,
My minio is 4 nodes with 4 disks on each node. One node is in shutdown mode now because of maintanance. Can it lead the issue?
Bucket versioning aren't enabled.
I don't found any interesting in syslog/dmesg which related to this issue.
And I still can't remove few buckets.

prakashsvmx commented

is the backed disk permission/user and time same on all nodes ?

AlexZIX commented

Yes, we have 4 same nodes with the same configurations. I've tried to connect to another node's web ui and remove this bucket and also got the same error.
No I found that few files also can't be removed and returns error 500:

{
"detailedMessage": "Resource requested is unwritable, please reduce your request rate",
"message": "an error occurred, please try again"
}

Error returns immediately after push the remove button.

AlexZIX commented

May be this output will helps. I made this steps:

  1. Enter to the bucket.
  2. Enter to the its settings.
  3. Push the remove button.

^Croot@minio-cold-1:~# mc admin trace -a --funcname admin* --funcname sts* --funcname s3* minio-cold | grep videos
2024-03-04T20:06:28.498 [200 OK] s3.GetBucketLocation minio-cold-1.bazissoft.ru:9001/videos/?location= 192.168.158.6 37.662ms ↑ 114 B ↓ 128 B
2024-03-04T20:06:28.499 [200 OK] admin.GetBucketQuotaConfig minio-cold-1.bazissoft.ru:9001/minio/admin/v3/get-bucket-quota?bucket=videos 192.168.158.6 37.432ms ↑ 114 B ↓ 42 B
2024-03-04T20:06:28.536 [404 Not Found] s3.GetBucketObjectLockConfig minio-cold-1.bazissoft.ru:9001/videos/?object-lock= 192.168.158.6 455µs ↑ 114 B ↓ 358 B
2024-03-04T20:06:28.500 [200 OK] s3.GetBucketLocation minio-cold-1.bazissoft.ru:9001/videos/?location= 192.168.158.6 38.769ms ↑ 114 B ↓ 128 B
2024-03-04T20:06:28.502 [200 OK] s3.GetBucketLocation minio-cold-1.bazissoft.ru:9001/videos/?location= 192.168.158.6 39.166ms ↑ 114 B ↓ 128 B
2024-03-04T20:06:28.501 [200 OK] s3.GetBucketLocation minio-cold-1.bazissoft.ru:9001/videos/?location= 192.168.158.6 40.012ms ↑ 114 B ↓ 128 B
2024-03-04T20:06:28.542 [404 Not Found] s3.GetBucketObjectLockConfig minio-cold-1.bazissoft.ru:9001/videos/?object-lock= 192.168.158.6 528µs ↑ 114 B ↓ 358 B
2024-03-04T20:06:28.540 [200 OK] s3.GetBucketPolicy minio-cold-1.bazissoft.ru:9001/videos/?policy= 192.168.158.6 39.062ms ↑ 114 B ↓ 417 B
2024-03-04T20:06:28.581 [404 Not Found] s3.GetBucketTagging minio-cold-1.bazissoft.ru:9001/videos/?tagging= 192.168.158.6 384µs ↑ 114 B ↓ 303 B
2024-03-04T20:06:28.542 [200 OK] s3.GetBucketVersioning minio-cold-1.bazissoft.ru:9001/videos/?versioning= 192.168.158.6 39.321ms ↑ 114 B ↓ 99 B
2024-03-04T20:06:39.823 [200 OK] s3.GetBucketLocation minio-cold-1.bazissoft.ru:9001/videos/?location= 192.168.158.6 32.279ms ↑ 114 B ↓ 128 B
2024-03-04T20:06:39.837 [200 OK] s3.GetBucketLocation minio-cold-1.bazissoft.ru:9001/videos/?location= 192.168.158.6 32.188ms ↑ 114 B ↓ 128 B
2024-03-04T20:06:39.836 [200 OK] s3.GetBucketLocation minio-cold-1.bazissoft.ru:9001/videos/?location= 192.168.158.6 32.463ms ↑ 114 B ↓ 128 B
2024-03-04T20:06:39.837 [200 OK] admin.GetBucketQuotaConfig minio-cold-1.bazissoft.ru:9001/minio/admin/v3/get-bucket-quota?bucket=videos 192.168.158.6 32.902ms ↑ 114 B ↓ 42 B
2024-03-04T20:06:39.870 [404 Not Found] s3.GetBucketObjectLockConfig minio-cold-1.bazissoft.ru:9001/videos/?object-lock= 192.168.158.6 580µs ↑ 114 B ↓ 358 B
2024-03-04T20:06:39.842 [200 OK] s3.GetBucketLocation minio-cold-1.bazissoft.ru:9001/videos/?location= 192.168.158.6 32.535ms ↑ 114 B ↓ 128 B
2024-03-04T20:06:39.857 [200 OK] s3.GetBucketPolicy minio-cold-1.bazissoft.ru:9001/videos/?policy= 192.168.158.6 39.836ms ↑ 114 B ↓ 417 B
2024-03-04T20:06:39.898 [404 Not Found] s3.GetBucketTagging minio-cold-1.bazissoft.ru:9001/videos/?tagging= 192.168.158.6 619µs ↑ 114 B ↓ 303 B
2024-03-04T20:06:39.870 [200 OK] s3.GetBucketVersioning minio-cold-1.bazissoft.ru:9001/videos/?versioning= 192.168.158.6 34.456ms ↑ 114 B ↓ 99 B
2024-03-04T20:06:39.876 [404 Not Found] s3.GetBucketEncryption minio-cold-1.bazissoft.ru:9001/videos/?encryption= 192.168.158.6 31.376ms ↑ 114 B ↓ 366 B
2024-03-04T20:06:39.886 [200 OK] s3.GetBucketLocation minio-cold-1.bazissoft.ru:9001/videos/?location= 192.168.158.6 30.782ms ↑ 114 B ↓ 128 B
2024-03-04T20:06:39.906 [200 OK] s3.GetBucketLocation minio-cold-1.bazissoft.ru:9001/videos/?location= 192.168.158.6 38.243ms ↑ 114 B ↓ 128 B
2024-03-04T20:06:39.946 [404 Not Found] s3.GetBucketObjectLockConfig minio-cold-1.bazissoft.ru:9001/videos/?object-lock= 192.168.158.6 799µs ↑ 114 B ↓ 358 B
2024-03-04T20:06:39.918 [404 Not Found] s3.GetBucketReplicationConfig minio-cold-1.bazissoft.ru:9001/videos/?replication= 192.168.158.6 32.378ms ↑ 114 B ↓ 346 B
2024-03-04T20:06:40.991 [200 OK] s3.GetBucketLocation minio-cold-1.bazissoft.ru:9001/videos/?location= 192.168.158.6 38.918ms ↑ 114 B ↓ 128 B
2024-03-04T20:06:41.031 [200 OK] s3.GetBucketPolicy minio-cold-1.bazissoft.ru:9001/videos/?policy= 192.168.158.6 41.428ms ↑ 114 B ↓ 417 B
2024-03-04T20:06:41.074 [404 Not Found] s3.GetBucketTagging minio-cold-1.bazissoft.ru:9001/videos/?tagging= 192.168.158.6 408µs ↑ 114 B ↓ 303 B
2024-03-04T20:06:44.054 [200 OK] s3.GetBucketLocation minio-cold-1.bazissoft.ru:9001/videos/?location= 192.168.158.6 40.277ms ↑ 114 B ↓ 128 B
2024-03-04T20:06:44.095 [503 Service Unavailable] s3.DeleteBucket minio-cold-1.bazissoft.ru:9001/videos/ 192.168.158.6 229.762ms ↑ 114 B ↓ 344 B

AlexZIX commented

One more interesting think I see. Console shows me that bucket is empty, but physically data exists on disks. Is that normal?

image

AlexZIX commented

Today I've upgraded my minio to version 2024-03-15T01:07:19Z and issue becomes solved. I've simply remove this bucket with no errors.