Async Delete Always Failed when Removing Experiments (using Minio)

[agfianf]

Hi, guys

i have settings services.conf in /opt/clearml/config to able delete minio files too if i delete experiment on clearml. here my settings:

storage_credentials{
    aws {
        s3 {
            # S3 credentials, used for read/write access by various SDK elements

            # default, used for any bucket not specified below
            key: "key_server_2"
            secret: "zNAYleEDeCnlzaXJsd7MvXnQhPmZehIA"
            region: "company-server-2"
            use_credentials_chain: true

            credentials: [
                {
                     # This will apply to all buckets in this host (unless key/value is specifically provided for a given bucket)
                    host: "my-ip:9000"
                    key: "key_server_2"
                    secret: "zNAYleEDeCnlzaXJsd7MvXnQhPmZehIA"
                    region: "company-server-2"
                    multipart: false
                    secure: false
                }
            ]
        }
    }
}

this is log from async_delete container

[2024-01-20 05:59:28,700] [1] [INFO] [clearml.redis_manager] Using override redis host redis
[2024-01-20 05:59:28,700] [1] [INFO] [clearml.redis_manager] Using override redis port 6379
[2024-01-20 05:59:28,711] [1] [INFO] [clearml.database] Initializing database connections
[2024-01-20 05:59:28,711] [1] [INFO] [clearml.database] Using override mongodb host mongo
[2024-01-20 05:59:28,711] [1] [INFO] [clearml.database] Using override mongodb port 27017
[2024-01-20 05:59:28,712] [1] [INFO] [clearml.database] Registering connection to auth-db (mongodb://mongo:27017/auth)
[2024-01-20 05:59:28,713] [1] [INFO] [clearml.database] Registering connection to backend-db (mongodb://mongo:27017/backend)
[2024-01-20 06:01:58,882] [1] [INFO] [clearml.JOB-async_urls_delete.py] Deleting s3 objects for company: d1bd92a3b039400cbafc60a7a5b1e52b, user: a174c4e36b0446a7b3b5dd1ff5261962
[2024-01-20 06:01:58,999] [1] [WARNING] [clearml.JOB-async_urls_delete.py] Failed to delete 5 files from AWS storage due to: The authorization header is malformed; the region is wrong; expecting 'company-server-2'.
[2024-01-20 06:02:59,065] [1] [INFO] [clearml.JOB-async_urls_delete.py] Deleting s3 objects for company: d1bd92a3b039400cbafc60a7a5b1e52b, user: a174c4e36b0446a7b3b5dd1ff5261962
[2024-01-20 06:02:59,093] [1] [WARNING] [clearml.JOB-async_urls_delete.py] Failed to delete 5 files from AWS storage due to: The authorization header is malformed; the region is wrong; expecting 'company-server-2'.
[2024-01-20 06:03:59,153] [1] [INFO] [clearml.JOB-async_urls_delete.py] Deleting s3 objects for company: d1bd92a3b039400cbafc60a7a5b1e52b, user: a174c4e36b0446a7b3b5dd1ff5261962
[2024-01-20 06:03:59,180] [1] [WARNING] [clearml.JOB-async_urls_delete.py] Failed to delete 5 files from AWS storage due to: The authorization header is malformed; the region is wrong; expecting 'company-server-2'.

is there anything i wrote false in configuration?
Thanks in advance!