Minio 6.0.2: Unexpected Behavior with Unavailable Endpoints

[teun2408]

Issue Summary:

Upgrading from Minio .NET SDK version 6.0.1 to 6.0.2 has introduced unexpected behavior when interacting with non-existent Minio server endpoints. Specifically, methods like BucketExistsAsync and GetObjectAsync do not behave as expected when provided with endpoints that are not running Minio servers. Instead of throwing exceptions or indicating errors, they return unexpected results, such as always returning true for bucket existence checks and returning empty streams for object downloads.

Steps to Reproduce:

1. Use Minio .NET SDK version 6.0.2.
2. Configure the SDK with an endpoint that does not host a Minio server (e.g., "127.0.0.1:11111").
3. Use the BucketExistsAsync method to check for the existence of a bucket that doesn't exist.
4. Use the GetObjectAsync method to download an object from a non-existent bucket.

using Minio;

var s3Client = new MinioClient()
    .WithEndpoint("127.0.0.1:11111")
    .WithCredentials("accesskey", "secretkey")
    .Build();

//Should throw an exception, but it does not in 6.0.2, in 6.0.1 it throws a System.AggregateException
//Instead, in 6.0.2 it just returns true even though the Minio server does not exist, nor the bucket.
bool bucketExists = s3Client.BucketExistsAsync(new Minio.DataModel.Args.BucketExistsArgs().WithBucket("bucketname")).Result;

if (bucketExists)
{
    Console.WriteLine("Bucket exists according to Minio");
}

MemoryStream stream = new MemoryStream();

// An exception should be thrown here, but it is not, instead, it returns an empty stream even though neither the server, bucket, or object exists.
await s3Client.GetObjectAsync(new Minio.DataModel.Args.GetObjectArgs().WithBucket("bucketname").WithObject("objectname").WithCallbackStream(x => x.CopyTo(stream)));

Console.WriteLine($"Stream length: {stream.Length}");

Expected Behavior:

• The BucketExistsAsync method should throw an exception or return false when provided with an endpoint that does not host a Minio server.
• The GetObjectAsync method should throw an exception when attempting to download an object from a non-existent bucket.

Actual Behavior:

• When using BucketExistsAsync with a non-existent Minio server endpoint, it consistently returns true, incorrectly indicating the existence of the bucket.
• When using GetObjectAsync with a non-existent bucket or object, it returns an empty stream instead of throwing an exception or indicating an error.

Environment Details:

• Minio .NET SDK version: 6.0.2
• Operating System: Windows 11
• .NET 8

[cvetomir-todorov]

Happens to me as well. Works with 6.0.1, breaks with 6.0.2. I am using Ubuntu 22.04 and .NET 8.

[OK111]

Also PutObjectAsync doesn't throw any errors in 6.0.2 .NET 8

[guillaumeagile]

The same. I tried to check the behavior of the driver with test containers, and got the same issue. I finally realized that the problem doesn't come from TestContainers, but from the driver.

[guillaumeagile]

Observed exactly that weird behavior when the endpoint is unreachable
like if the driver was talking to a "ghost" server that says yes to everything and that acts like a black hole when you upload to it.

Using .Net 8, TestContainers , xUnit and 6.0.2
service.AddMinio(configureClient => configureClient .WithEndpoint(uri) .WithCredentials(MinIoAdapter.AccessKey, MinIoAdapter.SecretKey) .WithSSL(false) .Build() );
where uri is not pointing to a running Minio instance.

await _minioClient.BucketExistsAsync(beArgs).ConfigureAwait(false);
or await _minioClient.PresignedGetObjectAsync(presignedGetObjectArgs).ConfigureAwait(false)
always returns something (with an empty container)

_minioClient.GetObjectAsync(getObjectArgs).ConfigureAwait(true)
always returns a 0 byte length memoryStream or file, even if I has uploaded an object before (with the right name)

I'm initializing my container with

_minioContainer = new MinioBuilder() .WithUsername(MinIoAdapter.AccessKey) .WithPassword(MinIoAdapter.SecretKey) .WithPortBinding(9000, true) .WithCleanUp(true) .Build();
it's also obvious when I remove the .WithUsername(MinIoAdapter.AccessKey) .WithPassword

[holterbades]

Imho this is a critical problem and this Issue should get the name "No error handling" because there is no actual error handling active anymore.

I encountered the following issues:

When i download a file with valid bucketName and objectName but the user/access-token does not have permission to download the object (e.g. due to a policy) or a wrong region was set in the client, no exception is thrown. So i can't verify that the download failed. The funny thing is, that instead of the object-content on the storage, the xml error response gets written to the output file.

var getObject = new GetObjectArgs()
    .WithBucket(bucketName)
    .WithObject(objectName)
    .WithFile("C:\\temp\\test2.txt");
var stat = await minio.GetObjectAsync(getObject);

Similar to this example, if i stream the object content directly to another destination (e.g. another S3 object store), i also get the error message content instead of an exception streamed to the destination. Furthermore, if i encounter an exception in the streaming action like outlined below, this exception is also catched and not delegated to the user code, which also makes it impossible to handle errors.

var getObject = new GetObjectArgs()
    .WithBucket(bucketName)
    .WithObject(objectName)
    .WithCallbackStream(stream =>
    {
        throw new Exception("Error uploading file to storage.");
    });
var stat = await minio.GetObjectAsync(getObject);

Imagine you use this action to stream the blob directly to an deep archive with retention for 7 years - in this case you get files with error messages instead of your intended object content - which you can't delete because of retention policies set on the target.

The only way you could verify that an error occoured is by validating the ObjectStat.ExtraHeaders returned by GetObjectAsync(...). But this clearly can't be a solution because it is provider dependent and would be a breaking change for clients out there.

I traced back the problem to changes in the commit 9772a373b92d81808118edbdbd653d301f4502ac, but I don't know what the intention for those changes was because it's not well documented neither in the commit nor in the linked issue (#987). But the problem is, that in this commit, the method call to HandleIfErrorResponse was unintentionally removed (this method has no other calls and if this removal was intended, the method HandleIfErrorResponse should have also been removed).

I opened a PullRequest #1057 to fix this issue.

[holterbades]

I think, the other problems mentioned in this issue should also get resolved with my fix, because this method ExecuteTaskAsync(...) -> ExecuteTaskCoreAsync(...) is the core method used by all operations.

[chrisddeh]

Hey.

has this been fixed? I've the same problem:
`public class MinioFilePersistence : IMinioFilePersistence, IDisposable
{
private readonly IMinioClient _minioClient;

    public MinioFilePersistence(MinioOptions options)
    {
        _minioClient = new MinioClient()
        .WithEndpoint(options.Endpoint + ":" + options.Port)
        .WithCredentials(options.AccessKey, options.SecretKey)
        .WithSSL(options.UseSSL)
        .Build();
    }

    public async Task StoreFile(MinioFile file)
    {
        _minioClient.SetTraceOn();
        var args = new PutObjectArgs()
            .WithBucket(file.Bucket)
            .WithObject(file.FileName)
            .WithStreamData(file.FileStream)
            .WithObjectSize(file.FileStream.Length)
            .WithContentType(file.ContentType);

        try
        {
            var result = await _minioClient.PutObjectAsync(args);

        }
        catch (Exception ex)
        {
            Console.WriteLine(ex.ToString());
        }
    }

    public void Dispose() {
        _minioClient.Dispose();
        GC.SuppressFinalize(this);
    }
}`

No exception is thrown with wrong credentials, when bucket does not exist, when I shutdown Minio...nothing.

Any suggestions?

[chrisddeh]

This line returns true, and I have shut down Minio completely:
bool found = await _minioClient.BucketExistsAsync(new BucketExistsArgs().WithBucket("9876")).ConfigureAwait(false);

:-(

[holterbades]

I also encountered the same problem when using version 6.0.3 again, but I honestly don't really want to investigate it again. The experience I had with the last PR was not very pleasant - the causer of this (in my oppinion severe) bug did not answer in a timely manner, is not answering this and the linked issues and blamed me for a failed unit test (where the reason was that the faulty changes on his side were the original issue). Furthermore this faulty NuGet package was out there for months so that many consumers updated to the buggy version without knowing it, resulting in many hours of research and probably lost data.

Since i saw the PR which lead to this issue I have no reason to trust this SDK. I no longer directly upload files with this anymore, I only use it to generate presigned URLs and other uncritical stuff and handle errors with an HttpClient on my own.

[chrisddeh]

ok thank you very much for your feedback. Then I'll do exactly the same as you. Fortunately, I have capped the library. I guess it will be a few hours of overtime for ChatGPT and me today :D

[holterbades]

Furthermore, I noticed there are preflight requests (HEAD-requests) with each request against the server, resulting in additional latency for each request (and maybe in increase costs if you use other S3 solutions which bill by request count). If you skip the SDK for the requests, you can efficiently skip those requests as well.

So only the "heavy lifting" of generating the signed URLs is done by the SDK - which is fine for my particular use-case (I really don't want to upload broken files when I can't retrieve them otherwise anymore).

[chrisddeh]

Thanks for the advice. I will probably do the same. Kind regards

[ebozduman]

@chrisddeh, @holterbades

This was a known issue in 6.0.2, but we think it was fixed in 6.0.3
We've tested the issue in 6.0.3 so many times and just right now, I've run a test with incorrect access key, with incorrect secret key and both keys incorrect and every time the test threw an exception correctly.

Minio.Exceptions.ErrorResponseException: MinIO API responded with message=HTTP status-code 403: Forbidden. Status code=Forbidden, response=The Access Key Id you provided does not exist in our records.

The request signature we calculated does not match the signature you provided. Check your key and signing method.

Please help us and give us more information so that we can reproduce the problem in those specific scenario(s) this issue still shows up.

[rhegner]

@ebozduman no unfortunately it is still not fixed and I've already opened a ticket for this (#1121) including test cases.

[ebozduman]

@rhegner ,
Thank you.
I'll make sure we address it right away.