service/s3: S3 does not appear to support path-style addressing

Question

service/s3: S3 does not appear to support path-style addressing

Closed this issue 3 months ago · 1 comments

Acknowledgements

I have searched (https://github.com/aws/aws-sdk/issues?q=is%3Aissue) for past instances of this issue
I have verified all of my SDK modules are up-to-date (you can perform a bulk update with go get -u github.com/aws/aws-sdk-go-v2/...)

Describe the bug

Let's assume bar.example.com is an S3 base endpoint that contains a bucket called foo. I can make requests against the bar bucket using virtual-style addressing at foo.bar.example.com, or by using path-style addressing at bar.example.com/foo. When I search for "S3 Custom command settings" on this page I see that I can switch between the two by setting

# ~/.aws/config
[profile bar]
s3 =
  addressing_style = path

This does not seem to work for the Go SDK.

Moreover, when set to auto, both the AWS CLI and boto3 are smart enough to figure out that they need to use path-style addressing because virtual-style addressing results in hitting a domain with a self-signed cert (the provider I'm using, which is a popular 3rd party, does not have a wildcard cert for their buckets). The Go SDK is unique in that it just fails with an invalid cert error.

Expected Behavior

I would expect the Go SDK to behave like boto3 and the AWS CLI. If you do a good job of fixing this issue, maybe AWS will let you rewrite the CLI in Go and then it can be fast.

Current Behavior

Not only does it not use path-style addressing when set to auto, it refuses to use path-style addressing when explicitly configured to use path-style addressing.

Reproduction Steps

	cfg, err := config.LoadDefaultConfig(context.TODO())
	if err != nil {
		panic(err)
	}
	// Doesn't matter if I set this here or in ~/.aws/config
	*cfg.BaseEndpoint = "https://bar.example.com"
	client := s3.NewFromConfig(cfg)

	bucket := "foo"
	prefix := "/"
	res, err := client.ListObjectsV2(context.TODO(), &s3.ListObjectsV2Input{
		Bucket: &bucket,
		Prefix: &prefix,
	})
	if err != nil {
		var oe *smithy.OperationError
		if errors.As(err, &oe) {
			log.Printf("failed to call service: %s, operation: %s, error: %v", oe.Service(), oe.Operation(), oe.Unwrap())
		}
	}

I'm getting an error in this example because of the self-signed cert, but you'll have the same problem with S3 if you wireshark your network traffic.

Possible Solution

No response

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

$ go mod graph
# ...
github.com/aws/aws-sdk-go-v2@v1.26.0 github.com/aws/smithy-go@v1.20.1
github.com/aws/aws-sdk-go-v2@v1.26.0 github.com/jmespath/go-jmespath@v0.4.0
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream@v1.6.1 github.com/aws/smithy-go@v1.20.1
github.com/aws/aws-sdk-go-v2/config@v1.27.9 github.com/aws/aws-sdk-go-v2@v1.26.0
github.com/aws/aws-sdk-go-v2/config@v1.27.9 github.com/aws/aws-sdk-go-v2/credentials@v1.17.9
github.com/aws/aws-sdk-go-v2/config@v1.27.9 github.com/aws/aws-sdk-go-v2/feature/ec2/imds@v1.16.0
github.com/aws/aws-sdk-go-v2/config@v1.27.9 github.com/aws/aws-sdk-go-v2/internal/ini@v1.8.0
github.com/aws/aws-sdk-go-v2/config@v1.27.9 github.com/aws/aws-sdk-go-v2/service/sso@v1.20.3
github.com/aws/aws-sdk-go-v2/config@v1.27.9 github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.23.3
github.com/aws/aws-sdk-go-v2/config@v1.27.9 github.com/aws/aws-sdk-go-v2/service/sts@v1.28.5
github.com/aws/aws-sdk-go-v2/config@v1.27.9 github.com/aws/smithy-go@v1.20.1
github.com/aws/aws-sdk-go-v2/config@v1.27.9 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.4
github.com/aws/aws-sdk-go-v2/config@v1.27.9 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.4
github.com/aws/aws-sdk-go-v2/config@v1.27.9 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.11.1
github.com/aws/aws-sdk-go-v2/config@v1.27.9 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.11.6
github.com/aws/aws-sdk-go-v2/credentials@v1.17.9 github.com/aws/aws-sdk-go-v2@v1.26.0
github.com/aws/aws-sdk-go-v2/credentials@v1.17.9 github.com/aws/aws-sdk-go-v2/feature/ec2/imds@v1.16.0
github.com/aws/aws-sdk-go-v2/credentials@v1.17.9 github.com/aws/aws-sdk-go-v2/service/sso@v1.20.3
github.com/aws/aws-sdk-go-v2/credentials@v1.17.9 github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.23.3
github.com/aws/aws-sdk-go-v2/credentials@v1.17.9 github.com/aws/aws-sdk-go-v2/service/sts@v1.28.5
github.com/aws/aws-sdk-go-v2/credentials@v1.17.9 github.com/aws/smithy-go@v1.20.1
github.com/aws/aws-sdk-go-v2/credentials@v1.17.9 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.4
github.com/aws/aws-sdk-go-v2/credentials@v1.17.9 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.4
github.com/aws/aws-sdk-go-v2/credentials@v1.17.9 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.11.1
github.com/aws/aws-sdk-go-v2/credentials@v1.17.9 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.11.6
github.com/aws/aws-sdk-go-v2/feature/ec2/imds@v1.16.0 github.com/aws/aws-sdk-go-v2@v1.26.0
github.com/aws/aws-sdk-go-v2/feature/ec2/imds@v1.16.0 github.com/aws/smithy-go@v1.20.1
github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.4 github.com/aws/aws-sdk-go-v2@v1.26.0
github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.4 github.com/aws/smithy-go@v1.20.1
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.4 github.com/aws/aws-sdk-go-v2@v1.26.0
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.4 github.com/aws/smithy-go@v1.20.1
github.com/aws/aws-sdk-go-v2/internal/v4a@v1.3.4 github.com/aws/aws-sdk-go-v2@v1.26.0
github.com/aws/aws-sdk-go-v2/internal/v4a@v1.3.4 github.com/aws/smithy-go@v1.20.1
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.11.1 github.com/aws/smithy-go@v1.20.1
github.com/aws/aws-sdk-go-v2/service/internal/checksum@v1.3.6 github.com/aws/aws-sdk-go-v2@v1.26.0
github.com/aws/aws-sdk-go-v2/service/internal/checksum@v1.3.6 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.11.6
github.com/aws/aws-sdk-go-v2/service/internal/checksum@v1.3.6 github.com/aws/smithy-go@v1.20.1
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.11.6 github.com/aws/aws-sdk-go-v2@v1.26.0
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.11.6 github.com/aws/smithy-go@v1.20.1
github.com/aws/aws-sdk-go-v2/service/internal/s3shared@v1.17.4 github.com/aws/aws-sdk-go-v2@v1.26.0
github.com/aws/aws-sdk-go-v2/service/internal/s3shared@v1.17.4 github.com/aws/smithy-go@v1.20.1
github.com/aws/aws-sdk-go-v2/service/s3@v1.53.0 github.com/aws/aws-sdk-go-v2@v1.26.0
github.com/aws/aws-sdk-go-v2/service/s3@v1.53.0 github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream@v1.6.1
github.com/aws/aws-sdk-go-v2/service/s3@v1.53.0 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.4
github.com/aws/aws-sdk-go-v2/service/s3@v1.53.0 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.4
github.com/aws/aws-sdk-go-v2/service/s3@v1.53.0 github.com/aws/aws-sdk-go-v2/internal/v4a@v1.3.4
github.com/aws/aws-sdk-go-v2/service/s3@v1.53.0 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.11.1
github.com/aws/aws-sdk-go-v2/service/s3@v1.53.0 github.com/aws/aws-sdk-go-v2/service/internal/checksum@v1.3.6
github.com/aws/aws-sdk-go-v2/service/s3@v1.53.0 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.11.6
github.com/aws/aws-sdk-go-v2/service/s3@v1.53.0 github.com/aws/aws-sdk-go-v2/service/internal/s3shared@v1.17.4
github.com/aws/aws-sdk-go-v2/service/s3@v1.53.0 github.com/aws/smithy-go@v1.20.1
github.com/aws/aws-sdk-go-v2/service/sso@v1.20.3 github.com/aws/aws-sdk-go-v2@v1.26.0
github.com/aws/aws-sdk-go-v2/service/sso@v1.20.3 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.4
github.com/aws/aws-sdk-go-v2/service/sso@v1.20.3 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.4
github.com/aws/aws-sdk-go-v2/service/sso@v1.20.3 github.com/aws/smithy-go@v1.20.1
github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.23.3 github.com/aws/aws-sdk-go-v2@v1.26.0
github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.23.3 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.4
github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.23.3 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.4
github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.23.3 github.com/aws/smithy-go@v1.20.1
github.com/aws/aws-sdk-go-v2/service/sts@v1.28.5 github.com/aws/aws-sdk-go-v2@v1.26.0
github.com/aws/aws-sdk-go-v2/service/sts@v1.28.5 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.4
github.com/aws/aws-sdk-go-v2/service/sts@v1.28.5 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.4
github.com/aws/aws-sdk-go-v2/service/sts@v1.28.5 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.11.1
github.com/aws/aws-sdk-go-v2/service/sts@v1.28.5 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.11.6
github.com/aws/aws-sdk-go-v2/service/sts@v1.28.5 github.com/aws/smithy-go@v1.20.1
go@1.22.1 toolchain@go1.22.1

Compiler and Version used

go version go1.22.1 linux/amd64

Operating System and version

6.8.1-arch1-1

Answer 1 · 2024-03-25T15:20:46.000Z

The documentation link you have there is specific to the CLI -- other SDKs to my knowledge, including Go v2 (and excluding botocore because that's what the CLI itself uses) do not support the s3.addressing_style shared config setting, or anything under the s3 top-level key.

The Go v2 SDK does support this behavior though, you can enable it in client config:

client := s3.NewFromConfig(cfg, func (o *s3.Options) {
    o.UsePathStyle = true
})