Add capability for S3 bucket users to choose between Path-Style Access and Virtual-hosted–style access
Closed this issue · 4 comments
Is your feature request related to a problem? Please describe.
Path-style access is a way of specifying the location of a bucket or object in Amazon S3 using a URL-like syntax. This is in contrast to virtual-hosted style access, which uses the bucket name as a subdomain in the URL. For example:
Virtual-hosted style: https://bucket-name.s3.amazonaws.com/object-key
Path-style: https://s3.amazonaws.com/bucket-name/object-key
In virtual-hosted style access, the bucket name must be a valid DNS name, which means it must not contain dots. This can be a problem if you need to access a bucket that has dots in its name. In addition, some regions may only support path-style access.
Describe the solution you'd like
- Add enablePathStyleAccess property to S3StorageCreateRequest proto file
- Make the property enablePathStyleAccess default to false. A client that wants to use path style access will set the value to true
- In the S3Util.java class, add the method withPathStyleAccessEnabled(boolean) to the AmazonS3ClientBuilder object.
Additional context
Some buckets are not valid DNS names. Setting this flag to true will result in path-style access being used for all requests.
@ushaheu thanks for the new feature request. We need to make changes in a few places to support this
- Add the pathStyleAccess boolean property at https://github.com/apache/airavata-mft/blob/master/services/resource-service/stub/src/main/proto/s3/S3Storage.proto#L23
- Do the same at https://github.com/apache/airavata-mft/blob/master/services/resource-service/server/src/main/java/org/apache/airavata/mft/resource/server/backend/sql/entity/S3StorageEntity.java#L28
- Set the pathStyleAccess to s3 client based on the property set in above protobuf structure https://github.com/apache/airavata-mft/blob/master/transport/s3-transport/src/main/java/org/apache/airavata/mft/transport/s3/S3Util.java#L100
@DImuthuUpe thanks for sharing this. I will raise a PR for this and share soon.
Opened a PR #111
Closing this as the PR was applied. Please open if the issue persists