awslabs/mountpoint-s3

Endpoint resolution & access points

jamesbornholt opened this issue · 7 comments

jamesbornholt commented

Currently our endpoint resolution logic is the very naive:

format!("{}.s3.{}.amazonaws.com", bucket, self.region);

But this doesn't support:

  • Dual-stack endpoints (IPv6)
  • FIPS endpoints
  • Transfer acceleration endpoints
  • Regions that don't end in amazonaws.com (e.g., China regions)
  • Access points
  • PrivateLink endpoints

It doesn't look like the CRT has any built-in support for dealing with these, so we might need to do it ourselves.

monthonk commented

Can we reuse some code from aws sdk rust? I see they have something like endpoint_resolver() in their code, so we don't have to implement it ourselves.

  • 👀2
jamesbornholt commented

Oh neat! That will save us from dealing with the other partitions, at least. We still need to figure out access points.

jamesbornholt commented

Thinking about it a bit more, there's roughly three things we need to do here:

  1. Correct endpoint resolution for regions outside the aws partition
  2. Allow the customer to provide an endpoint URL like the SDKs do—this should enable FIPS, IPv6, Transfer Acceleration, PrivateLink, and Multi-Region Access Points (and we should test these)
  3. Document that (single-region) Access Points can be used via access point aliases
jamesbornholt commented

Also need to test S3 on Outposts here.

jamesbornholt commented

#61 will let us specify custom endpoint URLs. But some of these things work without that:

  • Access points can be specified using bucket aliases
  • Gateway VPC endpoints are a networking-level thing and should just work

I've tested both these and they are working on mainline.

dannycjones commented

There's a little bit of experimental code on this branch which may be good to use as a reference point: https://github.com/awslabs/mountpoint-s3/tree/endpoint-rule-engine

  • 👍1
jamesbornholt commented

#428 wrapped this one up, except for testing, which is in #417.