aws/aws-sdk

Override service endpoint URL through configuration files/environment variables

kellertk opened this issue · 18 comments

kellertk commented

Summary
It would be nice to be able to override a service endpoint URL through configuration files, environment variables, or some other runtime mechanism instead of only in code.

Description
This issue de-duplicates several issues and pull requests from other AWS SDKs and various tools, specifically:

This Stack Overflow post also describes the feature request: https://stackoverflow.com/questions/32618216/override-s3-endpoint-using-boto3-configuration-file.

First of all, we’d like to thank everyone for their feedback and patience while we examine these requests and determine how to move forward. Up front, there is an assumption in some of the threads linked above that AWS doesn’t want to implement this because it encourages reuse of the SDK for non-AWS products:

Is it possible that this PR is not merged just because it allows developers to use other S3-like storage?
In that case, there is strong a conflict of interest between AWS services and the community. If I am right, this PR will never be merged in foreseeable future.

This is not the case, and we want to be clear that it is not a conflict of interest preventing the merging of that specific pull request. As @CodePint mentioned, “The functionality is already already exposed via the endpoint_url parameter which can be passed to the boto3 client function, the ability to configure it using an env var is purely an extension of that.” In the linked Stack Overflow post, a few commenters posted the same thing. When instantiating a service client in a recent-ish version of the SDK, you can specify the endpoint URL for that service. This feature request doesn’t change that functionality, but instead extends it through the use of a configuration file or environment variable.

So again, today everyone should be able to specify an endpoint URL for a service, but it needs to be done in the service client instantiation. This feature request is to explore methods that might make this more convenient or more ergonomic.

Implementation Challenges
A handful of issues have been opened across AWS SDKs requesting the ability to customize an endpoint URL either via an environment variable or profile configuration. There are many trade-offs to consider here that make the implementation of this change more challenging than it might seem. For example, a maintainer described in this comment how:

The thing with --endpoint-url is that the value set will most likely not apply across differing services. So if you configure a global endpoint url, it may work for one service, but you will run into errors if you try to make requests to other services.

This implementation detail is one item we need to consider (and configuration file management in general), but there are others. Before moving forward, we must consider how this feature will work across SDKs to ensure a consistent experience and maintain backwards compatibility. One important point is that we don’t want to implement this in only one of the SDKs, or only in the AWS CLI. In general, all of the AWS tools follow the same configuration file/environment variable syntax and we want to keep it that way - as @kapilt suggested, this change would need to be coordinated across the SDKs for consistency.

Next Steps
We acknowledge the demand for this feature and are actively discussing the options internally. Expect an update on this thread within a week with our progress on our decisions here. I can promise that we won’t have this done in a week - but I can also promise that by that time we’ll have had our internal discussion and be able to present some options for further discussion with the community. Anything that I can share, I will here.

We use 👍 to track community engagement on issues, so if this issue is relevant to you, please react with a 👍 emoji. Some of the linked threads above have a substantial number of reactions, which does help us gauge this feature’s important to our customers. If you’d like, you can also post more specific feedback relating to how you’d use a configuration file/environment managed endpoints and why the current method of specifying this info in the service client instantiation doesn’t work for you. We’ll take that into consideration as we design the feature.

flickerfly commented

those guys who were interested in the previous issue are dead now

"I'm not dead yet." - Monty Python

  • 😄10
codyhazelwood commented

I would be happy to see this implemented as environment variables. We currently do all of our AWS configuration with environment variables in development. Being able to set the endpoint there would allow us to call Lambdas in local SAM and S3 buckets with local MinIO, while reverting to the official AWS endpoints in prod with no code changes.

Of course, we can use a wrapper around the specific SDKs, but that isn't as nice as being able to directly import the SDK without thinking about it.

  • 👍6
kdaily commented

Hi all,

Thanks for your patience. I just linked a pull request (#230) that contains a proposal based on community comments and suggestions and our own discussions. This document proposes to extend the options for configuring the endpoint to allow users to provide an endpoint URL independently for each AWS service via an environment variable or a profile subsection in the shared configuration file.

You can skip directly to read the proposal here.

🗒️ Providing feedback

If you're already here and have high level feedback, we'll read and review and comments directly on this issue. If you have feedback on specific parts of the proposal, we'd appreciate if you provide feedback via the GitHub pull request by commenting on the lines where you have questions or suggestions.

After reading the proposal and comments, you can give feedback on the pull request in the following ways:

  1. If you generally approve of the direction, add 👍 reaction to top-level comment.
  2. If you prefer one of the alternatives listed, comment on the line where an alternative is proposed that you prefer. If there is already a comment for that alternative, add a 👍 to that comment.
  3. If there are any other suggestions/alternatives that you'd prefer, add a comment describing the suggestion and alternative. 👍 the comment if you agree with it.
  4. Comment with any additional questions.

Thank you!

  • 👍41
johnspurlock commented

I was really surprised to see there was still no corresponding option in the aws config file equivalent to the --endpoint-url command line arg in the aws cli, in 2022! Here's hoping it gets added soon.

The proposal in the previous comment (Shared configuration file) looks great!

  • ❤️8
e-compton commented

The global endpoint for all services would be a really useful addition for us. Having the ability to set a local url as the default massively reduces the risk of accidentally modifying real AWS resources when developing locally.

  • 👍12
martimors commented

I should be able to just set AWS_S3_ENDPOINT_URL in the environment to point to a non-AWS endpoint. It would make applications that were not designed for non-AWS S3 endpoints (like minio) be compatible by default.

  • 👍6
  • 🚀4
flickerfly commented

It doesn't have to be non-AWS endpoints. It is a problem of VPC endpoints that are provided by AWS for a variety of services.

  • 👍3
nlongton commented

I should be able to just set AWS_S3_ENDPOINT_URL in the environment to point to a non-AWS endpoint. It would make applications that were not designed for non-AWS S3 endpoints (like minio) be compatible by default.

We also need to disable ssl verification as the certificate doesnt match the endpoint URL

srgg commented

Is there any progress on that?

  • 👀16
j-slater commented

My use case involves using private VPC endpoints for AWS S3, which will be different depending on environment. Per AWS documentation, we are required to override S3 service endpoint URL to accomplish this.

theelderbeever commented

This feels like a really simple problem for it to be taking >8 years to accomplish...

For now I just use a bash script that I alias as aws. Comprehensive? No. Solves the problem that Amazon seem unwilling to get around to. Yes.

#!/bin/bash

if [[ -z "${AWS_ENDPOINT_URL}" ]]; then
    /opt/homebrew/bin/aws $@
else
    /opt/homebrew/bin/aws --endpoint-url ${AWS_ENDPOINT_URL} $@
fi

@e-compton @dingobar Maybe this at least smooths things over for you two.

  • ❤️7
markuszeller commented

If I can have different profiles why not have a default endpoint for that profile? If one is given at the command line, then it could override the profile. It nags me to put it always in.

vschwaberow commented

What a joke.

  • 👍2
  • 👎2
tim-finnigan commented

Wanted to share @kdaily's updates that this has been implemented in the AWS CLI and Python SDK Boto3:

I'm happy to announce that the ability to configure the endpoint URL via the shared configuration file and environment variables is now available in the AWS CLI v1 and v2 [and Python SDK]! You can now specify the endpoint to use for all service requests through the shared configuration file and environment variables, as well as specify the endpoint URL for individual AWS services.

To start using this feature, install the AWS CLI >=1.29.0 or >=2.13.0. [For Boto3: To start using this feature, install boto3>=1.28.0.]

To read more about this feature, see the documentation page "Service-specific Endpoints" in the AWS SDKs and Tools Reference Guide:

https://docs.aws.amazon.com/sdkref/latest/guide/feature-ss-endpoints.html

Look forward to a blog post demonstrating the use of this feature with the AWS CLI on the AWS Developer Tools Blog!

  • 👍12
  • 😄1
  • 🎉7
  • ❤️12
kdaily commented

I'm excited to announce that we have published an AWS Developer Blog post about this feature. Let us know in the feedback links on the post what you think!

https://aws.amazon.com/blogs/developer/new-improved-flexibility-when-configuring-endpoint-urls-with-the-aws-sdks-and-tools/

  • 👍10
  • ❤️5
github-actions commented

This issue is now closed.

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

fkjaekel commented

What about the Java SDK?

  • 👍2
joarobles commented

What about the Java SDK?

Asking the real question here