s3 auth does not seem to support node / service account roles - access key and secret required

[mohag]

When configuring S3 access without an access key / secret (the AWS client libraries then obtain it from the instance metadata service / elsewhere) Sharry fails to start with this error:

2024-01-09T13:45:19.382067754Z     Using given config file: /opt/sharry.conf    
2024-01-09T13:45:20.408081990Z pureconfig.error.ConfigReaderException: Cannot convert configuration to a sharry.restserver.config.Config. Failures are:
2024-01-09T13:45:20.408107580Z   at 'sharry.restserver.backend.files.stores.s3':
2024-01-09T13:45:20.408111172Z     - (/opt/sharry.conf: 32) Key not found: 'access-key'.
2024-01-09T13:45:20.408113512Z     - (/opt/sharry.conf: 32) Key not found: 'secret-key'.

(In this case Sharry is running on Kubernetes (EKS) and has a service account with a role that allows access to the bucket specified)

Sharry seems to be using Binny, which is based on the Minio library.

The normal AWS libraries normally follow this process to get credentials to access services. (see link)

In this case I want to use the WebIdentityTokenFileCredentialsProvider one. Node roles uses InstanceProfileCredentialsProvider (which fetches it from the AWS metadata service)

The Min.io Java SDK does seem to at least have some code to support these auth options. I'm not sure how they can be used.

Ideally Sharry should follow the normal process as followed by AWS SDK apps to obtain credentials, or provide an option to do that. (env vars can be quite useful in container environments) (The IAM roles for service accounts that I'm trying to use use env vars with the role name and an env var with the path to the file containing the token that is mounted into the container) (This is probably needed via Binny though)

Separate "s3" and "minio" values for "type" might be an option as well...

Currently the only real workaround seems to be to create an IAM user with API credentials instead of using a role (via instance profile or service account roles)

[mohag]

Someone seems to have a working setting with the Go SDK:
https://stackoverflow.com/questions/77541009/minio-sdk-support-service-account-connectivity-in-aws-eks

The support was added to the Java SDK here: minio/minio-java#1038

It seems like the client builder can be passed credentials or a credentialsProvider: https://minio-java.min.io/io/minio/MinioClient.Builder.html#credentialsProvider-io.minio.credentials.Provider-

[eikek]

Thank you for the heads up! I'm sorry that it currently doesn't work well for your case. All these AWS details are very new to me, I'm not very familiar with EKS/AWS setups. If the minio library supports this, then binny should be able to make use of it. In the meantime, if env vars would work, you can reference env vars in the config file as well (see this: https://github.com/lightbend/config#optional-system-or-env-variable-overrides). That should allow to move credentials into the sharry config from environment variables.

[mohag]

Creating an IAM user with API credentials and using those credentials does work. (The roles assigned to the instance (or Kubernetes service account) is a bit nicer to manage - there is no credentials that might need manual rotation). Sharry works great! (That xkcd basically describes why I decided to deploy it)

It might be worth to attempt the AWS credential provider if no access key is given? (Adding a setting, say auth_mode that can be set to aws might be another option and it allows for potential other autodetection methods for e.g. GCP or Azure, which might also have compatible enough bucket stores, but other ways to get credentials automatically)

(The env vars won't work for instance / service account creds - for instance creds they are absent - creds are retrieved from the metadata service. For Kubernetes Service account roles, the env vars contains the role name and token that can be used to get the creds)