Support for running Quilt in an EKS pod with OpenID authentication to AWS disappeared/missing

[nmajubu]

Reproduce

Trying to use either the Quilt CLI or the NF-Quilt plugin from within a Kubernetes pod (EKS) which has a Service Account attached with an IAM Role annotation to read the corresponding S3 bucket (for NF-Quilt publishdir) throws "access denied":

• either for trying to publish results to own bucket (NF-Quilt),
• or just trying to install the example hurdat2 data package from the Quilt quick start guide with the command line command quilt3 install examples/hurdat2 --registry s3://quilt-example --dest data - See the traceback for this command below.

Using the AWS CLI, both the private bucket, and the Quilt example bucket with hurdat2 is accessible (private bucket through IAM roles, and Quilt example bucket is public).

Debug Observation

If I unset the AWS_WEB_IDENTITY_TOKEN_FILE environment variable, the installation of the example hurdat2 package completes successfully. - Of course, this breaks the AWS CLI authentication and thus the connection to our own private bucket but even the Quilt example bucket if using the AWS CLI and not the quilt3 install command. (This env var is needed for the AWS CLI to authenticate with from the Kubernetes cluster to AWS, indicating the location of the previously recieved token)

My suspicion is that the presence of this variable directs Boto3 on a path that is not handled correctly by Quilt and getting access denied even for the public repo - similarly to when no credentials are provided at all (?).

Quilt3 version: 5.3.1
Boto3 version: 1.26.141

Traceback for

quilt3 install examples/hurdat2 --registry s3://quilt-example --dest data
Traceback (most recent call last):
File "/home/coder/.local/bin/quilt3", line 8, in
sys.exit(main())
File "/home/coder/.local/lib/python3.8/site-packages/quilt3/main.py", line 481, in main
return func(**kwargs)
File "/home/coder/.local/lib/python3.8/site-packages/quilt3/telemetry.py", line 135, in decorated
results = func(*args, **kwargs)
File "/home/coder/.local/lib/python3.8/site-packages/quilt3/packages.py", line 535, in install
copy_file_list(
File "/home/coder/.local/lib/python3.8/site-packages/quilt3/data_transfer.py", line 785, in copy_file_list
return _copy_file_list_internal(file_list, [None] * len(file_list), message, callback)
File "/home/coder/.local/lib/python3.8/site-packages/tenacity/init.py", line 289, in wrapped_f
return self(f, *args, **kw)
File "/home/coder/.local/lib/python3.8/site-packages/tenacity/init.py", line 379, in call
do = self.iter(retry_state=retry_state)
File "/home/coder/.local/lib/python3.8/site-packages/tenacity/init.py", line 322, in iter
return self.retry_error_callback(retry_state)
File "/home/coder/.local/lib/python3.8/site-packages/quilt3/data_transfer.py", line 511, in _copy_file_list_last_retry
return retry_state.fn(
File "/home/coder/.local/lib/python3.8/site-packages/quilt3/data_transfer.py", line 613, in _copy_file_list_internal
future.result()
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
return self.__get_result()
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/coder/.local/lib/python3.8/site-packages/quilt3/data_transfer.py", line 375, in download_part
resp = s3_client.get_object(**part_params)
File "/home/coder/.local/lib/python3.8/site-packages/botocore/client.py", line 530, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/coder/.local/lib/python3.8/site-packages/botocore/client.py", line 964, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the GetObject operation: Access Denied

[nmajubu]

On Friday, 26th of May, I was able to run a Nextflow pipeline that had NF-Quilt plugin publishing some basic mock results within the same EKS environment. Our EKS and AWS environment should have not changed since. Unfortunately, I cannot confirm the Quilt3 and Boto3 versions from that day.

[nmajubu]

I'm closing this issue as the issue seems resolved.