[PM-1978] amazon region is not being picked up correctly when creating a s3 bucket
Closed this issue · 8 comments
It seems like pegasus-s3 does not create a bucket in a region other than us-east-1
In my credentials.conf I have
[amazon]
endpoint = http://s3.amazonaws.com/
region = us-east-2
However, when I use pegasus-s3 with the above credentials file the bucket still gets created in us-east-1.
truncated pegasus-s3 debug log below
pegasus-s3 -d mkdir s3://bamboo@amazon/vahi-test
..
2024-09-25 10:42:12,508:DEBUG:botocore.hooks(238): Event before-call.s3.HeadBucket: calling handler <function inject_api_version_header_if_needed at 0x7fa9fd811620>
2024-09-25 10:42:12,508:DEBUG:botocore.endpoint(114): Making request for OperationModel(name=HeadBucket) with params: {'url_path': '/vahi-test', 'query_string': {}, 'method': 'HEAD', 'headers': {'User-Agent': 'Boto3/1.23.10 Python/3.6.8 Linux/4.18.0-553.16.1.el8_10.x86_64 Botocore/1.26.10'}, 'body': b'', 'url': 'http://s3.amazonaws.com/vahi-test', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7fa9fd302358>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'vahi-test'}}}
...
024-09-25 10:42:12,900:DEBUG:botocore.parsers(239): Response headers: {'x-amz-id-2': 'zMRPoQFzhTfuqZPWXZhhus+AxlXZHsVeuJh8SQVmzqLPkskth9Y5OTX4F3zWAkasKGHRCz+JM3Dv1ho4931LsmBk+PjT8jAN', 'x-amz-request-id': 'D5TP5E9MQNPZB9BC', 'Date': 'Wed, 25 Sep 2024 17:42:13 GMT', 'Location': '/vahi-test', 'Server': 'AmazonS3', 'Content-Length': '0'}
2024-09-25 10:42:12,900:DEBUG:botocore.parsers(240): Response body:
b''
2024-09-25 10:42:12,900:DEBUG:botocore.hooks(238): Event needs-retry.s3.CreateBucket: calling handler <botocore.retryhandler.RetryHandler object at 0x7fa9fd214eb8>
2024-09-25 10:42:12,900:DEBUG:botocore.retryhandler(211): No retry needed.
2024-09-25 10:42:12,900:DEBUG:botocore.hooks(238): Event needs-retry.s3.CreateBucket: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7fa9fd214fd0>>
The s3 web ui also confirms buckets gets created in us-east-1
Author: @vahi
this came, while debugging a user setup where they have this in their ~/.aws/config file
cat ~/.aws/config
[default]
region = us-east-2
That itself was triggering this error for me with 5.0.9dev
ROR: Command exited with non-zero exit code (1): /usr/bin/pegasus-s3 mkdir s3://vahi@amazon/titan2d-workdir-bucket-vahi 2024-09-24 19:40:33,144 INFO: -------------------------------------------------------------------------------- 2024-09-24 19:40:33,145 INFO: Starting transfers - attempt 2 2024-09-24 19:40:35,149 INFO: /usr/bin/pegasus-s3 mkdir s3://vahi@amazon/titan2d-workdir-bucket-vahi 2024-09-24 19:40:35,669 INFO: An error occurred (400) when calling the HeadBucket operation: Bad Request An error occurred (400) when calling the HeadBucket operation: Bad Request 2024-09-24 19:40:35,669 ERROR: Command exited with non-zero exit code (1): /usr/bin/pegasus-s3 mkdir s3://vahi@amazon/titan2d-workdir-bucket-vahi
And looking into that led me to this which gave me a clue
Removing the region from ~/.aws/config makes the error
Bad Request An error occurred (400) when calling the HeadBucket operation:
go away
Author: @vahi
Specifying the region in the endpoint also does not work
$ more ~/.pegasus/credentials.conf
[amazon]
#endpoint = http://s3.amazonaws.com/
endpoint = http://s3.us-east-2.amazonaws.com
we get an error
aise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (400) when calling the HeadBucket operation: Bad Request
this is because the request header constructed still refers to us-east-1
headers={'User-Agent': b'Boto3/1.23.10 Python/3.6.8 Linux/4.18.0-553.16.1.el8_10.x86_64 Botocore/1.26.10', 'X-Amz-Date': b'20240925T205445Z', 'X-Amz-Content-SHA256': b'e3b0c44298f
c1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=xxx/20240925/us-east-1/s3/aws4_request, SignedHeaders=host;
x-amz-content-sha256;x-amz-date, Signature=bf895e5b20069ec11307934d1b232e84cc434df46e4ab3fc8bc31030d1029293', 'amz-sdk-invocation-id': b'15cc718c-547d-4902-8c0c-8bce7faeee2d', 'am
z-sdk-request': b'attempt=1'}>
note credential is us-east-1.
Author: @vahi
To get this working, in pegasus-s3 we need to do the following
1.
boto3.client(
"s3",
endpoint_url=endpoint,
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key,
region_name="us-east-2"
)
2. when creating a bucket specify a location constraint
s3.create_bucket(Bucket=dest.bucket, CreateBucketConfiguration={"LocationConstraint": "us-east-2"})
- in the credentials.conf have the region specific endpoint
[amazon]
endpoint = http://s3.us-east-2.amazonaws.comAuthor: @vahi
Also, if region is specified in ~/.aws/config ,
[default]
region = us-east-2 then also we see this error
2024-09-30 12:12:39,865 INFO: /usr/bin/pegasus-s3 mkdir s3://user@amazon/vahi-test
2024-09-30 12:12:40,574 INFO: An error occurred (400) when calling the HeadBucket operation: Bad Request
An error occurred (400) when calling the HeadBucket operation: Bad Request
2024-09-30 12:12:40,575 ERROR: Command exited with non-zero exit code (1): /usr/bin/pegasus-s3 mkdir s3://user@amazon/vahi-test
commenting it out fixes it
Author: @vahi
@rynge my changes work fine with s3. and also updated the credentials.conf documentation here [https://pegasus.isi.edu/docs/5.0.9dev/reference-guide/data-management.html#pegasus-credentials-conf]
can you please check if other protocols that we rely on boto3 for work and close the ticket
Author: @vahi
we still need special handling for us-east-1
we should not set any create bucket configuration for us-east-1
aws/aws-sdk-js#3647
OSN tested, and works well!