AuthorizationHeaderMalformed in aws-s3-sink connector

[onion1984]

Hi,

I keep getting this error is my destination bucket is not at us-east-1. Is there any workaround for this?

org.jclouds.aws.AWSResponseException: request GET https://jicong-tokyo-msk-bucket.s3.amazonaws.com/?prefix=.indexes/lens-s3-sink-connector-test/s3sinktopic/00000/ HTTP/1.1 failed with code 400, error: AWSError{requestId='E04E3JZB1C2RASDT', requestToken='Np/477Ozrzro9XlPT1n4Q/lsnkYifdWjf6jyuyZcE4rI+0srFwCnuaCQwKwYdzSfS2+oKB7MixE=', code='AuthorizationHeaderMalformed', message='The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'ap-northeast-1'', context='{Region=ap-northeast-1, HostId=Np/477Ozrzro9XlPT1n4Q/lsnkYifdWjf6jyuyZcE4rI+0srFwCnuaCQwKwYdzSfS2+oKB7MixE=}'}

[davidsloan]

Please can you post your complete connector configuration, excluding anything sensitive?

[onion1984]

@davidsloan here is the connector config
name=s3sinkconnector
connector.class=io.lenses.streamreactor.connect.aws.s3.sink.S3SinkConnector
key.converter.schemas.enable=false
connect.s3.kcql=INSERT INTO xxxx-bucket:tutorial SELECT * FROM s3sinktopic WITH_FLUSH_COUNT = 1
aws.region=ap-northeast-1
tasks.max=2
topics=s3sinktopic
schema.enable=false
value.converter=org.apache.kafka.connect.storage.StringConverter
errors.log.enable=true
key.converter=[org.apache.kafka.connect.storage.StringConverter

[davidsloan]

Please could you try adding this property to your configuration and see if that makes a difference:

connect.s3.aws.client=AWS

[academyExcelleance]

I am also facing issue. I have configured connect.s3.aws.client=AWS property

I have tested in us-east-2 region

[Worker-07f222b4a4c877c74] org.jclouds.aws.AWSResponseException: request GET https://msk-s3-sink-source-connector.s3.amazonaws.com/?prefix=backup&max-keys=1000 HTTP/1.1 failed with code 400, error: AWSError{requestId='HAHBEZVMZQX3H7JQ', requestToken='HcQZw4XoYHf847sMkkB84LADQ0bYwdgbB147T4J34D/9hMsLv5hyiDn1aIDomVMXLQ1tkuIhrZE=', code='AuthorizationHeaderMalformed', message='The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-east-2'', context='{Region=us-east-2, HostId=HcQZw4XoYHf847sMkkB84LADQ0bYwdgbB147T4J34D/9hMsLv5hyiDn1aIDomVMXLQ1tkuIhrZE=}'}

Followings are configurations:

connector.class=io.lenses.streamreactor.connect.aws.s3.sink.S3SinkConnector
key.converter.schemas.enable=false
connect.s3.kcql=INSERT INTO msk-s3-sink-source-connector:backup SELECT * FROM LogTopic WITH_FLUSH_COUNT=10;INSERT INTO msk-s3-sink-source-connector:backup SELECT * FROM LogTopic1 WITH_FLUSH_COUNT=10;
aws.region=us-east-2
connect.s3.aws.client=AWS
tasks.max=2
topics=LogTopic,LogTopic1
schema.enable=false
value.converter=org.apache.kafka.connect.storage.StringConverter
errors.log.enable=true
key.converter=org.apache.kafka.connect.storage.StringConverte

[davidsloan]

@academyExcelleance

This is unlikely to be an issue with the connector. Have you very recently deleted the bucket in us-east-1 and recreated it in us-east-2?

Please see this answer on SO.

In short, because S3 Buckets have Eventual Consistency there may be some delay between the bucket being deleted and it being available to recreate in another region, which could cause this error message.

[academyExcelleance]

davidsloan
No, I have not deleted any bucket.

My another team member is also getting same issue.

[davidsloan]

@academyExcelleance

My another team member is also getting same issue.

With the same or a different bucket?

If you use a new bucket with a new bucket name, does it work?

[academyExcelleance]

This was new bucket for my team member.

I now tried with same configuration and another bucket in us-east-1. This worked

connector.class=io.lenses.streamreactor.connect.aws.s3.sink.S3SinkConnector
key.converter.schemas.enable=false
connect.s3.kcql=INSERT INTO lenses-s3-connector:backup SELECT * FROM LogTopic WITH_FLUSH_COUNT=10;INSERT INTO lenses-s3-connector:backup SELECT * FROM LogTopic1 WITH_FLUSH_COUNT=10;
aws.region=us-east-1
connect.s3.aws.client=AWS
tasks.max=2
topics=LogTopic,LogTopic1
schema.enable=false
value.converter=org.apache.kafka.connect.storage.StringConverter
errors.log.enable=true
key.converter=org.apache.kafka.connect.storage.StringConverter

Also the back up is working for two topics

[academyExcelleance]

Is it because of different region other than us-east-1.

I got the issue when used us-east-2 and my team member used mumbai (ap-south-1) region

[davidsloan]

I believe this issue was fixed when we migrated the connector from JClouds to the official AWS client.

Please let me know if you see this problem manifest again.