Cannot make secor work with AWS instance profile
dovka opened this issue · 8 comments
Hi guys,
we are prohibited form using the hardcoded AWS credentials
(that we had been using until now) and trying to make secor work with AWS instance profile or role.
I didn't find any documentation but from the secor.common.properties comment
we infer that we need to leave the credentials blank and set the role name.
`# AWS authentication credentials.
"
Leave empty if using IAM role-based authentication with s3a filesystem.
aws.access.key=
aws.secret.key=
Session token only required if using temporary S3 access keys
aws.session.token=
aws.role=
`
We specified the role name (not sure what to do with aws.session.token) but still get the same error:
2021-02-22 12:55:27,900 [main] (com.pinterest.secor.main.ConsumerMain:87) ERROR Consumer failed java.lang.RuntimeException: Failed to find required configuration option 'aws.access.key'. at com.pinterest.secor.common.SecorConfig.checkProperty(SecorConfig.java:725) at com.pinterest.secor.common.SecorConfig.getString(SecorConfig.java:731) at com.pinterest.secor.common.SecorConfig.getAwsAccessKey(SecorConfig.java:417) at com.pinterest.secor.util.FileUtil.configure(FileUtil.java:75) at com.pinterest.secor.main.ConsumerMain.main(ConsumerMain.java:70)
how to make secor use instance profile or role?
thank you for your help!
David
Henry,
thank you for your advice -
switching to s3a and HadoopS3UploadManager did the trick -
secor started implicitly using instance profile :)
Henry,
I will gladly do update the README file.
How do I do it - via pull request or just attach here?
We put the IAM roles authentication config in production yesterday and since then we see duoble the CPU usage by secor.
We need twice the amount of nodes (had 3 now need 6 or we are falling behind with data being queued up)
The only thing we changed is authentication.
Is it possible the the code checking on re-authentication (AWS token expiration) all the time and causes this CPU drain?
Thank you for your help Henry
Henry,
we run into an issue with secor instance profile in AWS China region.
The S3 code that works in regular AWS breaks in AWS China with this error:
com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: SMY7YNMPSYCKSKNJ
I asked for the clarification from AWS Support, and here is their answer:
"This request is sent to Amazon Global region, not China region;
So let us check the region setting first.
When using resources in China region, we must clearly specify the region information.
I saw you use Amazon Java SDK V1, you could use below code to set the region:
AmazonS3 s3 = AmazonS3ClientBuilder.standard().withRegion(Regions.CN-NORTH-1).build();
"
Would you please check the code,
how easy would it be to patch it up?
Thank you for your help!
David