Authentication problem with Lambda function
EloyTolosa opened this issue · 7 comments
Hi,
I'm having troubles accessing Athena from a Lambda function. I want to use an AccessKeyID and SecretAccessKey from an IAM User I created, which has the following permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "s3:PutObject",
"Resource": [
"arn:aws:s3:::webbeds-input-zone/datalake-api/*",
"arn:aws:s3:::datalake-api/*"
]
},
{
"Sid": "a1",
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::aws-athena-query-results-webbeds-datareader"
]
},
{
"Sid": "a2",
"Effect": "Allow",
"Action": [
"iam:CreateAccessKey",
"iam:GetUser",
"iam:ListAccessKeys"
],
"Resource": [
"arn:aws:iam::*:user/${aws:username}"
]
},
{
"Sid": "a3",
"Effect": "Allow",
"Action": [
"athena:StartQueryExecution",
"athena:StopQueryExecution",
"athena:GetQueryResultsStream",
"athena:GetQueryResults",
"athena:GetNamedQuery",
"athena:DeleteNamedQuery",
"athena:ListQueryExecutions",
"athena:ListNamedQueries",
"athena:GetWorkGroup",
"athena:CreateNamedQuery",
"athena:GetQueryExecution",
"athena:BatchGetNamedQuery",
"athena:BatchGetQueryExecution"
],
"Resource": [
"arn:aws:athena:eu-west-1:137693930675:workgroup/webbeds_datareader",
"arn:aws:athena:eu-west-1:137693930675:workgroup/primary"
]
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"s3:GetBucketPublicAccessBlock",
"s3:GetBucketPolicyStatus",
"s3:Get*",
"athena:GetNamespace",
"athena:ListWorkGroups",
"s3:List*",
"s3:GetBucketAcl",
"athena:GetCatalogs",
"athena:GetNamespaces",
"s3:GetAccountPublicAccessBlock",
"s3:ListAllMyBuckets",
"athena:GetExecutionEngine",
"athena:GetExecutionEngines",
"athena:GetTables",
"athena:GetTable",
"s3:GetBucketLocation",
"athena:ListWorkGroups",
"athena:ListDataCatalogs"
],
"Resource": "*"
},
{
"Sid": "VisualEditor2",
"Effect": "Allow",
"Action": [
"s3:Get*",
"s3:List*",
"s3:*"
],
"Resource": [
"arn:aws:s3:::aws-athena-query-results-webbeds-datareader/*",
"arn:aws:s3:::aws-athena-query-results-*"
]
},
{
"Sid": "VisualEditor3",
"Effect": "Allow",
"Action": [
"glue:GetDatabase",
"glue:CreateDatabase"
],
"Resource": [
"arn:aws:glue:eu-west-1:137693930675:catalog",
"arn:aws:glue:eu-west-1:137693930675:database/default"
]
},
{
"Sid": "VisualEditor4",
"Effect": "Allow",
"Action": "glue:GetTables",
"Resource": "arn:aws:glue:eu-west-1:137693930675:database/*"
},
{
"Sid": "VisualEditor5",
"Effect": "Allow",
"Action": [
"glue:GetDatabase",
"glue:GetPartition",
"glue:GetTables",
"glue:GetPartitions",
"glue:BatchGetPartition",
"glue:GetDatabases",
"glue:GetTable"
],
"Resource": [
"arn:aws:glue:eu-west-1:137693930675:database/default",
"arn:aws:glue:eu-west-1:137693930675:database/webjet_data_lake",
"arn:aws:glue:eu-west-1:137693930675:database/webbeds_callcentre",
"arn:aws:glue:eu-west-1:137693930675:database/webbeds_callcentre/*",
"arn:aws:glue:eu-west-1:137693930675:table/webjet_data_lake/*",
"arn:aws:glue:eu-west-1:137693930675:database/webbeds_searches",
"arn:aws:glue:eu-west-1:137693930675:table/webbeds_searches/*",
"arn:aws:glue:eu-west-1:137693930675:table/default",
"arn:aws:glue:eu-west-1:137693930675:table/default/*",
"arn:aws:glue:eu-west-1:137693930675:table/webbeds_callcentre/*",
"arn:aws:glue:eu-west-1:137693930675:catalog",
"arn:aws:glue:eu-west-1:137693930675:table/default/bookings_stream_last",
"arn:aws:glue:eu-west-1:137693930675:database/webbeds_masterdb",
"arn:aws:glue:eu-west-1:137693930675:table/webbeds_masterdb/*",
"arn:aws:glue:eu-west-1:137693930675:database/webbeds_datacc",
"arn:aws:glue:eu-west-1:137693930675:table/webbeds_datacc/*"
]
},
{
"Sid": "CreateEditDropViews",
"Effect": "Allow",
"Action": [
"glue:CreateTable",
"glue:UpdateTable"
],
"Resource": [
"arn:aws:glue:eu-west-1:137693930675:catalog",
"arn:aws:glue:eu-west-1:137693930675:database/webjet_data_lake",
"arn:aws:glue:eu-west-1:137693930675:table/webjet_data_lake/*"
]
},
{
"Sid": "VisualEditor9",
"Effect": "Allow",
"Action": [
"s3:Get*",
"s3:List*"
],
"Resource": [
"arn:aws:s3:::webbeds-input-zone*",
"arn:aws:s3:::webbeds-reporting-zone*",
"arn:aws:s3:::webbeds-master-zone*",
"arn:aws:s3:::webbeds-*-datasource*",
"arn:aws:s3:::webbeds-jactravel-*",
"arn:aws:s3:::webbeds-sunhotels-*",
"arn:aws:s3:::webbeds-five9-logs"
]
},
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "glue:GetTable",
"Resource": [
"arn:aws:glue:eu-west-1:137693930675:catalog",
"arn:aws:glue:eu-west-1:137693930675:table/webbeds_analysis/pricescrawler_*",
"arn:aws:glue:eu-west-1:137693930675:database/webbeds_analysis"
]
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": "glue:BatchCreatePartition",
"Resource": [
"arn:aws:glue:eu-west-1:137693930675:catalog",
"arn:aws:glue:eu-west-1:137693930675:database/webbeds_analysis",
"arn:aws:glue:eu-west-1:137693930675:table/webbeds_analysis/pricescrawler_*"
]
},
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::webbeds-sapcheck-logs/uploads/reports/*"
}
]
}
As you can see, there's plenty of permissions to execute queries to athena.
Also, I'm unsetting the environment variable AWS_SDK_LOAD_CONFIG by using
os.Unsetenv("AWS_SDK_LOAD_CONFIG")
as the README says.
But, whenever I make a call to athena, the next error message pops up:
I don't know what the problem could be, and this is driving me nuts.
I talked to amazon and they told me that maybe this is something related to this driver.
Thanks a lot in advance.
Hello @EloyTolosa Can you try the code in https://github.com/uber/athenadriver/tree/master/examples and see if they work? If they work, can do debug and compare the difference between your code and the example code?
Hi.
I have been looking around the code. I saw the link you provided, and I also looked around https://github.com/uber/athenadriver/blob/master/examples/auth.go which contains several authentication methods.
In the auth.go file, I cannot see any method that uses what I need.
I need the driver to use the AccessKeyID and SecretAccessKey I provide to the driver, which has all permissions to call executions to Athena.
What I can only see is that the AccessKeyID and the SecretAccessKey are dummy and do not have a purpose at all.
Am I wrong?
Is the only solution to give the Lambda function all the permissions instead of trying to pass it the credentials?
Thanks
Can you use it like this? @EloyTolosa
https://github.com/uber/athenadriver/blob/master/examples/auth.go#L31
Replace the dummy ones with your real ones.
Every time I try that method, it returns this message:
UnrecognizedClientException: The security token included in the request is invalid
I tried with fresh-new AccessKey,SecretAccessKey pair and AccessKey,SecretAccessKey pair given from Assume Role API Call using AWS SDK in Go.
What am I doing wrong? Why does it say that the token included in the request is invalid?
Hey there. I found the issue.
Whenever you try to assume a role in another account, you need to set the AwsSessionToken for your request to work.
I did that making a call to the sts.AssumeRole function and passing the session token to the config struct.
Thanks a lot for your time.
I have a suggestion. Maybe write this exact example as there are more people that I'm sure they want to assume a role in another account and have this exact problem. You could put the example in https://github.com/uber/athenadriver/blob/master/examples/auth.go, or at least explaining it in case someone needs it.
Great progress @EloyTolosa Can you please share the example code by creating a PR( you can add description in README.md or add the code in auth.go )?
Sure, I'll do it!