Fargate task sometimes fails with invalid security token
Closed this issue · 2 comments
When starting a Fargate task using the ingest Lambda with a payload such as:
{
"fargate": {
"url": "https://sentinel-stac.s3.amazonaws.com/catalog.json"
}
}it can sometimes fail with an error:
UnrecognizedClientException: The security token included in the request is invalid.
Once this happens it will continue to happen within the same Lambda container. A workaround is to force a reload of the Lambda code so that it will start a new container next time.
This might to be a bug with AWS, where credentials get messed up somehow within a container.
We would have to investigate further to get more info to determine if it is an actual bug or due to the way we are starting the task or how we define our Lambda role.
Since there is a workaround (just add an envvar and save, that will reload the function) there is no immediate action here other than to monitor.
Another observation....this seems to happen after a Lambda function has been used to ingest an SNS STAC record, and then the same container is used to try to invoke a Fargate task.
Wondering if the Lambda gets a session token that is valid for reading SNS messages, but then subsequently when used for Fargate it fails to negotiate a new session token.
Fargate tasks have been removed from deployment of sat-api. This type of batch ingestion should be done outside of the sat-api stack.