Lambda Timeouts on Ingest
Closed this issue · 2 comments
I've set up the api following the documentation and new packages layout. I wanted to manually start the ingest process so I was following #52 and starting with Landsat. However, when I run the step function I am hitting the lambda timeout limit of 300 seconds. Has anyone else hit this?
The logs posted in #52 seem to indicate that the upload of files should be < 180 seconds.
I'm deploying in us-west-2 so maybe there is latency issues?
START RequestId: c908ae00-aa34-49c5-af3c-954c3c90dc69 Version: $LATEST
2018-07-28T03:52:16.280Z c908ae00-aa34-49c5-af3c-954c3c90dc69 ingest event: {"satellite":"landsat","arn":"arn:aws:states:us-west-2:126405998223:stateMachine:LandsatMetadataProcessorStateMachine-n78KcI1tLErq","maxFiles":8,"maxLambdas":10,"linesPerFile":300}
2018-07-28T03:52:27.028Z c908ae00-aa34-49c5-af3c-954c3c90dc69 uploaded 250 files
2018-07-28T03:52:47.347Z c908ae00-aa34-49c5-af3c-954c3c90dc69 uploaded 500 files
2018-07-28T03:53:08.545Z c908ae00-aa34-49c5-af3c-954c3c90dc69 uploaded 750 files
2018-07-28T03:53:27.783Z c908ae00-aa34-49c5-af3c-954c3c90dc69 uploaded 1000 files
2018-07-28T03:53:56.126Z c908ae00-aa34-49c5-af3c-954c3c90dc69 uploaded 1250 files
2018-07-28T03:54:08.518Z c908ae00-aa34-49c5-af3c-954c3c90dc69 uploaded 1500 files
2018-07-28T03:54:28.771Z c908ae00-aa34-49c5-af3c-954c3c90dc69 uploaded 1750 files
2018-07-28T03:54:50.549Z c908ae00-aa34-49c5-af3c-954c3c90dc69 uploaded 2000 files
2018-07-28T03:55:05.618Z c908ae00-aa34-49c5-af3c-954c3c90dc69 uploaded 2250 files
2018-07-28T03:55:20.881Z c908ae00-aa34-49c5-af3c-954c3c90dc69 uploaded 2500 files
2018-07-28T03:55:38.163Z c908ae00-aa34-49c5-af3c-954c3c90dc69 uploaded 2750 files
2018-07-28T03:56:02.818Z c908ae00-aa34-49c5-af3c-954c3c90dc69 uploaded 3000 files
2018-07-28T03:56:30.963Z c908ae00-aa34-49c5-af3c-954c3c90dc69 uploaded 3250 files
2018-07-28T03:56:44.597Z c908ae00-aa34-49c5-af3c-954c3c90dc69 uploaded 3500 files
2018-07-28T03:56:59.005Z c908ae00-aa34-49c5-af3c-954c3c90dc69 uploaded 3750 files
END RequestId: c908ae00-aa34-49c5-af3c-954c3c90dc69
REPORT RequestId: c908ae00-aa34-49c5-af3c-954c3c90dc69 Duration: 300082.82 ms Billed Duration: 300000 ms Memory Size: 1024 MB Max Memory Used: 297 MB
2018-07-28T03:57:16.361Z c908ae00-aa34-49c5-af3c-954c3c90dc69 Task timed out after 300.08 seconds
Any ideas or help is much appreciated!
Must have been a latency issue.
I rebuilt the api in us-east-1 and it grabbed the files without timing out.
Glad it has worked out, I have run into timeouts before. If you consistently get timeout problems you can also reduce the linesPerFile parameter and it will split the CSV up into smaller files.
We're also working on having the ingestion be deployed as an ECS task using Fargate. This way it can ingest the entire file and not have to deal with splitting it up in order to run in Lambda functions.