Parser Crash when Fragments list > 1800

[chowmyster]

Using lambda function to convert cloud footage into .mkv files.
Having issues with converting more than 1800 fragments.

Error captured in CloudWatch attached
AWS Java Parser Crash Log.txt

[MixMasterMitch]

The root exception is Caused by: java.io.IOException: No space left on device This would suggest to me that all of the available drive space available to the lambda function was consumed. Per Lambda limits you can use up to 512 MB in the temp directory.

[chowmyster]

Is there any way to capture the memory usage on the lambda function such that I can log it to CloudWatch?

[chowmyster]

I have observed another crash occur.
This happened after trying to create another clip after the first crash observed above occurs.

Error captured from CloudWatch attached:
AWS Java Parser Crash Log 2.txt

[chowmyster]

Hey Mitchell,

I have done some more testing and found some more crashes occuring.
What I am seeing is that subsequent "Parser" runs is causing a crash.
In the log at:

2020-07-03T12:37:39.562-07:00
Total Fragments: 648

Is the first time Lambda function runs and it completes with no isses.

At time stamp:

2020-07-03T12:38:53.833-07:00
Total Fragments: 575

Is the second time the Lambda function runs with an even smaller fragment count. However, this one crashes with the writing to output failed.

Are the Lambda functions not tearing down all the way / clearing the Temp ?
If this is the case, how do I clear out the Temp directory such that subsequent requests that hit the lambda function dont crash?

Full Log is here:
AWS Java Parser Crash Log 3.txt

[chowmyster]

Added code to delete everything from temp before excution:
System.out.println(Runtime.getRuntime().exec("rm -rf /tmp/*"));

Did not help.

[MixMasterMitch]

Here is a blog that describes the way that lambda will reuse containers: https://aws.amazon.com/blogs/compute/container-reuse-in-lambda/

Is there any way to capture the memory usage on the lambda function such that I can log it to CloudWatch?

I am not aware of any out-of-the-box solution for monitoring the /tmp usage.

System.out.println(Runtime.getRuntime().exec("rm -rf /tmp/*")); Did not help.

Hmm. Perhaps also validate that the /tmp directory really is empty? Perhaps it takes a bit for space to be truly reclaimed? I am not sure. Some other things you could consider are:

1. Keep the data in memory instead of writing it to /tmp. I realize this may not be a feasible option depending on how the MKV files are used (i.e. with ffmpeg).
2. Use EFS for storage in the lambda. Here is a blog post about that: https://aws.amazon.com/blogs/aws/new-a-shared-file-system-for-your-lambda-functions/

[MushMal]

Resolving. Please reopen if you need further help