is it possible to circumvent AWS Lambda limit of 512MB /tmp for larger S3 media files?

Question

is it possible to circumvent AWS Lambda limit of 512MB /tmp for larger S3 media files?

Closed this issue 10 years ago · 6 comments

Hi, we are considering using yas3fs from within Lambda to retrieve larger than 512 MB mp4 files from S3 but there is currently a limit of 512MB total file system space for all storage needs of the Lambda function. Is there any way to set parameters of the buffering and cache mechanism whereby ffmpeg can process the mp4 on the fly while reading and yas3fs can serve the larger-than-512MB file using no more than, let's say 400MB local storage space?

Answer 1 · 2015-02-06T19:59:05.000Z

At this point that capability is not built in, and would require some significant changes to make that kind of behavior possible.

Answer 2 · 2015-02-08T04:39:52.000Z

Thanks for the reply!

If we completely disable the disk cache, would we be able to sequentially read an mp4 of 700MB start to end with a working memory of 1GB (while the same 512MB disk storage limitation is still in place)? BTW, we don't have any need to actually store the file locally. Just ffmpeg needs to read the file from start to finish.

Answer 3 · 2015-02-11T20:27:16.000Z

At this point I would suggest you simply try the configuration you just mentioned (no disk cache) to see how it will behave and report back. I have not tried that particular scenario.

Answer 4 · 2015-03-09T08:02:28.000Z

Closing the issue at this point, but I am ready to open it again if more info are required.

Answer 5 · 2018-04-07T05:46:58.000Z

@shafqatevo were you able to get ffmpeg working within lambda for larger files?

Answer 6 · 2018-10-09T11:29:06.000Z

@bitsofinfo how to set no disk cache