ArneVogel/concat

ffmpeg: too many open files

Closed this issue · 5 comments

mabrowning commented

ffmepg can only process 100-200 files at a time. We should perform a divide and conquer algorithm for runs with many chunks.

Also, if ffmpeg fails, the concat proceeds to... delete all those precious downloaded chunks! Oops.

Thanks for the tool, though!

ArneVogel commented

Hey,

I didnt knew about the ffmpeg file limit. I will look into a fix for that over christmas break.

Cheers

frozenpandaman commented

I'm getting this as well. I'm not sure if it's an ffmpeg issue, rather more of a system-wide thing. Running ulimit -n for me returns 256. I'm on Mac, so should be the same for Linux, but I'm not sure of the corresponding Windows command. Anyway, so my guess is that either users will have to manually increase this, or Concat will have to make ffmpeg open/close files more aggressively (e.g. combine into chunks of 256—or however many—VOD parts, then combine those into the final file).

mabrowning commented
mabrowning commented

Actually, it turns out that you can simply concatenate the .ts files together byte-for-byte, and then transmux them into mp4 using ffmpeg. This is what the ffmpeg -i concat:a|b|c argument is doing under the hood. Its a bit tricky to get ffmpeg to output a mpeg TS, and concatenating other mux formats ( e.g. MP4) requires creating a "concat script" and a different combination of arguments.

Thus, the simple solution to the original issue is concatenating all those loose .ts into a temporary before calling ffmpeg on it.... no need for "grouping"

  • 👍2
ArneVogel commented

I have changed that the chunks are now saved and combined as .ts files. With this change I tested it on windows and ubuntu and it has combined over 300 chunks with no problem for me. I have used a low quality setting (see release v0.2) though. Let me know if you will have problems with ffmpeg with v0.2.