FFMPEG segments to fragmented mp4
Er-rchydy opened this issue · 52 comments
I have a long video, i slice it using FFMPEG :
spawn("ffmpeg", [
'-i', './myvideo.mp4' ,
'-c:a', 'libfdk_aac', '-ac', '2', '-ab', '128k', '-c:v', 'libx264',
'-x264opts', 'keyint=24:min-keyint=24:no-scenecut',
'-f', 'segment', '-segment_time', '10', './video/%01d.mp4'
]);
what i'm trying to do is to turn those small videos to fragmented mp4 using mp4frag and send them via socketIO so i can play them using media source extension, can i achieve that using mp4frag ?
I have only used it doing live video with https://github.com/kevinGodell/mse-live-player. If it just a prerecorded video, you could generate all the files and playlist with ffmpeg. If you are trying to simulate the recording as live video, use the "-re" flag on the input.
To expand on what I mentioned earlier, if you are taking an existing video and making segments for streaming, then ffmpeg can do it without needing mp4frag. Target the output to -f hls and you can generate the m3u8 playlist and all of the segments saved to a directory. If you are using a newer version of ffmpeg, then you may have the HLS option to use "fmp4" instead of the default "mpegts" segmenting. Then, you would only need to setup a socket io server to relay those pieces to the browser. If you need help with the ffmpeg configuration, let me know. I have played with it quite a bit and I am sure we can figure something out.
@kevinGodell
Yes please i need help , i'm a beginner so i'm just playing with all those things to understand and practice different things.
what i'm looking for is to generate fragmented MP4 segments from a long video, so i can send them via socket to play them in the browser via media source extension, without hls or m3u8 if that makes sense.
to explain more: i have a long video and i want to slice it into smaller fragmented mp4, and store those small pieces ( 30 seconds each ) in a folder, and later on play them using media source extension.
My problem is that i slice the video, then send the first 30 seconds buffer to the browser, but i can't play it with media source extension, because as you know media source extension can't work with normal mp4, so i need those smaller videos to be fragmented mp4, i tried ffmpeg like this one
spawn("ffmpeg", [
'-i', 'bunny.mp4' ,
'-acodec', 'copy', '-vcodec', 'copy',
'-movflags', 'frag_keyframe+empty_moov+default_base_moof',
'-f', 'segment', '-segment_time', '30', './video/%01d.mp4'
]);
but still can't be played via media source extension, i hope you understand what i'm trying to do, i know there are lot of better ways using hls , m3u8, but i want to do it this way
First thing to know about ffmpeg, is you use codec copy, such as -c:v copy then you cannot alter the video, such as segment time or size or anything. What we need to do is use ffmpeg to generate an HLS playlist formatted as fmp4. When playing segmented mp4, there will always be the init file, and then the video segments. ffmpeg -i bunny.mp4 -f hls -hls_segment_type fmp4 -c:v copy playlist.m3u8 will generate a playlist file that lists the files, an init.mp4, and the video segments. Once you have that list and the files in proper format, you can use whatever method you choose to deliver them to your media source player. Remember, if you want to change the size of the segments, you must re-encode the video using a codec like libx264. Also, if you want to see some available options for the HLS muxer, run ffmpeg -h muxer=hls and it will give you some details.
Unfortunately this ffmpeg -i bunny.mp4 -f hls -hls_segment_type fmp4 -c:v copy playlist.m3u8 didn't produce fragmented mp4, it produce init.mp4 and m3u8 and bunch of m4s, i'm not looking for m4s files , i'm looking for a way to get fragmented mp4, this is the only way that worked for me so far:
ffmpeg -y -i bunny.mp4 -c:a libfdk_aac -ac 2 -ab 128k -c:v libx264 -x264opts 'keyint=24:min-keyint=24:no-scenecut' -ss 0 -t 30 v0.mp4 // get the first 30 seconds of the video
./mp4fragment v0.mp4 fragmented-v0.mp4 // mp4fragment from Bento4
the above code works perfectly for me, because it gives me a real fragmented mp4 video that works with media source extension, the only problem is that i need to execute it lot of times to slice the whole video.
i tried
ffmpeg -y -i bunny.mp4 -c:a libfdk_aac -ac 2 -ab 128k -c:v libx264 -x264opts 'keyint=24:min-keyint=24:no-scenecut' -f segment -segment_time 30 v%01d.mp4
then go through every single one with ./mp4fragment v0.mp4 fragmented-v0.mp4 . but the final fragmented videos didn't work with media source extension. i don't know why it worked using the first code but it didn't work using the second code, i mean what's the difference between -ss 0 -t 30 v0.mp4 and -f segment -segment_time 30 v%01d.mp4 ?
I think we have a breakdown of terminology between us. Hopefully I can be clearer. The ffmpeg cmd I gave you produces fragmented mp4 that are designed to play in media source extension. If you combine any single piece of m4s with the init.mp4, you get a complete mp4 that media source can play. The m4s simply contains the mdat data, which is the actual frames of video. The init.mp4 contains the header info that contains things such as width, height, codec, etc.
When using media source extension, you first feed the init.mp4 to the media source player, followed by all of the m4s fragments.
You are producing segments of complete mp4s instead of fragments of mp4s.
What media source player are you using to play back the video?
This is the shorter version of the code i'm using in client side:
var socket = io();
var video = document.querySelector('video');
var mimeCodec = 'video/mp4; codecs="avc1.64000d,mp4a.40.2"';
if ('MediaSource' in window && MediaSource.isTypeSupported(mimeCodec)) {
var mediaSource = new MediaSource;
video.src = URL.createObjectURL(mediaSource);
mediaSource.addEventListener('sourceopen', sourceOpen);
} else {
console.error('Unsupported MIME type or codec: ', mimeCodec);
}
function sourceOpen (_) {
var mediaSource = this;
var sourceBuffer = mediaSource.addSourceBuffer(mimeCodec);
sourceBuffer.mode = 'sequence';
socket.on('broadcast', function (newPiece) {
// here i'm getting the buffer of the video == buffer
sourceBuffer.addEventListener('updateend', function (_) {
video.play().then(function() { }).catch(function(error) { });
});
sourceBuffer.appendBuffer(buffer); // when the seconde video comes i append it's buffer
})
};
i guess know you will understand why i'm always pointing to fragmented mp4 pieces so i can send them to this code above, because if they are not fragmented mp4, they won't work when i append them, i hope you understand what i'm trying to achieve
I understand. Please try those m4s files that you created. First, send the init.mp4 to the media source player. After that, send each m4s fragment and you will see that it plays. Just disregard the m3u8 file that was created. Or, you could use ffmpeg to output mpeg dash files, which will be identical files output by HLS using the fmp4 option, but it will simply have a different playlist layout.
I tried them , they are working fine , the only problem that i had is when i send init.mp4 followed by playlist0.m4s , playlist1.m4s and so on they play but when i send init.mp4 followed by playlist5.m4s, playlist6.m4s the player can't play them. is there a way to make them work , because it's like live , the viewer that is watching from the beginning is going to get init.mp4, playlist0.m4s , playlist1.m4s .... , but if someone came 10 minutes after the live is began it will get something like init.mp4, playlist9.m4s , playlist11.m4s ...., how can i make the second situation work too.
thanks for your answers , they are really helping me to understand how all this is working
Unfortunately I have no way to test what you are trying and I am surprised that it wont let you play them out of order. There must be some meta data embedded in the segments that are making that are preventing the playback out of order.
Maybe try removing the source buffer mode "sequence" option from you player settings.
Or, use ffmpeg to generate dash files, which is pretty much the same thing. ffmpeg -i bunny.mp4 -f dash -c:v copy dash.playlist.
Another thing to try is to rebuild the files with HLS but include the -hls_playlist_type set to "event" or "vod" and maybe one of them will format the files so that they can be played starting at any fragment.
I tried them all but they didn't work, i didn't understand dash files because they are two categories init-stream0.m4s, chunk-stream0-00001.m4s, chunk-stream0-00002.m4s,... and init-stream1.m4s, chunk-stream1-00001.m4s, chunk-stream1-00002.m4s, ..., i tried to send to the browser the first category but they didn't work, then i tried the second one and they didn't work
Hi i guess i found the problem, like you said it suppose to play segments even if we send init.mp4 followed by playlist5.m4s, Firefox play them without any issue, but in Chrome it gives this error in chrome://media-internals/ :
error 1 : video frame with PTS 0us has negative DTS -80000us after applying timestampOffset, handling any discontinuity, and filtering against append window.
error 2 : Append: stream parsing failed. Data size=76039 append_window_start=0 append_window_end=inf.
If you have any idea on how to make Chrome play them, please help me with it.
Have you tried using -hls_playlist_type with the values event or vod yet? That might make ffmpeg encode it slightly different to play nice in chrome, but I am not sure.
About using ffmpeg mpeg dash output, that looks like maybe your camera has 2 video outputs streams and is including both. If that is the case, I believe you can filter that out or just simply ignore those stream1 files. That init + fragments might be coded to play nice with media source extension, since the format is targeting dash(which run on media source extension).
Yes i tried -hls_playlist_type with event and vod, i didn't play for both.
About dash i double check the dash.playlist and i found that the 2 files are one is video and the other one is audio, do you have any idea how to play them using MSE, i mean send them via Socket and append them ?
I might be wrong, but I thought when the audio and video was split, it was m4a for the audio and m4v for video. Can you post the console output after doing ffprobe url-to-your-camera so we can see what streams are available.
There is no camera, sometimes i test with a local video sometimes i test with IPTV, and also when i execute the dash command you gave above, ffmpeg -i bunny.mp4 -f dash -c:v copy dash.playlist it get just dash.playlist and m4s files , there are no m4a or m4v files at all.
This is the output of ffprobe :
Duration: N/A, start: 43572.338667, bitrate: N/A Program 1 Metadata: service_name : Service01 service_provider: FFmpeg Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(progressive), 704x480 [SAR 40:33 DAR 16:9], 25 fps, 25 tbr, 90k tbn, 50 tbc Stream #0:1[0x101]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 145 kb/s
Sorry, I forgot about that. I usually deal with live video. I make a quick gist based on one of my examples using a local mp4 file as the source with some notes on where you should access the data such as the init and segments. I have to run out right now so I hope you can figure out how to use the file system in nodejs and save the data to file, but that should be easy for you. https://gist.github.com/kevinGodell/21e8bb12cc460fe29c8e8071ba5bc17a
Let me know if you succeed, or not. Good luck.
I don't know why it won't work :
mp4frag.on('initialized', (data)=> { data.initialization.pipe(fs.createWriteStream('./init.mp4')); }); mp4frag.on('segment', (data)=> { data.pipe(fs.createWriteStream(seg-${counter}.m4s)); counter++; });
i get TypeError: data.initialization.pipe is not a function,
i also tried
mp4frag.on('initialized', (data)=> { fs.createWriteStream('./init.mp4').pipe(data.initialization); }); mp4frag.on('segment', (data)=> { fs.createWriteStream(seg-${counter}.m4s).pipe(data); counter++; });
i get Error: Cannot pipe, not readable, i don't know why it wont work
Maybe you should create the writestream first, then try piping. It may be that you are trying to pipe too soon before it is ready. Here is a similar implementation I did https://github.com/kevinGodell/mse-live-player/blob/master/record2.js#L32
it's not working, i guess i will go back and try to fix video frame with PTS 0us has negative DTS -80000us after applying timestampOffset, handling any discontinuity, and filtering against append window error that shows in chrome.
i don't know why is it too complicated to slice a video and send it through socket and play it using Media source extensions. it's like impossible to do it. i was searching for last 20 days every day and asking questions on stackoverflow and in here , but it never worked.
You are not able to use the file system to save any files at all? Are you running the script in a directory that you don't have write access?
I'm running the script on my mac, it's not about permissions , it's not working at all. it gives me this error TypeError: data.initialization.pipe is not a function, i mean this is not a permission error
I think you may be using writestream incorrectly with the piping. I made a new complete example for you here. I am able to create the init and m4s fragments consistently. Just adjust the fps and gop and other settings to what you need. Clone this repo and the run node tests/test6 https://github.com/kevinGodell/mp4frag/blob/master/tests/test6.js
thanks it's working now, how can i make the duration of the segments 30 seconds ?
I created and uploaded a 5 minute long mp4 for testing called 5min.mp4. I use that as the source to generate the fragments. I am not sure the output -fps that you are using, so you will have to do the math on that. I increased my -gop to 900 because I was outputting 30 fps. 30 fps * 30 seconds = 900. After running my file, it made 10 segments, which is what i was expecting if breaking a 5 minute video into 10 pieces of duration 30 seconds. The code can be found on test6.js.
it did make segments 30 seconds but it didn't play them, i get this error error Initialization segment misses expected aac track in chrome://media-internals/
If you are including audio, then you will have to pass a flag to ffmpeg to tell it to encode to aac with some instructions found @ https://trac.ffmpeg.org/wiki/Encode/AAC. If you just want to test to see if video will play with no audio, then pass the -an flag to remove audio completely(just for testing purposes).
When i add '-c:a', 'aac', '-b:a', '128k', i get this error :
/Users/mac/Desktop/test/mp4frag.js:316 throw new Error(MOOF hunt failed after ${this._moofHunts} attempts.); ^ Error: MOOF hunt failed after 40 attempts., i guess it's an issue with mp4frag.js, i tried both aac and libfdk_aac i got the same error. when i add -an flag i got error i posted earlier in chrome error Initialization segment misses expected aac track, i told you this may look simple at first but it's really complicated
The only way to get that error is to feed improperly structured mp4 data into mp4frag. Please post your full ffmpeg command used and I will test on bigbuckbunny to see if I get your error.
I use the same code in your example test6.js
`const params = [
/* log info to console */
'-loglevel', 'quiet',
'-stats',
/* use hardware acceleration if available */
'-hwaccel', 'auto',
/* use an artificial video input */
//'-re',
//'-f', 'lavfi',
//'-i', 'testsrc=size=1280x720:rate=20',
'-i', 'bunny.mp4',
/* set output flags */
//'-an',
'-c:v', 'libx264',
'-c:a', 'libfdk_aac', '-b:a', '128k',
'-movflags', '+frag_keyframe+empty_moov+default_base_moof',
'-f', 'mp4',
'-vf', `fps=${fps},scale=${scale}:-1,format=yuv420p`,
//'-frames', frameLimit,
'-g', gop,
'-profile:v', 'main',
'-level', '3.1',
'-crf', '25',
'-metadata', 'title=test mp4',
'pipe:1'
];`
I see the problem. The moof size is bigger than the chunk size because you are testing on mac, which has some weird pipe size limit of 8192, whereas linux is usually 65536 and windows is around 93000. I was never able to figure out how to increase the pipe or fd(file descriptor) size no matter what I tried on mac. The big buck bunny video I tested with has some erratic sizes of data and some exceed the sizes that I get from live ip cameras. I could probably adapt my code to check for this. Is this a one time use or will you need to use this often?
I changed gop to const gop = 200; and fps to const fps = 15;, now i see the files. but unfortunately we full in the same error that i told you about earlier video frame with PTS 0us has negative DTS -133333us after applying timestampOffset, handling any discontinuity, and filtering against append window
when i send init.mp4 followed by seg-2.m4s.
Thanks for your time i really appreciate your help, i will get back to this when i have some time.
I have been working on a solution all day yesterday and I have figured how to generate the segments of 30 seconds duration. Also, I added support to mp4frag that can handle the larger sizes than what it was designed to deal with. I will be pushing an update soon for mp4frag and test6.js which use big buck bunny as a source. The almost ~10 minute long video gets successfully split into 20 pieces, which should be about 30 seconds each. I will let you know when I have my updates online and they pass deployment tests in travis and appveyor.
I finished testing and it looks good for handling the larger file input. I also changed test7.js to do the fragmenting. It may be what you need. https://github.com/kevinGodell/mp4frag/blob/master/tests/test7.js
it's working now , thank you so much
Hi Kevin, i wanna ask you a favor, i hope if you can help me, i'm looking for a way to listen for segments that has been made by FFMPEG and broadcast them immediately, without any delay , something you did in your mp4frag but i want it without any video manipulation so it can make segments as fast as possible, something like this:
const ffmpeg spawn("ffmpeg", [
'-y', '-i', 'bunny.mp4' ,
'-hls_init_time', '30',
'-hls_time' ,'30' , '-f', 'hls', '-hls_segment_type', 'fmp4',
'-c:v', 'copy', './chunks/playlist.m3u8' ]);
ffmpeg.on('initialized', (data)=> {
io.emit('initMp4', {uri: './chunks/init.mp4'});
});
ffmpeg.on('segment', (data)=> {
// broadcast the segment immediately as soon as it has been made
io.emit('broadcast', {uri: './chunks/currentSegment.m4s'});
});
The problem with having video segments 30 seconds duration is that you will always have to wait 30 seconds before you can access the first piece spit out by ffmpeg. That is why i keep my pieces as short as possible. Take a look at my media source player project https://github.com/kevinGodell/mse-live-player/blob/master/public/player.js and maybe it will give you some ideas.
You didn't understand , 30 seconds video segments is a short for me i will make it 1 minute , the problem is not with the duration, the problem is how to know if new segment has been made, what i'm doing right now is to watch a directory for any changes using this library chokidar, but that's not efficient, because it get triggered before ffmpeg finish the segment, so i'm looking if i can do something like what you did in mp4frag something like this:
const ffmpeg spawn("ffmpeg", [
'-y', '-i', 'bunny.mp4' ,
'-hls_init_time', '30',
'-hls_time' ,'30' , '-f', 'hls', '-hls_segment_type', 'fmp4',
'-c:v', 'copy', './chunks/playlist.m3u8'
]);
ffmpeg.on('initialized', (data)=> {
io.emit('initMp4', {uri: './chunks/init.mp4'});
});
ffmpeg.on('segment', (data)=> {
// broadcast the segment immediately as soon as it has been made
io.emit('broadcast', {uri: './chunks/currentSegment.m4s'});
});
If you are giving those commands to ffmpeg, then it will write the files directly to the directory where the playlist is targeted. You will not be able to listen to any events from ffmpeg that tell you what pieces it has output, other than just raw data output on ffmpeg.stdout. You may be able to read the stderr output from the spawned ffmpeg process by listening to data event on ffmpeg.stderr.on('data') and set the -loglevel to something other than quiet. The reason i wrote mp4frag was because ffmpeg did not give me the control i needed to get the pieces of mp4 fragments.
Actually, i just had a thought. Just keep reading the playlist file and parse it to get the path of the segments. It is pretty human readable and you should be able to make some regex to extract the important data.
reading the playlist file to get the path of the segments is a good idea, thank you so much
@Er-rchydy I just ran into that chrome bug complaining about the timestampOffset. My situation was a little different than yours, but maybe the solution is worth noting here. I am consuming an HLS feed and doing a codec copy on it and sending it out via media source to play in the browser. Changing sourceBuffer.mode from sequence to segment stopped Chrome from complaining. Still works in firefox and safari, too
@kevinGodell First of all thank you for putting this out and for the code samples. I'm trying to create a multi bitrate m3u8 (similar to https://docs.peer5.com/guides/production-ready-hls-vod/) but using one second fragments instead, would it be possible to use an ARRAY of params (for the different resolutions) with mp4frag? I was also wondering why did you chose frameLimit value of 2001 and fps value of 200 in test6.js?
Thanks again ;)
In reference to test6, I use settings like that to try to break my code. I put it under worse situations than it would normally be used under so that I can try to expose any weakness. I went with 2001 to see that doing the math.ceil matched up with an extra segment create for that 1 extra frame, wheres the other segments should have been filled with 200 frames. 2000 frames / 200 fps = 10 frames of 1 second duration. 2001 - 2200 frames should make 11 segments.
Mp4frag is simply parsing data coming from ffmpeg. It does not do any encoding or resizing and would not be able to make a multiple bitrate playlist. Also, each mp4frag instance can only parse one stream.
You could tell a single ffmpeg to output 3 different fragmented mp4 stream of different sizes on pipe:1 ,pipe:2, and pipe:3 and create an mp4frag for each. Then you would have to listen for the 'segment' even on each because that is when the m3u8 is changed and you would have to manually combine them into a master playlist. It sounds tricky, but it could be done.
Is this for a live input stream or pre-recorded video?
@kevinGodell Thanks for the feedback, it's for pre-recorded videos. Hmm, I could listen to the "exit" command (and check for the the error code) since all streams are piped in a single command this should work? would mp4frag generate separate m3u8 files which I can join?
The master playlist on the other hand should be relatively straight forward (I think) I can dynamically populate the content and write the file, given we know the m3u8 file names
I guess that should have been my first question about it being live or pre-recorded. mp4frag is meant for parsing live streaming video. Using it would not give you accurate timestamps. What you are trying to do can easily be done in ffmpeg without the need for any external library. I could guide you on which ffmpeg params to set for you to generate such a list. I just did it recently and put 4 different types of hls lists together using nothing but ffmpeg to generate them. 4 types of hls: fmp4 vod, fmp4 event, mpegts vod, mpegts event. Take a look https://kevingodell.github.io/streams/index.html src https://github.com/kevinGodell/streams
Oh hey, I really forgot that I did this just 6 days ago. Turns out that I documented the 4 different ways that I created the HLS playlists @ https://github.com/kevinGodell/streams/blob/master/create_source2.js and also in 3, 4, and 5. That has the complete ffmpeg command that you need to do the 1 second segments.
Oh great thank you! I guess I can create 4 renditions in the same ffmpeg command then manually create the master playlist. I saw you wrote a comment that crf doesn't take 0 I wonder if you've experimented with lower values than 25. I'm not sure what gop does, can I assume it's always equal to fpsare you did? level can be omitted since you're setting a profile value`?
Last question (sorry if it sounds naive), you have void and event playlist types, does event mean I I will be continuously loop through such playlist (sort of a 24/7 "live" playlist)?
If you set crf to 0(lossless), then you cannot set profile because profile does not support lossless encoding. Level and profile work together, I think.
The crf value will affect the quality and size of files. You will have to experiment to see what is acceptable quality and filesize for your implementation. Lower number = higher quality. 0 - 51
There are 3 types of playlist types, vod (video on demand), event, and live (which uses 0 as value). For what you are doing, you should use vod.
gop is group of pictures. Set that to be the same as the fps if you want to have segments that are 1 second duration. If you want 2 second long segments, then gop should be double the fps, I think.
I am not sure about the looping thing. I think that would be up to the player to automatically start over after reaching the end.
Got it, ok then I'll experiment with those, thank you very much for your time and help kevin!! very much appreciated!
@kevinGodell It all worked well now I'm successfully creating, serving and playing a multi bitrate fmp4 master m3u8 playlist!
Given I have all of that in place and I know how to server it on demand, what would be a good starting point to serving it live (in the same multi bitrate format) if I may ask?
I have personally never tried using a multi-bitrate playlist, but that feature is supposed to be supported by the hls.js library. Check out the source of https://kevingodell.github.io/streams/hls_mpegts_vod/ to see how I used hls.js.
I am not 100% sure what you mean by serving it live, but any http server technology should work as long as you make all of the fragments available in a public folder.