Get current frame/snapshot

[ramtinak]

Hi, I was wondering to know how could we get the current frame (by timespan)?
This peace of code will return an IInuptStream of a frame that we can use it.

        public static async Task<IInputStream> GetThumbnailAsync(StorageFile file,TimeSpan duration)
        {
            var mediaClip = await MediaClip.CreateFromFileAsync(file);
            var mediaComposition = new MediaComposition();
            mediaComposition.Clips.Add(mediaClip);
            return await mediaComposition.GetThumbnailAsync(
                duration, 0, 0, VideoFramePrecision.NearestFrame);
        }

but it's not working for HEVC/x265 codec.
Is there be a code or anything to get current frame in FfmpegInterop? or will be available in future?

[Ps: I'm not a c++ programmer]

[lukasf]

It is currently not possible to directly get frames from FFmpegInterop, and MediaClip class cannot be created from a MediaStreamSource (which is what FFmpegInterop creates). So right now you are out of luck, unless someone adds this directly into FFmpegInterop.

[brabebhin]

It is not that difficult to obtain what you want, just not using the MediaComposition class. But it involves a bit of refactoring on the OnStarting and OnSampleRequest events.

You can use the code in the Starting event to seek to where you want in the stream and the code in GetNextSample to obtain a video frame buffer. You can proceed to extracting a thumbnail from said buffer. It wouldn't really involve much of C++ programming, just winRT, which is similar to C#, aside from the different syntax, and everything will be synchronous here anyway. So no need for async/await madness.

My guess is that MediaComposition probably does something similar behind the scenes. MediaStreamSources are just much lower level.

[lukasf]

[Ps: I'm not a c++ programmer]

It does not sound like he wants to build this into FFmpegInterop himself ;) Sure it would be possible, but you need some basic knowledge of C++/CX, and also dig a bit into ffmpeg workings (convert pixel format to rgb).

[brabebhin]

The only basic knowledge needed is to expose 2 methods to the ABI and feed the resulting buffer into his favorite C# imaging library.

Surely someone can do a method to expose a sample(I can see several usecases for this) at a specific timestamp, but until someone does that he can just hack one with minimal documentation he can get from MSDN.

[lukasf]

@ramtinak I have added a branch "lukasf/frame-grabber". It contains code to grab frames from a video file, and also some helper functions to encode them as jpg/bmp/png file. Please note that it does not allow you to capture the "current live frame" during playback of a video with FFmpegInterop. It will instead open the file (again) with FFmpegInterop, grab the frame, and close the file afterwards. An option exists to toggle "exact seek" on off for the grabber. If it is off, this means that the grabber will grab the closest previous key frame, which can be a lot faster than decoding from key frame up to the precise position. This can be helpful when getting thumbnails where speed is more important than a specific position.

Take a look at the C++ sample. I also tried to add the same code to the C# sample. But due to a strange problem with my new PC, I currently cannot compile the C# sample. So I don't know if the code in the C# sample works, it might contain typos, but you can also give it a try.

[ramtinak]

@lukasf
Hi, first of all thank you very much.
second of all wow! great jub, fast seek is very fast!!!!!!!!

---

I tested both C# and C++ sample.
result:
when exactseek is true, when you call ExtractVideoFrameAsync method nothing happened and file save picker never prompt up.
but when exactSeek is false, everything works well.

---

You missed currentFile = file; in C# sample project.

and in void ExtractFrame(object sender... in C# sample you should put these peace of code:

                    if (file != null)
                    {
                        var outputStream = await file.OpenAsync(FileAccessMode.ReadWrite);
                        await frame.EncodeAsJpegAsync(outputStream);
                        outputStream.Dispose();
                        await Windows.System.Launcher.LaunchFileAsync(file);
                    }

---

Update:
some media could not open with this ffmpeginterop version
Error:
MF_MEDIA_ERR_DECODE: HRESULT - 0xC00D36B1

Note: this problem will disappears when you relaunch the app.

---

Thanks a lot.

[lukasf]

@ramtinak Thanks, glad to hear that it works for you! Just a few days ago, I pushed a new commit to the branch, which already fixes the problems you mentioned. If you pull the latest changes, things should work.

About opening media problem, I cannot reproduce this right now. But it might be because this branch was based on an older version of the yuv-output branch. The frame grabber code does not really change anything for normal decoding. I just merged the latest changes from yuv-output. Hope this solves your issues.