Native support for codecs/formats (TS,H265,mpeg2++)

Question

Native support for codecs/formats (TS,H265,mpeg2++)

mifi opened this issue 6 years ago · 63 comments

Due to losslesscut using chromium's html5 video tag which only supports certain container formats and codecs, we currently need to "convert to supported format" when trying to open exotic formats or codecs. This could be improved with the help of ffmpeg.

Workarounds

Compile custom electron

https://github.com/electron/build-tools

Convert to supported format

Currently it's possible to cut those files by first using the File -> Convert to supported format (slow) function, however this is a re-encoding (although fast) of the whole file, just to be able to view it in the HTML5 video player (note that the original and cut versions will not be re-encoded and are still lossless.)

Improve current proxy/dummy code

Currently a <canvas> element is used, and ffmpeg will stream low-resolution jpeg images to that canvas.

Drawbacks:

Slow, laggy, low resolution, low fps
No audio

This could be improved:

Electron (Chromium) might never support h265

It seems like Chrome and most other browsers are not going to be supporting H265 as it stands now, and since we are using Chromium (Electron), this will be a problem.
It might be possible to enable a flag to enable H265 decoding, but it requires some modding of electron.

Update: Chromium now supports h265 (hardware decoding only)

Answer 1 · 2019-05-05T07:47:20.000Z

heres the solution from the chrnoium developers
https://github.com/henrypp/chromium/blob/master/hevc_support.md
https://github.com/henrypp/chromium/blob/master/hevc_support_new.md

Answer 2 · 2019-05-05T07:48:36.000Z

Now that more and more cameras (including DJI Mavic 2) are starting to support H265, it would be nice to have support for this here as well.
Currently it's possible to cut those files by first using the File -> Convert to friendly format (slow) function, however this is a re-encoding (although fast) of the whole file, just to be able to view it in the HTML5 video player (note that the original and cut versions will not be re-encoded and are still lossless.)

It seems like Chrome and other browsers are not going to be supporting H265 as it stands now, and since we are using Chromium (Electron), this will be a problem.
It might be possible to enable a flag to enable H265 decoding, but it requires some modding of electron.

If anyone has some bright ideas, shout out!

https://stackoverflow.com/questions/36382568/h-265-hevc-web-browser-support

webtorrent/webtorrent-desktop#561

electron/libchromiumcontent#198

https://github.com/strukturag/libde265.js

heres the solution from the chrnoium developers
https://github.com/henrypp/chromium/blob/master/hevc_support.md
https://github.com/henrypp/chromium/blob/master/hevc_support_new.md

Answer 3 · 2019-05-06T15:40:52.000Z

I'm not sure how to do this with electron. (LosslessCut uses Electron which uses Chromium.) Feel free to open an issue at electron to see if this can be integrated. I think it is not a trivial task, so I don't have time to look at this now.

Answer 4 · 2019-05-30T05:45:15.000Z

Lately I swap my gear and see, that recording in HEVC format gives me better quality over H264 even with lower bitrate. I would love to switch to HEVC for all my projects although only Lossless Cut stops me from doing that - https://i.imgur.com/avu5AQE.png (after converting to friendly format).

Is there a chance this will work at some point or should I still sit on old format? Thanks.

Answer 5 · 2019-05-30T07:32:54.000Z

I'm seeing that too with GoPro 7 HEVC videos and fast convert. The slow one works.

Answer 6 · 2019-05-31T00:05:31.000Z

You're right! It works with slow one (That quality xD). Enough for cutting no worries :)
Thanks, that will do for time being.

Answer 7 · 2019-08-12T21:45:31.000Z

Great! the reason for the low quality is to make it as fast as remotely possible, and only to be used as a preview.

Answer 8 · 2019-08-31T13:39:22.000Z

As LossLess-cut is build in electron do you ever considered to use mpv as player component? I've just found that a project to do that it's already there... https://github.com/Kagami/mpv.js/ with also some samples implementation

Answer 9 · 2019-09-01T22:14:25.000Z

Thanks for the tip! I didn't know about mpv.js, but if it's easy to integrate then that's definitely an option.
Kagami/mpv.js#27

Answer 10 · 2019-11-04T12:07:49.000Z

In 1.5.0 I have implemented a hack that lets us cut H265 videos, but scrubbing and playing only works at a very low fps and currently without audio.

Answer 11 · 2019-12-04T22:00:15.000Z

Getting this error when trying to merge 2 H.265 videos together. Is there something I can do about this?

Answer 12 · 2020-03-26T17:14:49.000Z

Is it possible to add support for grabbing a frame at 4k/hevc to jpg even though the preview is not native?

Answer 13 · 2020-03-28T14:51:36.000Z

Is it possible to add support for grabbing a frame at 4k/hevc to jpg even though the preview is not native?

I have implemented this now

Answer 14 · 2020-03-28T16:23:02.000Z

Nice, thanks! Can't wait to test it in the next release!

Answer 15 · 2020-03-29T11:27:44.000Z

Check out newest version now

Answer 16 · 2020-03-29T15:31:05.000Z

Thanks, tested 3.17.9. It's working great:)

Answer 17 · 2020-03-29T15:49:13.000Z

@mifi Is there any news on supporting H265 natively?

Answer 18 · 2020-03-30T10:17:05.000Z

No. I think it needs a lot of work because have to compile a custom electron with a custom chromium that uses a custom ffmpeg. Not something i will prioritize unless this work gets funded somehow, or if we can find the work of someone else who did this already.

Answer 19 · 2020-12-11T21:10:01.000Z

This is still a problem in the current build and I was wondering if progress is being made on it as video standards move forward

Answer 20 · 2020-12-11T21:17:36.000Z

AFAIK chromium/electron still doesn't support H265 and they refuse to support it due to expensive licensing: https://bugs.chromium.org/p/chromium/issues/detail?id=684382
So we still need to build a custom electron with a custom chromium to get this native support

Answer 21 · 2022-02-17T07:18:55.000Z

I came across this guide "it’s not as difficult as it might sound" Electron with custom Chromium build

and then the instructions for HEVC support in chromium

Also noticed this gem from the chromium forums that is currently being implemented Issue 1286132: Implement HEVC for video playback and video conferencing on Windows

Answer 22 · 2022-02-17T16:16:37.000Z

Any updates on native HEVC implementation? The solution of making a dummy proxy is pretty brilliant, but it would be nice to see my videos in full res.

Answer 23 · 2022-02-17T16:24:24.000Z

@gmcmicken nice finds. I've not taken on the task of building a custom chromium. However I do have an idea with the https://developer.mozilla.org/en-US/docs/Web/API/MediaSource to hopefully allow for more smooth streaming from ffmpeg to the player. It may also allow playing audio. Note that it still needs to encode the video while playing but I think h264 encoding can be done very fast these days. Could also set the duration which would remove the need for the html5ified dummy file.

Answer 24 · 2022-07-22T11:31:59.000Z

There seems to be some good news: Sida Zhu has implemented support for hw decoding for HEVC in chromium v104.0.5084.0, disabled by default and enabled with the PlatformHEVCDecoderSupport feature.
(https://github.com/StaZhu/enable-chromium-hevc-hardware-decoding) .
chromium 104 needs electron nighty build 21 (https://github.com/electron/releases) .
Mikael, could you build losslesscut updated to this electron version and with feature enabled? Thanks a lot!

Answer 25 · 2022-07-22T11:58:42.000Z

that's fantastic! Then I think once electron 20 becomes stable we can upgrade to that. Note that upgrading losslesscut to newest electron is quite a big job. But I plan on doing it soon, now that electron>v8 slowness on macos has been fixed #714

Answer 26 · 2022-07-23T10:10:42.000Z

You are right, also latest electron 20 beta has chromium 104, with the 8-weeks release cycle we should hopefully have a stable 20 soon. I understand that going from electon 8 to 20 could be a big jump, I'll donate for your effort. Thanks.

Answer 27 · 2022-09-25T10:14:37.000Z

Hi Mikael,
do you have some progress on this improvement?

Answer 28 · 2022-09-25T18:14:56.000Z

Hi. Unfortunately not, as ive been very busy with other projects. But, assuming electron now supports natively, it will be solved once Llc upgrades electron, which is one of the first next big tasks

Answer 29 · 2022-09-27T20:52:25.000Z

Hi. Unfortunately not, as ive been very busy with other projects. But, assuming electron now supports natively, it will be solved once Llc upgrades electron, which is one of the first next big tasks

I will definitely buy the app store version to subsidise a couple of cups of coffee for your effort ;)

Answer 30 · 2022-11-22T15:03:07.000Z

I'm testing with Electron v21 now and with app.commandLine.appendSwitch('enable-features','PlatformHEVCDecoderSupport') h265 seems to be working! so expect an alpha release of losslesscut soon

Answer 31 · 2022-11-23T15:48:12.000Z

OMG, It works wonderfully! I'm gonna cry now)) Quick, precise, responsive, original quality, no preview conversion... this makes my workflow much comfortable (I'm shooting long scripted 4k videos, and use LosslessCut to markdown them, cut good takes, and it really helps me save much storage space). Thank you, thank you, thank you, @mifi!

Answer 32 · 2022-11-24T08:18:48.000Z

Yep. In v3.48.2 HEVC works like a charm!

Answer 33 · 2022-11-25T17:18:10.000Z

HEVC cutting

Works like a charm on my Fujifilm X-T4 10 bit HEVC!

Im going to cut and delete more than half of my 14 TB storage now and save a ton from storage space costs now and in the future. Will toss a coin to my @mifi

Answer 34 · 2022-11-25T18:07:09.000Z

This is great. Is the app in Microsoft Store going to be updated also? Thanks

Answer 35 · 2022-11-25T19:14:43.000Z

thank you so much for working on this app and integrating this feature :) this will improve my housekeeping tremendously

Answer 36 · 2022-11-26T01:59:32.000Z

@titobf yes, but i want to iron out bugs first before releasing to stores. already a few bugs coming in.

Answer 37 · 2022-11-26T04:07:05.000Z

I'm having an issue. I can't seem to convert h.265 to the player in 3.48.2. It just yells at me that the file is not natively supported, I try to convert, it does, and then it yells again. Here's the media info I pulled from VLC. I could send the file as well if you needed to reproduce the bug on your end.

Stream 0
Codec: MPEG-H Part2/HEVC (H.265) (hevc)
Type: Video
Video resolution: 1920x1080
Buffer dimensions: 1920x1088
Frame rate: 29.970030
Decoded format: Planar 4:2:0 YUV 10-bit LE
Orientation: Top left

Stream 1
Codec: A52 Audio (aka AC3) (a52 )
Language: English
Type: Audio
Channels: Stereo
Sample rate: 48000 Hz
Bits per sample: 32

Stream 2
Codec: Text subtitles with various tags (subt)
Language: English
Description: English CC
Type: Subtitle

EDIT: Nevermind, fixed it. New bug though. The program freaks out how I mentioned above if you're accessing a file off a network share. When I moved the file to the same PC, however, it was able to load up perfectly fine. So it has some issue with network shares (I'm currently using windows' built-in drive sharing if you wanted to replicate it yourself)

Answer 38 · 2022-11-27T22:22:23.000Z

Just banding onto everyone else that is thanking you for HEVC support. My previous workflow was converting 12-13 hevc files to h264 using nvenc which took 25~ minutes before i could start any work.

This is excellent!

Answer 39 · 2022-12-07T01:04:51.000Z

https://chromestatus.com/feature/5186511939567616

Enables support for decoding HEVC video on platforms where hardware (e.g., GPU, media accelerator, etc) for decoding HEVC is available (Android 5.0+, macOS 11+, with supported hardware on Windows 8+ and ChromeOS).

I wonder how it will work on macOS 10.15 for example? Regarding this commit

@mifi

Answer 40 · 2022-12-07T07:43:03.000Z

Issue: can't see HEVC video in the main window -- just see black

Environment: Mac, Mojave (10.14.6), LC 3.48.2. Footage shot on GoPro 11 in HEVC format.

Interestingly, if I choose "Show thumbnails" the thumbnails do render. Audio plays correctly, and I can perform cuts. I just don't see the video in the main window. H.264 video does display.

UPDATE 22.12.10 - Upgrading to Mac OS 11 (Monterey) solved the problem. Thanks nosferatu500.

Answer 41 · 2022-12-07T08:12:18.000Z

@JerriAtric

HEVC is available (Android 5.0+, macOS 11+, with supported hardware on Windows 8+ and ChromeOS)

Answer 42 · 2022-12-07T08:15:15.000Z

I think the simplest solution here is to split the array for supported and unsupported systems with respect to HEVC.

Answer 43 · 2022-12-07T08:26:26.000Z

Oh I assumed it works everwhere.
@JerriAtric do you see any error message in the developer tools?
I wonder how we can detect if hardware hevc support is enabled from inside losslesscut. Maybe we just need to try it and check if the video tag gives an error? And if it gives ane rror then fallback to generate preview

edit: maybe we need to use this api:
https://github.com/StaZhu/enable-chromium-hevc-hardware-decoding#mediacapabilities

Answer 44 · 2022-12-07T08:38:01.000Z

Source

let obj = document.createElement('video');
console.log(obj.canPlayType('video/mp4;codecs="hev1.1.6.L120.90"'));

Answer 45 · 2022-12-18T07:42:53.000Z

Same issue as JerriAtric above, but on Win11 22621.963. Exact same behavior:

Issue: can't see HEVC video in the main window -- just see black
Interestingly, if I choose "Show thumbnails" the thumbnails do render. Audio plays correctly, and I can perform cuts. I just don't see the video in the main window. H.264 video does display.

Answer 46 · 2022-12-18T08:24:59.000Z

@dpallen are you running newest llc beta? you can work around this for now by either downgrading, or file -> convert to supported format

Answer 47 · 2022-12-18T18:04:39.000Z

Thanks for the quick reply, but I resolved my issue after some more testing. My footage used 4:4:4 chroma subsampling. Using 4:2:0 HEVC footage works perfectly in llc 3.48.2 beta.

Answer 48 · 2022-12-23T10:31:46.000Z

So to summarize, in order to detect if the player/device supports a file, (for example HEVC) we need to:

const mediaConfig = {
  type: 'file',
  video: {
    contentType : 'video/mp4;codecs="hev1.1.6.L120.90"',
    width: 1920,
    height: 1080,
    bitrate: 10000, 
    framerate: 30,
  }
}

const { supported } = await navigator.mediaCapabilities.decodingInfo(mediaConfig);
// result.supported indicates whether or not the video with given profile, width, and height can played well on the browser

The problem is to find the contentType parameter. It is described here: https://developer.mozilla.org/en-US/docs/Web/Media/Formats/codecs_parameter but it varies by format/codec. Need to find a library where someone implemented this logic into a library for all formats/codecs. I would think that the output from ffprobe could be transformed into this codec parameter.

Potential implementations for each format/codec:

AV1 spec implementation
MP4