Is there a way to list all possible encoders and options for a particular system?
IchBinGROOT opened this issue · 15 comments
I really need to get going to transcode my library, it has blown up incredibly over the last few months. I have read many of the issues regarding how to select the options (#22, #79, #87, #127, #137, some more) but I am not satisfied with the output yet.
One problem could be that I have only the options to transcode on my laptop (Lenovo L580) or my NAS (QNAP TS-251+, never got it to work on it). I think it would be very helpful if there was a way to just list all possible knobs for a particular system.
I have found this post about how to objectively measure the quality of two video files. So my next goal (as I cannot afford new hardware to transcode on, unfortunately) would be to create some kind of automated process that takes a raw rip of a given film and transcodes it using different parameters from an n-dimensional tensor in order to find out which are the best for both my systems. Before I spend to much time on this, do you know if anyone has attempted to do such a thing before?
To construct such a tensor from which I can then select all the different arguments (e.g. by slicing it into n-dimensional vectors) I would need to know every possible (maybe not every possible, but every realistically feasible) combination of parameters. Is there a way to output such a list? Or is there a way that I just tell you some of the hardware specs on my systems and you could propose how I should start?
Thank you for this amazing project
...but I am not satisfied with the output yet
Unfortunately, I think that comment encapsulates the whole ugly truth. There isn't a way to objectively measure the quality of two video files, as the quality is inherently subjective. If it was objective, then there would only be one (or at least a small) set of options that everyone used.
Since it's subjective, I think the best option is to find something that produces output that you find acceptable, and then watch for outliers. When you have your short list of outliers, or titles that don't look good to you with your default encoding method, then start working through those to find another method that works better for you.
I'm sure others will be able to chime in with their own approaches towards this problem, and I fully expect them to be different. For me, x264 works perfectly fine, using Don's defaults. I know others prefer x265, or the output of the Nvidia hardware accelerated encoder. And then there's AV1, which seems to work well for some titles. All of those choices end up being based on individual constraints (more/less CPU, GPU, storage) coupled with personal preference.
Anyhoo ... that's my 2c. 😎
@IchBinGROOT Thanks for using other-transcode!
And my answer would be basically the same as what @ttyS0 wrote. There's no way to compare quality objectively. Netflix has tried to invent a system but it really doesn't work. At least I don't think so. And I've been doing this a long time.
Chasing perfect quality with transcoding is not worth the effort. A transcoding will always contain quality loss, no matter which encoder and options you use. What you half to decide is how much quality loss you can tolerate for your particular playback system.
Also, the easiest way to improve quality with any transcoding is to simply increase the bitrate and use 10-bit output. 🤷♀️
Ok, so you would recommend just playing around with different settings until I find something I am happy with, right? Then the question remains which settings to try, or as you describe it in most of your post which knobs to turn. I know that my internal graphics chip (GPU-Z says it is this one) does support the use of the --hevc flag but it slows things down noticeably. Where would you start to test different settings?
@IchBinGROOT are you using --qsv when you use --hevc, other-transcode --qsv --hevc?
If you aren't including the QSV option then you aren't using the GPU, which would definitely introduce slowness.
The wiki includes a lot of what you are asking but it can be tricky to find on GitHub, have you discovered it?
@IchBinGROOT The other-transcode tool should detect your GPU and automatically select the QSV encoder. However, depending on your configuration, detection can sometimes fail. So, @klogg416 is correct, adding the --qsv option may be necessary.
Your Kaby Lake Refresh GPU should have support for 10-bit HEVC encoding. However, you might need to do some additional configuration tweaks to your system for FFmpeg to recognize it. This is, unfortunately, a bit of a black art with HEVC for QSV. I had a similar problem years ago when I setup my system. I'm sorry that I don't recall exactly what I did to solve the problem but I'll ask the HiveMind™ if they know because many of them had to do the same thing.
As for encoding options, just start with the defaults. The QSV encoder doesn't have that many options anyway.
I think the GPU is recognised, at least it maxes out when I look in the Task Manager. And if I understand the wiki correctly (yes, I have found it, but thanks @klogg416 )
Using the --hevc option with the Nvidia NVENC, Intel Quick Sync Video or x265 encoders will automatically select 10-bit output.
then I should already have 10 Bit encoding if I pass the --hevc flag, shouldn't I? At the moment I transcode with
other-transcode.bat --hevc --add-audio all /path/to/movie.mkv
sometimes with --target 3000 if I think the quality is not enough. What else would you adjust?
@IchBinGROOT Yes, you should get 10-bit output just using the --hevc option if the QSV encoder is selected.
The default output bitrate for 1080p input is 4000 Kbps so are you lowering the target bitrate?
Or is your input at DVD resolution because that default is 1000 Kbps and raising it to 3000 is excessive and a complete waste of bitrate.
And you don't really need any other options.
@IchBinGROOT a few thoughts on QSV here. It looks like from some of the terminology you're using that you're probably on a Windows machine - is that correct?
First, can you get hardware-accelerated encoding working properly with H.264 output? Running other-transcode --qsv (with no --hevc option) on one of your inputs should produce output with limited impact on CPU usage. Test that first so that you know whether you have a broad-based QSV issue, or just an HEVC QSV issue. Based on your hardware, you should get north of 150 fps on a typical QSV encode to H.264, and while QSV's HEVC encoder will definitely be slower, I'm guessing you should still get somewhere in the 80-100 fps range. And unless your CPU is very underpowered, you shouldn't be seeing extensive CPU usage on these encodes.
If H.264 output with QSV is still not working, you may need to make use of the --qsv-device option, which is documented in the --help full output. It's possible ffmpeg isn't properly picking up on your intel GPU, especially if you are using a system with multiple GPUs. You can check whether the GPU's encoding capabilities are being properly utilized in task manager (again, assuming Windows).
Finally - if you are on a system with multiple GPUs, you may want to try plugging a monitor into the onboard HDMI or DisplayPort - or a dummy plug, which can be had for a few bucks a pop on Amazon or eBay and simulate a monitor's power draw so that the iGPU doesn't go to sleep and mess with functionality. This doesn't always solve things, but it never hurts if the other things aren't working. (Again, I'm not sure if you're actually on a setup with multiple GPUs here - so you can disregard this part if you are using the iGPU as your primary display output).
And then, as far as quality is concerned, it is indeed a dangerous game to go down to a --target 3000 on a Blu-ray-quality input. That may look fine with some titles, but the defaults are the defaults for a reason.
I've definitely driven myself round the bend troubleshooting QSV issues, so happy to keep helping if you need additional assistance.
Oh - and I might do a quick other-transcode --qsv --dry-run [input] - with and without --hevc - to see what other-transcode is passing to the ffmpeg command line. If you post that here we can tell you whether it's properly passing the hardware acceleration arguments or if it's somehow not picking things up.
@donmelton Yes, something about half of my input files are DVD rips, the other half TV recordings of shows (which is perfectly legal where I live as long as I don't distribute them). I tried --target 2000 a few times, but that didn't give that much of a quality improvement, whereas I was able to see differences (between 2000 and 3000) when I watched scenes on my WQHD monitor without that much bigger files.
@wintervaler first of all, yes, I am on a Windows 10 Pro machine.
Hardware:
CPU: i5-8250U
GPU: [Intel UHD Graphics 620](Intel UHD Graphics 620)
32GB RAM
I tried the following different options:
1
other-transcode.bat --qsv '..\Audials Movies\Das Netz.mkv'
which translates to this:
ffmpeg -loglevel error -stats -i "..\Audials Movies\Das Netz.mkv" -map 0:0 -c:v h264_qsv -b:v 6000k -look_ahead:v 1 -color_primaries:v bt709 -color_trc:v bt709 -colorspace:v bt709 -metadata:s:v title= -disposition:v default -map 0:1 -c:a:0 copy -metadata:s:a:0 title= -disposition:a:0 default -sn -metadata:g title= -default_mode passthrough "Das Netz.mkv"
Which goes to speeds of about 190 fps while running at 85% CPU and 100% GPU
2
other-transcode.bat --qsv --10-bit '..\Audials Movies\Das Netz.mkv'
which translates to this:
ffmpeg -loglevel error -stats -i "..\Audials Movies\Das Netz.mkv" -map 0:0 -c:v h264_qsv -pix_fmt:v p010le -b:v 6000k -look_ahead:v 1 -color_primaries:v bt709 -color_trc:v bt709 -colorspace:v bt709 -metadata:s:v title= -disposition:v default -map 0:1 -c:a:0 copy -metadata:s:a:0 title= -disposition:a:0 default -sn -metadata:g title= -default_mode passthrough "Das Netz.mkv"
and doesn't work
3
other-transcode.bat --qsv --hevc '..\Audials Movies\Das Netz.mkv'
which translates to this:
ffmpeg -loglevel error -stats -i "..\Audials Movies\Das Netz.mkv" -map 0:0 -c:v hevc_qsv -pix_fmt:v p010le -b:v 4000k -maxrate:v 12000k -load_plugin:v hevc_hw -color_primaries:v bt709 -color_trc:v bt709 -colorspace:v bt709 -metadata:s:v title= -disposition:v default -map 0:1 -c:a:0 copy -metadata:s:a:0 title= -disposition:a:0 default -sn -metadata:g title= -default_mode passthrough "Das Netz.mkv"
with speeds of 50 fps at 40% CPU and 100% GPU, compressing from 16GB to about 3GB
4
other-transcode.bat --qsv --hevc --target 3000 '..\Audials Movies\Das Netz.mkv'
which translates to this:
ffmpeg -loglevel error -stats -i "..\Audials Movies\Das Netz.mkv" -map 0:0 -c:v hevc_qsv -pix_fmt:v p010le -b:v 3000k -maxrate:v 12000k -load_plugin:v hevc_hw -color_primaries:v bt709 -color_trc:v bt709 -colorspace:v bt709 -metadata:s:v title= -disposition:v default -map 0:1 -c:a:0 copy -metadata:s:a:0 title= -disposition:a:0 default -sn -metadata:g title= -default_mode passthrough "Das Netz.mkv"
and runs with speeds of 60 fps at 70% CPU and 100% GPU, compressing from 16GB to 2.2GB
5
other-transcode --hevc '..\Hüter des Lichts.mkv'
which translates to this:
ffmpeg -loglevel error -stats -i "..\Hüter des Lichts.mkv" -map 0:0 -c:v hevc_qsv -pix_fmt:v p010le -b:v 1000k -maxrate:v 6000k -load_plugin:v hevc_hw -color_primaries:v bt470bg -color_trc:v gamma28 -colorspace:v bt470bg -metadata:s:v title= -disposition:v default -map 0:1 -c:a:0 copy -metadata:s:a:0 title= -disposition:a:0 default -sn -metadata:g title= -default_mode passthrough "Hüter des Lichts.mkv"
at 150 fps, 30% CPU and 90% GPU from 5.1GB (two languages audio) to 0.8GB (one language), bitrate of 1250
6
other-transcode --hevc --add-audio all '..\Hüter des Lichts.mkv'
ffmpeg -loglevel error -stats -i "..\Hüter des Lichts.mkv" -map 0:0 -c:v hevc_qsv -pix_fmt:v p010le -b:v 1000k -maxrate:v 6000k -load_plugin:v hevc_hw -color_primaries:v bt470bg -color_trc:v gamma28 -colorspace:v bt470bg -metadata:s:v title= -disposition:v default -map 0:1 -c:a:0 copy -metadata:s:a:0 title= -disposition:a:0 default -map 0:2 -c:a:1 aac -ac:a:1 2 -metadata:s:a:1 title= -disposition:a:1 0 -map 0:3 -c:a:2 copy -metadata:s:a:2 title= -disposition:a:2 0 -sn -metadata:g title= -default_mode passthrough "Hüter des Lichts.mkv"
at 170 fps, 50% CPU and 100% GPU from 5.1GB to 1.01GB with a bitrate of around 1600
7
other-transcode --hevc --target 3000 --add-audio all '..\Hüter des Lichts.mkv'
ffmpeg -loglevel error -stats -i "..\Hüter des Lichts.mkv" -map 0:0 -c:v hevc_qsv -pix_fmt:v p010le -b:v 3000k -maxrate:v 6000k -load_plugin:v hevc_hw -color_primaries:v bt470bg -color_trc:v gamma28 -colorspace:v bt470bg -metadata:s:v title= -disposition:v default -map 0:1 -c:a:0 copy -metadata:s:a:0 title= -disposition:a:0 default -map 0:2 -c:a:1 aac -ac:a:1 2 -metadata:s:a:1 title= -disposition:a:1 0 -map 0:3 -c:a:2 copy -metadata:s:a:2 title= -disposition:a:2 0 -sn -metadata:g title= -default_mode passthrough "Hüter des Lichts.mkv"
at 170 fps, 50% CPU and 100% GPU from 5.1GB to 2GB at a bitrate of 3100
The first 4 are transcodings of a TV recording, the last of a DVD rip. Do these numbers help in any way?
@IchBinGROOT Well, if you like the way they look at that bitrate then proceed. Just keep in mind that you're probably not getting very much compression.
That performance all looks relatively normal, yep! Seems like things are working. The reason your second example doesn't work is I believe because it's trying to do 10-bit H.264 output, which I think is unsupported (perhaps one of the other HiveMind members can correct me if I'm misremembering).
If you're not using 10-bit output, you may get some additional speed by passing --qsv-decoder to other-transcode. But that won't work with HEVC 10-bit output because of the pixel format conversion to p010le (10-bit). There is a way to do this in QSV hardware now with ffmpeg but I think there were some stability issues last time we looked at that so I don't believe other-transcode has incorporated it yet.
@IchBinGROOT Just looking at your examples now. The second one doesn't work because QSV does not currently support 10-bit H.264 encoding. All the others more or less make sense based on your options.
I think the big slowdown you're seeing is because that's when you're increasing your target bitrate. Increasing bitrate means that the encoder does more processing and, more importantly, your system does more processing I/O to your disc.