RuntimeWarning: invalid value encountered in subtract

[nav9]

Checklist

• I am using ffmpeg-quality-metrics 3.1.7
• I am using the latest stable version of ffmpeg or a recent build from Git master ffmpeg version N-109879-g42bf52b4c5 Copyright (c) 2000-2023 the FFmpeg developers built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1) configuration: --enable-gpl --enable-libx264 --enable-libx265 --enable-nonfree --enable-libvmaf --enable-version3.

Expected behavior
No runtime warnings should be shown.

Actual behavior
A warning was shown:

/home/navin/.pyenv/versions/3.10.1/lib/python3.10/site-packages/numpy/core/_methods.py:233: RuntimeWarning: invalid value encountered in subtract
  x = asanyarray(arr - arrmean)

For this line:
print("answer :",ffqm.get_global_stats()["ssim"]["ssim_y"]["average"])

Command

from ffmpeg_quality_metrics import FfmpegQualityMetrics
ffqm = FfmpegQualityMetrics("sample.mkv", "sample.mp4")
metrics = ffqm.calculate(["ssim", "psnr"])
print(metrics.keys())
print(f"SSIM of first frame: {metrics['ssim'][0]}")
print(f"PSNR of first frame: {metrics['psnr'][0]}")
print("sum: ", sum([frame["ssim_y"] for frame in metrics["ssim"]]) / len(metrics["ssim"]))
print("answer :",ffqm.get_global_stats()["ssim"]["ssim_y"]["average"])

The videos used can be any from this page.

Environment (please complete the following information):

• Ubuntu 20.04
• Python 3.10.1
• ffmpeg version N-109879-g42bf52b4c5

[slhck]

I cannot reproduce this under macOS.

$ cat test.py
from ffmpeg_quality_metrics import FfmpegQualityMetrics

ffqm = FfmpegQualityMetrics(
    ref="/Users/werner/Downloads/sample_1920x1080.mkv",
    dist="/Users/werner/Downloads/sample_1280x720.mkv",
)
metrics = ffqm.calculate(["ssim", "psnr"])
print(metrics.keys())
print(f"SSIM of first frame: {metrics['ssim'][0]}")
print(f"PSNR of first frame: {metrics['psnr'][0]}")
print(
    "sum: ", sum([frame["ssim_y"] for frame in metrics["ssim"]]) / len(metrics["ssim"])
)
print("answer :", ffqm.get_global_stats()["ssim"]["ssim_y"]["average"])

$ python test.py
dict_keys(['psnr', 'ssim'])
SSIM of first frame: {'n': 1, 'ssim_y': 0.891, 'ssim_u': 0.99, 'ssim_v': 0.985, 'ssim_avg': 0.923}
PSNR of first frame: {'n': 1, 'mse_avg': 47.04, 'mse_y': 69.91, 'mse_u': 1.08, 'mse_v': 1.53, 'psnr_avg': 31.41, 'psnr_y': 29.69, 'psnr_u': 47.79, 'psnr_v': 46.29}
sum:  0.8533988183161019
answer : 0.853

➜ python --version
Python 3.11.0
➜ pip3 list | grep numpy
numpy                  1.24.1

Can you reproduce the error with the specific sample_1920x1080.mkv and sample_1280x720.mkv files?

[slhck]

Note that "sum" is not the "sum" but an average — just a very convoluted way of doing it. You don't need it and can use the global stats as-is.

[nav9]

"sum" is not the issue here. It's just about that warning message. I could reproduce it. This time I used sample_640x360.mkv from here, and used the same file as the reference and distorted video, just to be sure that nothing else influences it FfmpegQualityMetrics("sample_640x360.mkv", "sample_640x360.mkv").
This is on Ubuntu 20.04 (perhaps you could use a virtualbox).
Perhaps it is related to PyEnv? I'm using Python 3.10.1 via PyEnv.
Since it's just a warning, if it is of no consequence, please ignore it.

[slhck]

used the same file as the reference and distorted video

That explains it! You should not compare a reference against itself. This will lead to infinite values. I will add a check to the input arguments that warns the user when doing so.

[nav9]

You are right. I had initially converted an .mkv video to .mp4 losslessly, and then used the mkv video as a reference and the mp4 video as distorted. That's why it showed the warning. Very interesting. So rather than check the input parameters, perhaps the code should check if there's a quality difference and/or catch the runtime warning.

[slhck]

That's an edge case I'm not particularly sure is worth fixing. The exact response of a metric for the same input depends on the metric itself. I'd rather pass on the values as calculated.