vs-average

A VapourSynth plugin for averaging clips together.
Early release, not all features are tested. Please report any issues either here, or in EoE's dms (See bottom)

Description

vs-average is a VapourSynth plugin for averaging multiple clips together. The general idea is to be able to take multiple sources of the same video, and average them together, to negate effects of lossy compression. The plugin can also be effective for temporal blurring.

Supported Formats

Sample Type & Bits Per Sample:

Mean: 8, 10, 12, 16 and 32 bit integer, 16 and 32 bit float.
Median: All integer formats (8..32), only 32 bit float. (f16 will be added in a later commit)

Color Family: Gray, RGB, YUV or YCoCg.

Sub-Sampling: any.

Note that the input format must be the same for all inputted clips.

Usage

Mean

Mean will set the output pixel to the average (or mean) of the input pixels from each clip.

average.Mean(clip[] clips[, int output_depth=clips[0].format.bits_per_sample], int preset, float[] multipliers)

clips:
List of clips to be processed. Must be the of the same format, length, fps, ect.
output_depth:
Bitdepth of the output. Will default to the same as clips[0].
Only 8, 16 or 32 bit is supported.
Since all calculations are done interally as f64s, it's far more efficient to input your sources as 8 bit, and return as 16 bit with the increased precision. In the case that you want to directly output the clip returned by Mean, I'd suggest you return a 16 bit clip, and dither down using resize.Point or similar, even for 16 -> 8 bit, due to an internal rounding error. Significant improvements can be observed over returning a higher bitdepth clip, and dithering down, than a lower bitdepth clip. For this same reason, returning a 10 or 12 bit clip is not supported (And also because I'm lazy). For more information, see the comments in mean.rs.
preset:
Integer based preset value for per frame type weightings. See below for how this works. Currently only one preset is implemented. Any other inputs than the ones stated below (or none) will be interpreted as multipliers=[0, 0, 0] (no weighting).
1. Reverse (default) x264/5 based ip/pb qp offset ratios. (--ipratio 1.4 --pbratio 1.3). Works for other encoders/ratios as well (though may be less effective)
  Equivalent to multipliers=[1.82, 1.3, 1.0]
multipliers:
I, P and B frame multipliers, for per frame weighting. Useful for when you want average.Mean to favour I frames. Can be used to (simply) reverse higher b and p frame quantization. As an example, if you only wanted I and P frames to be selected for averaging, you could use multipliers=[1, 1, 0]. Differences with larger amounts of clips when using preset=1 are negligable, but do exist, so it might be worth leaving on. It may also be worth increasing the I and P frame multipliers by a small amount from preset=1, especially if fast motion interpolation algorithms were used on sources, thereby further decreasing the quality of P and B frames.

Median

Median will set the output pixel to the Median (middle value of the sorted data) of the input pixels from each clip.

average.Median(clip[] clips])

clips:
List of clips to be processed. Must be the of the same format, length, fps, ect.

Note that since Median does not define an output_depth parameter, any input of an even number of clips (where the average of the middle two valeus is taken to be the mean) will likely induce another (though smaller) rounding error. I'll fix this at some point too.

Examples

Take the Mean of 3, 8 bit input clips, and output as 16 bit.

clips = [clip_a, clip_b, clip_c]

mean = core.average.Mean(clips, output_depth=16)

Take the Median of 3 clips.

clips = [clip_a, clip_b, clip_c]

mean = core.average.Median(clips)

Simple temporal blur

# add an extra frame to the start of our clip so it's one frame behind
slow_clip = clip[0] + clip
# drop the first frame of our clip so it's one frame ahead
fast_clip = clip[1:]

# average our slow, original, and fast clips together to get a temporal blur.
temporal_blur = core.average.Mean([slow_clip, clip, fast_clip], output_depth=16)

Compilation

cargo build --release

FAQ

Q: Why did you implement support for 16 bit floats?
A: Mainly because if the end user wanted to work in floats, and had 8 bit sources, f16s are far smaller in memory usage, and are implemented on all not-under-a-rock CPUs since the cavemen were around circa 2009. Further processing can be done in 32 bit. For more information, see mean.rs or contact me on Discord (See below).
Q: How fast is vs-average?
A: Pretty fast, vs-average implements multithreading in the default build, allowing for a huge throughput. In fact, the main bottlenecks I've noticed is drive latency (Observed up to ~100MiB/s sustained random reads), and decode speed (decoding 11 AVC bitstreams isn't easy). For fastest speeds with multiple clips, I'd suggest using something akin to dgdecodenv for decoding acceleration via CUDA, and all source files on one (or multiple) SSDs.

Note that all my tests have been performed using lsmas.LWLibavSource to index and decode souce files, on an R9 3900x with 32GiB of ram.
Q: Do you plan to write more useless plugins?
A: Yup, rust is pretty cool, vapoursynth-rs is brilliant, and I'm full of dumb ideas :^)

Contributors

EoE
- Discord: End of Eternity#6292
Nephren
- Discord: Rin-go#8647

kageru/vs-average