frankenstone toolbox

This is the frankenstone toolbox to evaluate and unify models and features for user-generated video quality.

requirements

ubuntu >=22.04
nvencc should run (binary is included)
ffmpeg install via conda (see conda base setup) or globally
python 3.11 in a conda environment with cuda (see conda base setup)
python dependencies: python3 -m pip install -r requirements.txt
Important to run the model, you need a GPU, e.g. Nvidia 3090 with at least 12 GB of GPU memory (for 4K videos), tested with Nvidia 3090 Ti 24 GB GPU Ram.

conda base setup

conda create -n ENV python=3.11
conda activate ENV
conda install cudnn cudatoolkit ffmpeg

cd $CONDA_PREFIX
mkdir -p ./etc/conda/activate.d
touch ./etc/conda/activate.d/env_vars.sh

edit ./etc/conda/activate.d/env_vars.sh as follows:

#!/bin/sh
export LD_LIBRARY_PATH=$CONDA_PREFIX/lib

deactivate and reactivate conda env

conda deactivate
conda activate ENV

install tensorflow with pip (see https://www.tensorflow.org/install/pip)

python3 -m pip install tensorflow[and-cuda]

check if gpu/gpus are detected:

python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices())"

this command should show the gpu/gpus, and should in the best case not complain, that "something is missing"

usage

A call of ./frankenstone.py --help should result in: (important run it within your conda environment)

usage: frankenstone.py [-h] [--frame_sampling]
                       [--features_folder FEATURES_FOLDER]
                       video [video ...]

frankenstone toolbox for UGC video quality models/features

positional arguments:
  video                 video to process

options:
  -h, --help            show this help message and exit
  --frame_sampling, -fs
                        use frame sampling (default: False)
  --features_folder FEATURES_FOLDER
                        only for calculate features, folder to store the
                        features (default: features)

stg7 2024

origin of the name

frankenstone is a bad translation of frankenstein, which is a reference to the Frankenstein novel by Mary Shelley. There Frankenstein's monster is somehow "put together" by different pieces, thus the proposed toolbox is similar to this "hacking together" approach.

acknowledgments

If you use this software in your research, please include a link to the repository and reference the following paper. Do not forget to also cite the corresponding models, e.g., NVENC, DOVER, Q-Align, VILA, MUSIQ, and FasterVQA if you use them in your research.