A performance benchmark for blind video quality assessment (BVQA) models on user-generated databases, for the UGC-VQA problem studied in the paper UGC-VQA: Benchmarking blind video quality assessment for user generated content.
- >= python 3.6.7
- sklearn
Compute feature matrix on a given dataset and put it in data/ folder, with MOS array stored in the same order (We have provided the MOS arrays of three UGC datasets). The code evaluates the extracted features through 100 iterations of train-test splits and returns the median (std) SRCC/KRCC/PLCC/RMSE performances. Note that it is not applicable for deep learning based models.
$ python src/evaluate_bvqa_features.py
$ python src/evaluate_bvqa_features.py [-h] [--model_name MODEL_NAME]
[--dataset_name DATASET_NAME]
[--feature_file FEATURE_FILE]
[--mos_file MOS_FILE] [--out_file OUT_FILE]
[--color_only] [--log_short] [--use_parallel]
[--num_iterations NUM_ITERATIONS]
[--max_thread_count MAX_THREAD_COUNT]
| BVQA Dataset | Download | Paper |
|---|---|---|
| KoNViD-1k (2017) | KoNViD-1k | Hosu et al. QoMEX'17 |
| LIVE-VQC (2018) | LIVE-VQC | Sinno et al. TIP'19 |
| YouTube-UGC (2019) | YouTube-UGC | Wang et al. MMSP'19 |
| Methods | Download | Paper |
|---|---|---|
| BRISQUE | BRISQUE | Mittal et al. TIP'12 |
| NIQE | NIQE | Mittal et al. TIP'13 |
| ILNIQE | ILNIQE | Zhang et al. TIP'15 |
| GM-LOG | GM-LOG | Xue et al. TIP'14 |
| HIGRADE | HIGRADE | Kundu et al. TIP'17 |
| FRIQUEE | FRIQUEE | Ghadiyaram et al. JoV'17 |
| CORNIA | BIQC_Toolbox | Ye et al. CVPR'12 |
| HOSA | BIQA_Toolbox | Xu et al. TIP'16 |
| VIIDEO | VIIDEO | Mittal et al. TIP'16 |
| V-BLIINDS | V-BLIINDS | Saad et al. TIP'14 |
| TLVQM | nr-vqa-consumervideo | Korhenen et al. TIP'19 |
| VIDEVAL | VIDEVAL_release | Tu et al. CoRR'20 |
The median SRCC (std SRCC) of 100 repititions of different methods on different datasets.
| Methods | KoNViD-1k | LIVE-VQC | YouTube-UGC |
|---|---|---|---|
| BRISQUE | 0.6567 (0.0351) | 0.5925 (0.0681) | 0.3820 (0.0519) |
| NIQE | 0.5417 (0.0347) | 0.5957 (0.0571) | 0.2379 (0.0487) |
| IL-NIQE | 0.5264 (0.0294) | 0.5037 (0.0712) | 0.2918 (0.0502) |
| GM-LOG | 0.6578 (0.0324) | 0.5881 (0.0683) | 0.3678 (0.0589) |
| HIGRADE | 0.7206 (0.0302) | 0.6103 (0.0680) | 0.7376 (0.0338) |
| FRIQUEE | 0.7472 (0.0263) | 0.6579 (0.0536) | 0.7652 (0.0301) |
| CORNIA | 0.7169 (0.0245) | 0.6719 (0.0473) | 0.5972 (0.0413) |
| HOSA | 0.7654 (0.0224) | 0.6873 (0.0462) | 0.6025 (0.0344) |
| VGG-19 | 0.7741 (0.0288) | 0.6568 (0.0536) | 0.7025 (0.0281) |
| ResNet-50 | 0.8018 (0.0255) | 0.6636 (0.0511) | 0.7183 (0.0281) |
| VIIDEO | 0.2988 (0.0561) | 0.0332 (0.0856) | 0.0580 (0.0536) |
| V-BLIINDS | 0.7101 (0.0314) | 0.6939 (0.0502) | 0.5590 (0.0496) |
| TLVQM | 0.7729 (0.0242) | 0.7988 (0.0365) | 0.6693 (0.0306) |
| VIDEVAL | 0.7832 (0.0216) | 0.7522 (0.0390) | 0.7787 (0.0254) |
The median PLCC (std PLCC) of 100 repititions of different methods on different UGC-VQA datasets.
| Methods | KoNViD-1k | LIVE-VQC | YouTube-UGC |
|---|---|---|---|
| BRISQUE | 0.6576 (0.0342) | 0.6380 (0.0632) | 0.3952 (0.0486) |
| NIQE | 0.5530 (0.0337) | 0.6286 (0.0512) | 0.2776 (0.0431) |
| IL-NIQE | 0.5400 (0.0337) | 0.5437 (0.0707) | 0.3302 (0.0579) |
| GM-LOG | 0.6636 (0.0315) | 0.6212 (0.0636) | 0.3920 (0.0549) |
| HIGRADE | 0.7269 (0.0287) | 0.6332 (0.0652) | 0.7216 (0.0334) |
| FRIQUEE | 0.7482 (0.0257) | 0.7000 (0.0587) | 0.7571 (0.0324) |
| CORNIA | 0.7135 (0.0236) | 0.7183 (0.0420) | 0.6057 (0.0399) |
| HOSA | 0.7664 (0.0207) | 0.7414 (0.0410) | 0.6047 (0.0347) |
| VGG-19 | 0.7845 (0.0246) | 0.7160 (0.0481) | 0.6997 (0.0281) |
| ResNet-50 | 0.8104 (0.0229) | 0.7205 (0.0434) | 0.7097 (0.0276) |
| VIIDEO | 0.3002 (0.0539) | 0.2146 (0.0903) | 0.1534 (0.0498) |
| V-BLIINDS | 0.7037 (0.0301) | 0.7178 (0.0500) | 0.5551 (0.0465) |
| TLVQM | 0.7688 (0.0238) | 0.8025 (0.0360) | 0.6590 (0.0302) |
| VIDEVAL | 0.7803 (0.0223) | 0.7514 (0.0420) | 0.7733 (0.0257) |
If you use this code for your research, please cite our papers.
@article{tu2020ugc,
title={UGC-VQA: Benchmarking Blind Video Quality Assessment for User Generated Content},
author={Tu, Zhengzhong and Wang, Yilin and Birkbeck, Neil and Adsumilli, Balu and Bovik, Alan C},
journal={arXiv preprint arXiv:2005.14354},
year={2020}
}
Zhengzhong Tu, zhengzhong.tu@utexas.edu