[CVPR 2023] Exploring Discontinuity for Video Frame Interpolation

This is the official PyTorch implementation of our paper:

Exploring Discontinuity for Video Frame Interpolation
Sangjin Lee*, Hyeongmin Lee*, Chajin Shin, Hanbin Son, Sangyoun Lee (*Equal Contribution)
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2023, Highlight

[Paper(Arxiv)] [Paper(CVPR)]

Overview

Abstract

We propose three techniques that can make the existing deep learning-based VFI architectures robust to practical videos that contain various unnatural objects with discontinuous motions. First is a novel data augmentation strategy called figure-text mixing (FTM) which can make the models learn discontinuous motions during training stage without any extra dataset. Second, we propose a simple but effective module that predicts a map called discontinuity map (D-map), which densely distinguishes between areas of continuous and discontinuous motions. Lastly, we propose loss functions to give supervisions of the discontinuous motion areas which can be applied along with FTM and D-map. We additionally collect a special test benchmark called Graphical Discontinuous Motion (GDM) dataset consisting of some mobile games and chatting videos. Applied to the various state-of-the-art VFI networks, our method significantly improves the interpolation qualities on the videos from not only GDM dataset, but also the existing benchmarks containing only continuous motions such as Vimeo90K, UCF101, and DAVIS.

Dataset

We construct a new test set called Graphic Discontinuous Motion (GDM) dataset which consists of high resolution videos of game scenes with abundant discontinuous motions. The dataset can be downloaded at: [Google Drive]

Evaluation

Prepared the dataset for evaluation
- Vimeo90K Septuplet (oringinal training + test set)
- DAVIS
Download the pretrained model and put the checkpoints folder under /src.

For evaluation, you can use the command below.

cd src
python test.py --gpu [gpu_id] --model ['AdaCoF' or 'CAIN' or 'VFIT'] --loss [True or False]

Note

Code, Dataset and models are only available for non-commercial research purposes.

If you have any questions, please feel free to contact me :)

sglee97@yonsei.ac.kr

Citation

@InProceedings{Lee_2023_CVPR,
    author    = {Lee, Sangjin and Lee, Hyeongmin and Shin, Chajin and Son, Hanbin and Lee, Sangyoun},
    title     = {Exploring Discontinuity for Video Frame Interpolation},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2023},
    pages     = {9791-9800}
}