PyTorch implementation of "Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation" by Jiang H., Sun D., Jampani V., Yang M., Learned-Miller E. and Kautz J. [Project] [Paper]
Check out our paper "Deep Slow Motion Video Reconstruction with Hybrid Imaging System" published in TPAMI.
Results on UCF101 dataset using the evaluation script provided by paper's author. The get_results_bug_fixed.sh script was used. It uses motions masks when calculating PSNR, SSIM and IE.
| Method | PSNR | SSIM | IE |
|---|---|---|---|
| DVF | 29.37 | 0.861 | 16.37 |
| SepConv - L_1 | 30.18 | 0.875 | 15.54 |
| SepConv - L_F | 30.03 | 0.869 | 15.78 |
| SuperSloMo_Adobe240fps | 29.80 | 0.870 | 15.68 |
| pretrained mine | 29.77 | 0.874 | 15.58 |
| SuperSloMo | 30.22 | 0.880 | 15.18 |
This codebase was developed and tested with pytorch 0.4.1 and CUDA 9.2 and Python 3.6. Install:
For GPU, run
conda install pytorch=0.4.1 cuda92 torchvision==0.2.0 -c pytorchFor CPU, run
conda install pytorch-cpu=0.4.1 torchvision-cpu==0.2.0 cpuonly -c pytorch- TensorboardX for training visualization
- tensorflow for tensorboard
- matplotlib for training graph in notebook.
- tqdm for progress bar in video_to_slomo.py
- numpy
In order to train the model using the provided code, the data needs to be formatted in a certain manner. The create_dataset.py script uses ffmpeg to extract frames from videos.
For adobe240fps, download the dataset, unzip it and then run the following command
python data\create_dataset.py --ffmpeg_dir path\to\folder\containing\ffmpeg --videos_folder path\to\adobe240fps\videoFolder --dataset_folder path\to\dataset --dataset adobe240fpsFor custom dataset, run the following command
python data\create_dataset.py --ffmpeg_dir path\to\folder\containing\ffmpeg --videos_folder path\to\adobe240fps\videoFolder --dataset_folder path\to\datasetThe default train-test split is 90-10. You can change that using command line argument --train_test_split.
Run the following commmand for help / more info
python data\create_dataset.py --hIn the train.ipynb, set the parameters (dataset path, checkpoint directory, etc.) and run all the cells.
or to train from terminal, run:
python train.py --dataset_root path\to\dataset --checkpoint_dir path\to\save\checkpointsRun the following commmand for help / more options like continue from checkpoint, progress frequency etc.
python train.py --hTo get visualization of the training, you can run tensorboard from the project directory using the command:
tensorboard --logdir log --port 6007and then go to https://localhost:6007.
You can download the pretrained model trained on adobe240fps dataset here.
You can convert any video to a slomo or high fps video (or both) using video_to_slomo.py. Use the command
# Windows
python video_to_slomo.py --ffmpeg path\to\folder\containing\ffmpeg --video path\to\video.mp4 --sf N --checkpoint path\to\checkpoint.ckpt --fps M --output path\to\output.mkv
# Linux
python video_to_slomo.py --video path\to\video.mp4 --sf N --checkpoint path\to\checkpoint.ckpt --fps M --output path\to\output.mkvIf you want to convert a video from 30fps to 90fps set fps to 90 and sf to 3 (to get 3x frames than the original video).
Run the following commmand for help / more info
python video_to_slomo.py --hYou can also use eval.py if you do not want to use ffmpeg. You will instead need to install opencv-python using pip for video IO.
A sample usage would be:
python eval.py data/input.mp4 --checkpoint=data/SuperSloMo.ckpt --output=data/output.mp4 --scale=4Use python eval.py --help for more details
More info TBA
Parts of the code is based on TheFairBear/Super-SlowMo