ESTRNN & BSD

Efficient Spatio-Temporal Recurrent Neural Network for Video Deblurring (ECCV2020 Spotlight)

Journal version (under review; new BSD dataset)

Conference version (old BSD dataset)

by Zhihang Zhong, Ye Gao, Yinqiang Zheng, Bo Zheng

This work presents an efficient RNN-based model and the first real-world dataset for video deblurring :)

Results

Results on REDS

Results on GOPRO

Results on BSD

Prerequisites

Python 3.6
PyTorch 1.6 with GPU
opencv-python
scikit-image
lmdb
thop
tqdm
tensorboard

Beam-Splitter Deblurring Dataset (BSD)

We have collected a new real-world video deblurring dataset (BSD) with more scenes and better setups (center-aligned), using the proposed beam-splitter acquisition system:

The configurations of the new BSD dataset are as below:

Results on different setups of BSD:

Pretrained models for different setups:

Please download checkpoints and unzip it under the main directory.

Example command to run a pre-trained model:

python main.py --test_only --test_checkpoint ./checkpoints/ESTRNN_C80B15_BSD_3ms24ms.tar --ds_config 3ms24ms --video

Training

Please download and unzip the dataset file for each benchmark.

Then, specify the <path> (e.g. "./dataset/ ") where you put the dataset file and the corresponding dataset configurations in the command, or change the default values in "./para/paramter.py".

Training command is as below:

python main.py --data_root <path> --dataset BSD --ds_config 2ms16ms --data_format RGB

You can also tune the hyper parameters such as batch size, learning rate, epoch number, etc. (P.S.: the actual batch size for ddp mode is num_gpus*batch_size)

python main.py --lr 1e-4 --batch_size 4 --num_gpus 2 --trainer_mode ddp

If you want to train on your own dataset, please refer to "/data/how_to_make_dataset_file.ipynb".

Citing

If you use any part of our code, or ESTRNN and BSD are useful for your research, please consider citing:

@inproceedings{zhong2020efficient,
  title={Efficient spatio-temporal recurrent neural network for video deblurring},
  author={Zhong, Zhihang and Gao, Ye and Zheng, Yinqiang and Zheng, Bo},
  booktitle={European Conference on Computer Vision},
  pages={191--207},
  year={2020},
  organization={Springer}
}

@misc{zhong2021efficient,
      title={Efficient Spatio-Temporal Recurrent Neural Network for Video Deblurring}, 
      author={Zhihang Zhong and Ye Gao and Yinqiang Zheng and Bo Zheng and Imari Sato},
      year={2021},
      eprint={2106.16028},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

lingdomuw/ESTRNN