/Learning-Blind-Video-Temporal-Consistency-with-PWCNet

Re-implementation of ECCV'18 paper to smooth videos

Primary LanguagePythonMIT LicenseMIT

Learning Blind Video Temporal Consistency with PWCNet

Table of Contents

  1. Introduction
  2. Requirements and Dependencies
  3. Installation
  4. Dataset
  5. Apply Pre-trained Models
  6. Training and Testing
  7. Evaluation
  8. Image Processing Algorithms
  9. Acknowledge

Introduction

Our method is written based on Learning Blind Video Temporal Consistency, which takes the original unprocessed and per-frame processed videos as inputs to produce a temporally consistent video. Our approach is agnostic to specific image processing algorithms applied on the original video. The difference from our method to LBVC is that our method uses PWC-Net, a newer method to calculate the optical flow on pytorch1.0.0 and python3.6.

Requirements and dependencies

Our code is tested on Ubuntu 16.04 with cuda 9.0 and cudnn 7.0.

Installation

Download repository:

git clone git@github.com:zhaoyuzhi/Learning-Blind-Video-Temporal-Consistency-with-PWCNet.git

Dataset

cd data
./download_data.sh [train | test | all]
cd ..

I used the same dataset as LBVC did. The dataset include train and test. The Dataset including origin frames and per-frame processed videos.

Apply pre-trained models

I have trained the W3_D1_C1_I1 task, and the trained model are in: https://drive.google.com/file/d/1Nz6vqzIR0p2vGWIiQDwc6pV-i5DsOAw_/view?usp=sharing

You can just download it in the folder of this repository and unzip it to get the pre-trained model.

Test pre-trained model:

python test.py -method trained_model_example -epoch 100 -dataset DAVIS -task colorization/ECCV16

The output frames are saved in data/test/trained_model_example/epoch_100/colorization/ECCV16.

Training and testing

Train a new model:

python train.py -datasets_tasks W3_D1_C1_I1

The default parameters are specified in train.py. lists/train_tasks_W3_D1_C1_I1.txt specifies the dataset-task pairs for training.

Test a model:

python test.py -method MODEL_NAME -epoch N -dataset DAVIS -task WCT/wave

Check the checkpoint folder for the MODEL_NAME. The output frames are saved in data/test/MODEL_NAME/epoch_N/WCT/wave/DAVIS.

You can also generate results for multiple tasks using the following script:

python batch_test.py -method output/MODEL_NAME/epoch_N

which will test all the tasks in lists/test_tasks.txt.

Evaluation

Temporal Warping Error

To compute the temporal warping error, we first need to generate optical flow and occlusion masks:

python compute_flow_occlusion.py -dataset DAVIS -phase test

The flow will be stored in data/test/fw_flow/DAVIS. The occlusion masks will be stored in data/test/fw_occlusion/DAVIS.

Then, run the evaluation script:

python evaluate_WarpError.py -method output/MODEL_NAME/epoch_N -task WCT/wave

LPIPS

Download LPIPS repository and change LPIPS_dir in evalate_LPIPS.py if necesary (default path is ../LPIPS).

Run the evaluation script:

python evaluate_LPIPS.py -method output/MODEL_NAME/epoch_N -task WCT/wave

Batch evaluation

You can evaluate multiple tasks using the following script:

python batch_evaluate.py -method output/MODEL_NAME/epoch_N -metric LPIPS
python batch_evaluate.py -method output/MODEL_NAME/epoch_N -metric WarpError

which will evaluate all the tasks in lists/test_tasks.txt.

Test on new videos

To test our model on new videos or applications, please follow the folder structure in ./data.

Given a video, we extract frames named as %05d.jpg and save frames in data/test/input/DATASET/VIDEO.

The per-frame processed video is stored in data/test/processed/TASK/DATASET/VIDEO, where TASK is the image processing algorithm applied on the original video.

Image Processing Algorithms

We use the following algorithms to obtain per-frame processed results:

Style transfer

Image Enhancement

Intrinsic Image Decomposition

Image-to-Image Translation

Colorization

Acknowledge

Thank for the team of LBVC, their test give a new idea to deal the problem of video temporal consistency. And thank for The team of PWC-Net providing a more efficient method to calculate optical flow.