/HINet

Primary LanguagePythonOtherNOASSERTION

PWC PWC PWC PWC PWC PWC PWC

HINet: Half Instance Normalization Network for Image Restoration

Liangyu Chen, Xin Lu, Jie Zhang, Xiaojie Chu, Chengpeng Chen

In this paper, we explore the role of Instance Normalization in low-level vision tasks. Specifically, we present a novel block: Half Instance Normalization Block (HIN Block), to boost the performance of image restoration networks. Based on HIN Block, we design a simple and powerful multi-stage network named HINet, which consists of two subnetworks. With the help of HIN Block, HINet surpasses the state-of-the-art (SOTA) on various image restoration tasks. For image denoising, we exceed it 0.11dB and 0.28 dB in PSNR on SIDD dataset, with only 7.5% and 30% of its multiplier-accumulator operations (MACs), 6.8 times and 2.9 times speedup respectively. For image deblurring, we get comparable performance with 22.5% of its MACs and 3.3 times speedup on REDS and GoPro datasets. For image deraining, we exceed it by 0.3 dB in PSNR on the average result of multiple datasets with 1.4 times speedup. With HINet, we won 1st place on the NTIRE 2021 Image Deblurring Challenge - Track2. JPEG Artifacts, with a PSNR of 29.70.

Network Architecture

arch

Installation

This implementation based on BasicSR which is a open source toolbox for image/video restoration tasks.

python 3.6.9
pytorch 1.5.1
cuda 10.1
git clone https://github.com/megvii-model/HINet
cd HINet
pip install -r requirements.txt
python setup.py develop --no_cuda_ext

Quick Start (Single Image Inference)


Image Restoration Tasks


Image denoise, deblur, derain.

Image Denoise - SIDD dataset (Click to expand)
  • prepare data

    • mkdir ./datasets/SIDD

    • download the [train]( SIDD-Medium sRGB Dataset in https://www.eecs.yorku.ca/~kamel/sidd/dataset.php) set and unzip it. Then move Data (./SIDD_Medium_Srgb/Data) set to ./datasets/SIDD/ . Download val files (ValidationNoisyBlocksSrgb.mat and ValidationGtBlocksSrgb.mat) in ./datasets/SIDD/ .

    • it should be like:

      ./datasets/SIDD/Data
      ./datasets/SIDD/ValidationNoisyBlocksSrgb.mat
      ./datasets/SIDD/ValidationGtBlocksSrgb.mat
    • python scripts/data_preparation/sidd.py

      • crop the train image pairs to 512x512 patches.
  • eval

  • train

    • python -m torch.distributed.launch --nproc_per_node=8 --master_port=4321 basicsr/train.py -opt options/train/SIDD/HINet.yml(HINet_0.5x.yml) --launcher pytorch
    • data in lmdb format will lose about 0.01 value in PSNR
Image Deblur - GoPro dataset (Click to expand)
  • prepare data

    • mkdir ./datasets/GoPro

    • download the train set in ./datasets/GoPro/train and test set in ./datasets/GoPro/test (refer to MPRNet)

    • it should be like:

      ./datasets/
      ./datasets/GoPro/
      ./datasets/GoPro/train/
      ./datasets/GoPro/train/input/
      ./datasets/GoPro/train/target/
      ./datasets/GoPro/test/
      ./datasets/GoPro/test/input/
      ./datasets/GoPro/test/target/
    • python scripts/data_preparation/gopro.py

      • crop the train image pairs to 512x512 patches.
  • eval

    • download pretrained model to ./experiments/pretrained_models/HINet-GoPro.pth
    • python basicsr/test.py -opt options/test/GoPro/HINet-GoPro.yml
  • train

    • python -m torch.distributed.launch --nproc_per_node=8 --master_port=4321 basicsr/train.py -opt options/train/GoPro/HINet.yml --launcher pytorch
Image Deblur - REDS dataset (Click to expand)
  • prepare data

    • mkdir ./datasets/REDS

    • download the train / val set from train_blur, train_sharp, val_blur, val_sharp to ./datasets/REDS/ and unzip them.

    • it should be like

      ./datasets/
      ./datasets/REDS/
      ./datasets/REDS/val/
      ./datasets/REDS/val/val_blur_jpeg/
      ./datasets/REDS/val/val_sharp/
      ./datasets/REDS/train/
      ./datasets/REDS/train/train_blur_jpeg/
      ./datasets/REDS/train/train_sharp/
      
    • python scripts/data_preparation/reds.py

      • flatten the folders and extract 300 validation images.
  • eval

    • download pretrained model to ./experiments/pretrained_models/HINet-REDS.pth
    • python basicsr/test.py -opt options/test/REDS/HINet-REDS.yml
  • train

    • python -m torch.distributed.launch --nproc_per_node=8 --master_port=4321 basicsr/train.py -opt options/train/REDS/HINet.yml --launcher pytorch
Image Derain - Rain13k dataset (Click to expand)
  • prepare data

    • mkdir ./datasets/Rain13k

    • download the train set and test set (refer to MPRNet)

    • it should be like

      ./datasets/
      ./datasets/Rain13k/
      ./datasets/Rain13k/train/
      ./datasets/Rain13k/train/input/
      ./datasets/Rain13k/train/target/
      ./datasets/Rain13k/test/
      ./datasets/Rain13k/test/Test100/
      ./datasets/Rain13k/test/Rain100H/
      ./datasets/Rain13k/test/Rain100L/
      ./datasets/Rain13k/test/Test2800/
      ./datasets/Rain13k/test/Test1200/
      
  • eval

    • download pretrained model to ./experiments/pretrained_models/HINet-Rain13k.pth

    • For Test100:

      • python basicsr/test.py -opt options/test/Rain13k/HINet-Test100.yml
    • For Rain100H

      • python basicsr/test.py -opt options/test/Rain13k/HINet-Rain100H.yml
    • For Rain100L

      • python basicsr/test.py -opt options/test/Rain13k/HINet-Rain100L.yml
    • For Test2800

      • python basicsr/test.py -opt options/test/Rain13k/HINet-Test2800.yml
    • For Test1200

      • python basicsr/test.py -opt options/test/Rain13k/HINet-Test1200.yml
  • train

    • python -m torch.distributed.launch --nproc_per_node=8 --master_port=4321 basicsr/train_rain.py -opt options/train/Rain13k/HINet.yml --launcher pytorch

Results


Some of the following results are higher than the original paper as we optimized some hyper-parameters.

NTIRE2021 Deblur Track2 ResultSIDD ResultGoPro Result
REDDS ResultRain13k Result

License

This project is under the MIT license, and it is based on BasicSR which is under the Apache 2.0 license.

Citations

If HINet helps your research or work, please consider citing HINet.

@InProceedings{Chen_2021_CVPR,
    author    = {Chen, Liangyu and Lu, Xin and Zhang, Jie and Chu, Xiaojie and Chen, Chengpeng},
    title     = {HINet: Half Instance Normalization Network for Image Restoration},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
    month     = {June},
    year      = {2021},
    pages     = {182-192}
}

Contact

If you have any questions, please contact chenliangyu@megvii.com or luxin@megvii.com .