/BSRGAN

Designing a Practical Degradation Model for Deep Blind Image Super-Resolution (ICCV, 2021) (PyTorch) - We released the training code!

Primary LanguagePython

Designing a Practical Degradation Model for Deep Blind Image Super-Resolution

visitors

Kai Zhang, Jingyun Liang, Luc Van Gool, Radu Timofte
Computer Vision Lab, ETH Zurich, Switzerland

[Paper] [Code] [Training Code]

Our work is the beginning rather than the end of real image super-resolution.

  • News (2021-08-31): We upload the training code.
  • News (2021-08-24): We upload the BSRGAN degradation model.
from utils import utils_blindsr as blindsr
img_lq, img_hq = blindsr.degradation_bsrgan(img, sf=4, lq_patchsize=72)
  • News (2021-07-23): After rejection by CVPR 2021, our paper is accepted by ICCV 2021. For the sake of fairness, we will not update the trained models in our camera-ready version. However, we may update the trained models in github.
  • News (2021-05-18): Add trained BSRGAN model for scale factor 2.
  • News (2021-04): Our degradation model for face image enhancement: https://github.com/vvictoryuki/BSRGAN_implementation

Training

  1. Download KAIR: git clone https://github.com/cszn/KAIR.git
  2. Put your training high-quality images into trainsets/trainH or set "dataroot_H": "trainsets/trainH"
  3. Train BSRNet
    1. Modify train_bsrgan_x4_psnr.json e.g., "gpu_ids": [0], "dataloader_batch_size": 4
    2. Training with DataParallel
    python main_train_psnr.py --opt options/train_bsrgan_x4_psnr.json
    1. Training with DistributedDataParallel - 4 GPUs
    python -m torch.distributed.launch --nproc_per_node=4 --master_port=1234 main_train_psnr.py --opt options/train_bsrgan_x4_psnr.json  --dist True
  4. Train BSRGAN
    1. Put BSRNet model (e.g., '400000_G.pth') into superresolution/bsrgan_x4_gan/models
    2. Modify train_bsrgan_x4_gan.json e.g., "gpu_ids": [0], "dataloader_batch_size": 4
    3. Training with DataParallel
    python main_train_gan.py --opt options/train_bsrgan_x4_gan.json
    1. Training with DistributedDataParallel - 4 GPUs
    python -m torch.distributed.launch --nproc_per_node=4 --master_port=1234 main_train_gan.py --opt options/train_bsrgan_x4_gan.json  --dist True
  5. Test BSRGAN model 'xxxxxx_E.pth' by modified main_test_bsrgan.py
    1. 'xxxxxx_E.pth' is more stable than 'xxxxxx_G.pth'

✨ Some visual examples: oldphoto2; butterfly; comic; oldphoto3; oldphoto6; comic_01; comic_03; comic_04

Testing code

Main idea

Design a new degradation model to synthesize LR images for training:

  • 1) Make the blur, downsampling and noise more practical
    • Blur: two convolutions with isotropic and anisotropic Gaussian kernels from both the HR space and LR space
    • Downsampling: nearest, bilinear, bicubic, down-up-sampling
    • Noise: Gaussian noise, JPEG compression noise, processed camera sensor noise
  • 2) Degradation shuffle: instead of using the commonly-used blur/downsampling/noise-addition pipeline, we perform randomly shuffled degradations to synthesize LR images

Some notes on the proposed degradation model:

  • The degradation model is mainly designed to synthesize degraded LR images. Its most direct application is to train a deep blind super-resolver with paired LR/HR images. In particular, the degradation model can be performed on a large dataset of HR images to produce unlimited perfectly aligned training images, which typically do not suffer from the limited data issue of laboriously collected paired data and the misalignment issue of unpaired training data.

  • The degradation model tends to be unsuited to model a degraded LR image as it involves too many degradation parameters and also adopts a random shuffle strategy.

  • The degradation model can produce some degradation cases that rarely happen in real-world scenarios, while this can still be expected to improve the generalization ability of the trained deep blind super-resolver.

  • A DNN with large capacity has the ability to handle different degradations via a single model. This has been validated multiple times. For example, DnCNN is able to handle SISR with different scale factors, JPEG compression deblocking with different quality factors and denoising for a wide range of noise levels, while still having a performance comparable to VDSR for SISR. It is worth noting that even when the super-resolver reduces the performance for unrealistic bicubic downsampling, it is still a preferred choice for real SISR.

  • One can conveniently modify the degradation model by changing the degradation parameter settings and adding more reasonable degradation types to improve the practicability for a certain application.

Comparison

These no-reference IQA metrics, i.e., NIQE, NRQM and PI, do not always match perceptual visual quality [1] and the IQA metric should be updated with new SISR methods [2]. We further argue that the IQA metric for SISR should also be updated with new image degradation types, which we leave for future work.

[1] "NTIRE 2020 challenge on real-world image super-resolution: Methods and results." CVPRW, 2020.
[2] "PIPAL: a large-scale image quality assessment dataset for perceptual image restoration." ECCV, 2020.

More visual results on RealSRSet dataset

Left: real images | Right: super-resolved images with scale factor 4

Visual results on DPED dataset

Without using any prior information of DPED dataset for training, our BSRGAN still performs well.

Citation

@inproceedings{zhang2021designing,
    title={Designing a Practical Degradation Model for Deep Blind Image Super-Resolution},
    author={Zhang, Kai and Liang, Jingyun and Van Gool, Luc and Timofte, Radu},
    booktitle={IEEE International Conference on Computer Vision},
    pages={4791--4800},
    year={2021}
}

Acknowledgments

This work was partly supported by the ETH Zurich Fund (OK), a Huawei Technologies Oy (Finland) project, and an Amazon AWS grant.