/SPSR

Pytorch implementation of Structure-Preserving Super Resolution with Gradient Guidance (CVPR 2020 & TPAMI 2021)

Primary LanguagePython

SPSR

PyTorch implementation of Structure-Preserving Super Resolution with Gradient Guidance (CVPR 2020) [arXiv][CVF]

Extended version: Structure-Preserving Image Super-Resolution (TPAMI 2021) [arXiv]

If you find our work useful in your research, please consider citing:

@ARTICLE{ma2021structure,  
  author={Ma, Cheng and Rao, Yongming and Lu, Jiwen and Zhou, Jie},  
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},   
  title={Structure-Preserving Image Super-Resolution},   
  year={2021},  
  volume={},  
  number={},  
  pages={1-1},  
  doi={10.1109/TPAMI.2021.3114428}}

@inproceedings{ma2020structure,
  title={Structure-Preserving Super Resolution with Gradient Guidance},
  author={Ma, Cheng and Rao, Yongming and Cheng, Yean and Chen, Ce and Lu, Jiwen and Zhou, Jie},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2020}
}

Dependencies

  • Python 3 (Recommend to use Anaconda)
  • PyTorch >= 1.0
  • NVIDIA GPU + CUDA
  • Python packages: pip install numpy opencv-python lmdb pyyaml
  • TensorBoard:
    • PyTorch >= 1.1: pip install tb-nightly future
    • PyTorch == 1.0: pip install tensorboardX

Dataset Preparation

Download Datasets

Commonly used training and testing datasets can be downloaded here.

Preprocess Datasets

We also provide code to preprocess the datasets here.

  • After downloading the original datasets, please store them to a specific GT folder.

  • You can obtain the LR, HR and Bicubic-upsampled versions of the datasets.

  • Then you can extract sub-images with unified scales for training.

  • The training sets can also be transformed into LMDB format for faster IO speed.

Training

To train an SPSR model:

python train.py -opt options/train/train_spsr.json

  • The json file will be processed by options/options.py. Please refer to this for more details.

  • Before running this code, please modify train_spsr.json to your own configurations including:

    • the proper dataroot_HR and dataroot_LR paths for the data loader (More details)
    • saving frequency for models and states
    • whether to resume training with .state files
    • other hyperparameters
    • loss function, etc.
  • You can find your training results in ./experiments.

  • During training, you can use Tesorboard to monitor the losses with tensorboard --logdir tb_logger/NAME_OF_YOUR_EXPERIMENT

  • You can choose to use a pretrained RRDB model as a parameter initialization by setting the pretrain_model_G option in options/train/train_spsr.json. Please download the pretrained model from Google Drive or Baidu Drive (extraction code muw3) and place RRDB_PSNR_x4.pth into ./experiments/pretrain_models.

Testing

To generate SR images by an SPSR model:

python test.py -opt options/test/test_spsr.json

  • Similar to training, the configurations can be modified in the test_spsr.json file.

  • You can find your results in ./results.

  • We provide our SPSR model used in our paper that can be downloaded in Google Drive or Baidu Drive (extraction code muw3). Download spsr.pth and put it into ./experiments/pretrain_models. Then modify the directory of pretrained model in test_spsr.json and run test.py.

  • You can put your own LR images in a certain folder and just change the dataroot_LR setting in test_spsr.json and run test.py.

Evaluation Toolbox

We provide an easy and useful evaluation toolbox to simplify the procedure of evaluating SR results. In this toolbox, you can get the MA, NIQE, PI, PSNR, SSIM, MSE, RMSE, MAE and LPIPS values of any SR results you want to evaluate.

Results

Visual Results

Quantitative Results

From the below two tables of comparison with perceptual-driven SR methods, we can see our SPSR method is able to obtain the best PI and LPIPS performance and comparable PSNR and SSIM values simultaneously. The top 2 scores are highlighted.

PI/LPIPS comparison with perceptual-driven SR methods.

Method Set5 Set14 BSD100 General100 Urban100
Bicubic 7.3699/0.3407 7.0268/0.4393 7.0026/0.5249 7.9365/0.3528 6.9435/0.4726
SFTGAN 3.7587/0.0890 2.9063/0.1481 2.3774/0.1769 4.2878/0.1030 3.6136/0.1433
SRGAN 3.9820/0.0882 3.0851/0.1663 2.5459/0.1980 4.3757/0.1055 3.6980/0.1551
ESRGAN 3.7522/0.0748 2.9261/0.1329 2.4793/0.1614 4.3234/0.0879 3.7704/0.1229
NatSR 4.1648/0.0939 3.1094/0.1758 2.7801/0.2114 4.6262/0.1117 3.6523/0.1500
SPSR 3.2743/0.0644 2.9036/0.1318 2.3510/0.1611 4.0991/0.0863 3.5511/0.1184

PSNR/SSIM comparison with perceptual-driven SR methods.

Method Set5 Set14 BSD100 General100 Urban100
Bicubic 28.420/0.8245 26.100/0.7850 25.961/0.6675 28.018/0.8282 23.145/0.9011
SFTGAN 29.932/0.8665 26.223/0.7854 25.505/0.6549 29.026/0.8508 24.013/0.9364
SRGAN 29.168/0.8613 26.171/0.7841 25.459/0.6485 28.575/0.8541 24.397/0.9381
ESRGAN 30.454/0.8677 26.276/0.7783 25.317/0.6506 29.412/0.8546 24.360/0.9453
NatSR 30.991/0.8800 27.514/0.8140 26.445/0.6831 30.346/0.8721 25.464/0.9505
SPSR 30.400/0.8627 26.640/0.7930 25.505/0.6576 29.414/0.8537 24.799/0.9481

Acknowledgement

The code is based on BasicSR, MA, NIQE, PI, SSIM and LPIPS.

Contact

If you have any questions about our work, please contact macheng17@mails.tsinghua.edu.cn