yzhouas/reproducible-image-restoration-state-of-the-art

Image Restoration Paper and Code, including Image Denoising, Image Supre-resolution, Image Inpainting etc.

Reproducible-image-restoration-state-of-the-art

State-of-the-arts of deep-learning-based image restoration works, including image super-resolution, denoising, inpainting, enhancement, and general restoration etc. Some of the codes may not be official, please double check them. The lists under each subsections may have overlaps. This list is maintained by Yuqian Zhou at IFP UIUC.

Information Sources

This collection is inspired and re-organized from the following sources,

• Denoising
  • summary by Bihan Wen
  • summary by flyywh
• Super-resolution
  • summary by yapengtian
  • survey of SISR

Image Super-resolution

We follow the survey of SISR to organize the related works in network design ideas. Lists under each subsections may have overlaps.

Supervised Methods

The deep learning based super-resolution starts from SRCNN.

• SRCNN [Web] [Code] [PDF]
  • Image Super-Resolution Using Deep Convolutional Networks(TPAMI15), Dong et al.

Residual Learning

• VDSR [Web] [Code] [PDF]
  • Accurate image superresolution using very deep convolutional networks (CVPR16), Kim et al.
• Memnet [Web] [Code] [PDF]
  • Memnet: A persistent memory network for image restoration(ICCV17), Tai et al.
• RED [Web] [Code] [PDF]
  • Image Restoration Using Very Deep Convolutional Encoder-Decoder Networks with Symmetric Skip Connections (NIPS2016), Mao et al.
• DRRN [Web] [Code] [PDF]
  • Image Super-Resolution via Deep Recursive Residual Network(CVPR17), Tai et al.
• IDN [Web] [Code] [PDF]
  • Fast and Accurate Single Image Super-Resolution via Information Distillation Network (CVPR18), Hui et al.
• EDSR [Web] [Code] [PDF]
  • Enhanced Deep Residual Networks for Single Image Super-Resolution(NTIRE2017), Lim et al.
• WDSR [Web] [Code] [PDF]
  • Wide Activation for Efficient and Accurate ImageSuper-Resolution(NTIRE2018), Yu et al.
  • Rank 1st model, wide activation
• RCAN [Web] [Code] [PDF]
  • Image Super-Resolution Using Very Deep Residual Channel Attention Networks(ECCV18), Zhang et al.
• RDN [Web] [Code] [PDF]
  • Residual Dense Network for Image Super-Resolution(CVPR18), Zhang et al.
• MSRN [Web] [Code] [PDF]
  • Multi-scale Residual Network for Image Super-Resolution(ECCV18), Li et al.
• DSRN [Web] [Code] [PDF]
  • Image Super-resolution via Dual-state Recurrent Neural Networks (CVPR18), Han et al.

Recursive Learning

Recurvise learning introduced in super-resolution is for larger receptive field and reduced parameters.

• DRCN [Web] [Code] [PDF]
  • Deeply-recursive convolutional network for image super-resolution(CVPR16), Kim et al.
• DRRN [Web] [Code] [PDF]
  • Image Super-Resolution via Deep Recursive Residual Network(CVPR17), Tai et al.
• Memnet [Web] [Code] [PDF]
  • Memnet: A persistent memory network for image restoration(ICCV17), Tai et al.
• CARN-M [Web] [Code] [PDF]
  • Fast, Accurate, and Lightweight Super-Resolution with Cascading Residual Network(ECCV18), Ahn et al.
• NLRN[Web] [Code] [PDF]
  • Non-Local Recurrent Network for Image Restoration (NeurIPS 2018), Liu et al.

Recurvise learning can resolve large scaling factor problem by solving multiple smaller factor problems.

• DSRN [Web] [Code] [PDF]
  • Image Super-resolution via Dual-state Recurrent Neural Networks (CVPR18), Han et al.
  • Explore HR-LR relationship
• LapSRN [Web] [Code] [PDF]
  • Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution(CVPR17), Lai et al.
• MS-LapSRN [Web] [Code] [PDF]
  • Fast and accurate image super-resolution with deep laplacian pyramid networks(TPAM18), Lai et al.

Multi-path Learning

Better separate modeling performance, but increasing the parameter size greatly.

• LapSRN [Web] [Code] [PDF]
  • Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution(CVPR17), Lai et al.
• DSRN [Web] [Code] [PDF]
  • Image Super-resolution via Dual-state Recurrent Neural Networks (CVPR18), Han et al.
  • Explore HR-LR relationship
• PixelSR [Web] [Code] [PDF]
  • Pixel Recursive Super Resolution(ICCV17), Dahl et al.
• MSRN [Web] [Code] [PDF]
  • Multi-scale Residual Network for Image Super-Resolution(ECCV18), Li et al.
  • Inspired by Inception Module
  • Two convolution operations inside each block for multiple-scales
• EDSR [Web] [Code] [PDF]
  • Enhanced Deep Residual Networks for Single Image Super-Resolution(NTIRE2017), Lim et al.
  • Scale-specific Multi-path Learning
• WDSR [Web] [Code] [PDF]
  • Wide Activation for Efficient and Accurate ImageSuper-Resolution(NTIRE2018), Yu et al.
  • Rank 1st model, wide activation

Dense Connections

• SR-DenseNet [Web] [Code] [PDF]
  • Image Super-Resolution Using Dense Skip Connections(CVPR17), Tong et al.
• Memnet [Web] [Code] [PDF]
  • Memnet: A persistent memory network for image restoration(ICCV17), Tai et al.
• CARN [Web] [Code] [PDF]
  • Fast, Accurate, and Lightweight Super-Resolution with Cascading Residual Network(ECCV18), Ahn et al.
• RDN [Web] [Code] [PDF]
  • Residual Dense Network for Image Super-Resolution(CVPR18), Zhang et al.
• ESRGAN [Web] [Code] [PDF]
  • ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks (ECCVW18), Wang et al.
• DBPN [Web] [Code] [PDF]
  • Deep Back-Projection Networks for Super-Resolution(CVPR18), Haris et al.

Attention-based

• RCAN [Web] [Code] [PDF]
  • Image Super-Resolution Using Very Deep Residual Channel Attention Networks(ECCV18), Zhang et al.
  • Channel-wise attention for different scales.
• Attention-FH [Web] [Code] [PDF]
  • Attention-aware face hallucination via deep reinforcement learning(ICCV17), Cao et al.
  • Motivated by human attention shifting mechanism
  • Face hallucination task

Special Types of Convolution

• IRCNN [Web] [Code] [PDF]
  • Learning Deep CNN Denoiser Prior for Image Restoration (CVPR17), Zhang et al.
  • Dilated Conv
• CARN-M [Web] [Code] [PDF]
  • Fast, Accurate, and Lightweight Super-Resolution with Cascading Residual Network(ECCV18), Ahn et al.
  • Group Conv

Wavelet Transform Domain

• DWSR [Web] [Code] [PDF]
  • Deep Wavelet Prediction for Image Super-resolution(CVPRW17), et al.
• Wavelet-SRNet [Web] [Code] [PDF]
  • Wavelet-SRNet: A Wavelet-based CNN for Multi-scale Face Super Resolution(ICCV17), Huang et al.
  • For face super-resolution task
• MWCNN [Web] [Code] [PDF]
  • Multi-level Wavelet-CNN for Image Restoration(CVPR18), Liu et al.

Other Special Designs

• PixelSR [Web] [Code] [PDF]
  • Pixel Recursive Super Resolution(ICCV17), Dahl et al.
  • Generate pixel by pixel
• EDSR-PP[Web] [Code] [PDF]
  • Efficient module based singleimage super resolution for multiple problems(CVPRW18), Park et al.
  • Incorporate pyramid pooling to EDSR: global and local contents included

Unsupervised Methods

When lacking in paired LR-HR data, unsupervised methods will be more effective in real-world scenarios.

Zero-shot

• ZSSR [Web] [Code] [PDF]
  • "Zero Shot" Super-Resolution using Deep Internal Learning(CVPR18),Shocher et al.
  • Kernal estimation + image-specific CNN network trained with constructed datasets
  • utilizing the internal image statistics

Weekly-supervised

• DegradationGAN [Web] [Code] [PDF]
  • To learn image super-resolution, use a gan to learn how to do image degradation first(ECCV18), Bulat et al.
  • Face super-resolution task
• CinCGAN [Web] [Code] [PDF]
  • Unsupervised Image Super-Resolutionusing Cycle-in-Cycle Generative Adversarial Network(CVPRW18), Yuan et al.
  • Cycle Consistency

Deep Image Prior

• Deep Image Prior [Web] [Code] [PDF]
  • Deep Image Prior(CVPR18), Ulyanov et al.
  • Handcrafted prior

Image Denoising

Normal CNN

• TNRD [Web] [Code] [PDF]
  • Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration (TPAMI 2016), Chen et al.

Residual Learning / DenseNet / Recursive Nets

• DnCNN [Web] [Code] [PDF]
  • Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising (TIP2017), Zhang et al.
• MemNet [Web] [Code] [PDF]
  • MemNet: A Persistent Memory Network for Image Restoration (ICCV2017), Tai et al.
• WIN [Web] [Code] [PDF]
  • Learning Pixel-Distribution Prior with Wider Convolution for Image Denoising (Arxiv), Liu et al.
• F-W Net [Web] [Code] [PDF]
  • L_p-Norm Constrained Coding With Frank-Wolfe Network (Arxiv), Sun et al.
• NLCNN [Web] [Code] [PDF]
  • Non-Local Color Image Denoising with Convolutional Neural Networks (CVPR 2017), Lefkimmiatis.
• Deep image prior [Web] [Code] [PDF]
• xUnit [Web] [Code] [PDF]
  • xUnit: Learning a Spatial Activation Function for Efficient Image Restoration (Arxiv), Kligvasser et al.
• UDNet [Web] [Code] [PDF]
  • Universal Denoising Networks : A Novel CNN Architecture for Image Denoising (CVPR 2018), Stamatios Lefkimmiatis.
• Wavelet-CNN [Web] [Code] [PDF]
  • Multi-level Wavelet-CNN for Image Restoration (Arxiv), Liu et al.
• FFDNet [Web] [Code] [PDF]
  • FFDNet: Toward a Fast and Flexible Solution for CNN-Based Image Denoising (TIP), Zhang et al.
• FC-AIDE [Web] [Code] [PDF]
  • Fully Convolutional Pixel Adaptive Image Denoiser (Arxiv), Cha et al.
• Noise2Noise [Web] [TF Code] [Keras Unofficial Code] [PDF]
  • Noise2Noise: Learning Image Restoration without Clean Data (ICML 2018), Lehtinen et al.
• UDN [Web] [Code] [PDF]
  • Universal Denoising Networks- A Novel CNN Architecture for Image Denoising (CVPR 2018), Lefkimmiatis.
• N3 [Web] [Code] [PDF]
  • Neural Nearest Neighbors Networks (NIPS 2018), Plotz et al.
• NLRN [Web] [Code] [PDF]
  • Non-Local Recurrent Network for Image Restoration (NIPS 2018), Liu et al.
• RDN+ [Web] [Code] [PDF]
  • Residual Dense Network for Image Restoration (CVPR 2018), Zhang et al.

UNet

• RED [Web] [Code] [PDF]
  • Image Restoration Using Very Deep Convolutional Encoder-Decoder Networks with Symmetric Skip Connections (NIPS2016), Mao et al.

Focusing on Real Noise

• CBDNet [Web] [Code] [PDF]
  • Toward Convolutional Blind Denoising of Real Photographs (ECCV18), Guo et al.
• Pixel-shuffle (PD) [Web] [Code] [PDF]
  • When AWGN-based Denoiser Meets Real Noises(Arxiv2019), Zhou et al.

Combined with Sparsity and Low-rankness

• STROLLR-2D [PDF] [Code]
  • When Sparsity Meets Low-Rankness: Transform Learning With Non-Local Low-Rank Constraint for Image Restoration (ICASSP 2017), Wen et al.

Combined with High-Level Tasks

• Meets High-level Tasks [PDF] [Code]
  • When Image Denoising Meets High-Level Vision Tasks: A Deep Learning Approach (IJCAI 2018), Liu et al.

Image Deblurring

• IRCNN [Web] [Code] [PDF]
  • Learning Deep CNN Denoiser Prior for Image Restoration (CVPR17) , Zhang et al.
• DeblurGAN [Web] [Code] [PDF]
  • DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks(CVPR18), Kupyn et al.

Image Inpainting

TODO

Image Enhancement

TODO

Image Restoration Tasks in a Single Model

• SRMD [Web] [Code] [PDF]
  • Learning a Single Convolutional Super-Resolution Network for Multiple Degradations (CVPR18), Zhou et al.

Useful Tools

• Motion Blur Generation [Web] [Code] [Paper]
  • Modeling the Performance of Image Restoration from Motion Blur(TIP2012), Boracchi and Foi et al.

Novel Benchmark

Novel benchmark is the ones captured in real-world senarios.

Super-resolution Benchmark

TODO

Denoising Benchmark

• ReNOIR [Web] [Data] [PDF]
  • RENOIR - A Dataset for Real Low-Light Image Noise Reduction (Arxiv 2014), Anaya, Barbu.
• Darmstadt [Web] [Data] [PDF]
  • Benchmarking Denoising Algorithms with Real Photographs (CVPR 2017), Tobias Plotz, Stefan Roth.
• PolyU [Web] [Data] [PDF]
  • Real-world Noisy Image Denoising: A New Benchmark (Arxiv), Xu et al.

Commonly Used Training/Evaluation Dataset for All Restoration Tasks

• BSD [Web]
• Kodak [Web]
• USC SIPI-Misc [Web]

Commonly Used Image Quality Metric Code

• PSNR (Peak Signal-to-Noise Ratio) [Wiki] [Matlab Code] [Python Code]
• SSIM (Structural similarity) [Wiki] [Matlab Code] [Python Code]
• NIQE (Naturalness Image Quality Evaluator) [Web] [Matlab Code] [Python Code]