Purpose: We aim to provide a summary of diffusion model-based image processing, including restoration, enhancement, coding, and quality assessment. More papers will be summarized.

Xin Li, Yulin Ren, Xin Jin, Cuiling Lan, Xingrui Wang, Wenjun Zeng, Xinchao Wang, Zhibo Chen

University of Science and Technology of China (USTC), National University of Singapore (NUS), Microsoft Research Asia (MSRA), Eastern Institute of Technology (EIT)

Brief intro: The survey for diffusion model-based IR has been released.

🔖 News!!!

2023-09-19: Updated new related works before 15/09/2023 in this GitHub.
2023-11-24: Updated new related works before 10/11/2023 in this GitHub.

📌 About new works. If you want to incorporate your studies (e.g., the link of paper or project) on diffusion model-based image processing in this repository. Welcome to raise an issue or email us. We will incorporate it into this repository and our survey report as soon as possible.

🌟 Features

Survey for diffusion model-based Image Restoration (Arxiv version is released)
Summary for diffusion model-based Image/Video Compression
Summary for diffusion model-based Quality Assessment

Diffusion model-based Image Restoration/Enhancement

Diffusion model for Image Super resolution
Diffusion model for Image Restoration
Diffusion model for Image Inpainting
Diffusion model for Image Shadow Removal
Diffusion model for Image Denoising
Diffusion model for Image Dehazing
Diffusion model for Image Deblurring
Diffusion model for Medical IR
Diffusion model for Low-Light Enchancement
Diffusion model for other tasks
Benchmark Datasets

Image Super-Resolution

Models	Paper	First Author	Training Way	Venue	Topic
SR3	Image super-resolution via iterative refinement	Chitwan Saharia	Supervised	TPAMI2022	Super-resolution
SRDiff	SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Models	Haoying Li	Supervised	Neurocomputing 2022	Super-resolution
SDE-SR	Face super-resolution using stochastic differential equations	Marcelo dos Santos	Supervised	SIBGRAPI 2022	Super-resolution
IDM	Implicit diffusion models for continuous super-resolution	Sicheng Gao	Supervised	CVPR2023	Super-resolution
PromptSR	Image Super-Resolution with Text Prompt Diffusion	Zheng Chen	Supervised	Preprint'23	Super-resolution
ACDPMSR	ACDMSR: Accelerated Conditional Diffusion Models for Single Image Super-Resolution	Axi Niu	Supervised	Preprint'23	Super-resolution
DiffIR	DiffIR: Efficient diffusion model for image restoration	Bin Xia	Supervised	ICCV 2023	Super-resolution
CDM	Cascaded Diffusion Models for High Fidelity Image Generation	Jonathan Ho	Supervised	J.Mach.Learn.Res 2022	Super-resolution
StableSR	Exploiting diffusion prior for real-world image super-resolution	Jianyi Wang	Supervised	Preprint'23	Real-World Super-resolution
SR3+	Denoising diffusion probabilistic models for robust image super-resolution in the wild	Hshmat Sahak	Supervised	Preprint'23	(Real-World) Super-resolution
ResShift	ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting	Zongsheng Yue	Supervised	Preprint'23	Real-World Super-resolution
TextDiff	TextDiff: Mask-Guided Residual Diffusion Models for Scene Text Image Super-Resolution	Baolin Liu	Supervised	Preprint'23	Super-resolution
PASD	Pixel-Aware Stable Diffusion for Realistic Image Super-resolution and Personalized Stylization	Tao Yang	Supervised Preprint'23	Super-resolution
EDiffSR	EDiffSR: An Efficient Diffusion Probabilistic Model for Remote Sensing Image Super-Resolution	Yi Xiao	Supervised	Preprint'23	Super-resolution
PartDiff	PartDiff: Image Super-resolution with Partial Diffusion Models	Kai Zhao	Supervised	Preprint'23	Super-resolution
Diwa	Waving Goodbye to Low-Res: A Diffusion-Wavelet Approach for Image Super-Resolution	Brian Moser	Supervised	Preprint'23	Super-resolution
YODA	YODA: You Only Diffuse Areas. An Area-Masked Diffusion Approach For Image Super-Resolution	Brian B. Moser	Supervised	Preprint'23	Super-resolution
CDPMSR	CDPMSR: Conditional Diffusion Probabilistic Models for Single Image Super-Resolution	Axi Niu	Supervised	Preprint'23	Super-resolution
ResDiff	ResDiff: Combining cnn and diffusion model for image superresolution	Shuyao Shang	Supervised	Preprint'23	Super-resolution
--	IMAGE SUPER-RESOLUTION VIA LATENT DIFFUSION: A SAMPLING-SPACE MIXTURE OF EXPERTS AND FREQUENCY-AUGMENTED DECODER APPROACH	Feng Luo	Supervised	Preprint'23	Super-resolution
--	Solving Diffusion ODEs with Optimal Boundary Conditions for Better Image Super-Resolution	Yiyang Ma	Supervised	Preprint'23	Super-resolution

Image Restoration

Model	Paper	First Author	Training Way	Venue	Topic
Palette	Palette: Image-toimage diffusion models	Chitwan Saharia	Supervised	SIGGRAPH 2022	Multi-task Restoration
Refusion	Refusion: Enabling large-size realistic image restoration with latent-space diffusion models	Ziwei Luo	Supervised	CVPRW 2023	Multi-task Restoration
InDI	Inversion by direct iteration: An alternative to denoising diffusion for image restoration	Mauricio Delbracio	Supervised	Preprint'22	Multi-task Restoration
Weather-Diff	Restoring vision in adverse weather conditions with patch-based denoising diffusion models	Ozan Özdenizci	Supervised	TPAMI2022	Multi-task Restoration
IR-SDE	Image Restoration with Mean-Reverting Stochastic Differential Equations	Ziwei Luo	Supervised	ICML 2023	Multi-task Restoration
SUD^2	SUD ˆ2: Supervision by denoising diffusion models for image reconstruction	Matthew A. Chan	Supervised	Preprint'23	Multi-task Restoration
ILVR	ILVR: Conditioning method for denoising diffusion probabilistic models	Jooyoung Choi	Zero-shot	ICCV2021	Image Restoration
CCDF	Come-closer-diffusefaster: Accelerating conditional diffusion models for inverse problems through stochastic contraction	Hyungjin Chung	Zero-shot	CVPR2022	Multi-task Restoration
SNIPS	SNIPS: Solving noisy inverse problems stochastically	Bahjat Kawa	Zero-shot	NeurIPS 2021	Multi-task Restoration
DDRM	Denoising diffusion restoration models	Bahjat Kawar	Zero-shot	ICLR2022	Multi-task Restoration
DDNM	Zero-shot image restoration using denoising diffusion null-space model	Yinhuai Wang	Zero-shot	ICLR2023	Multi-task Restoration
DPS	Diffusion Posterior Sampling for General Noisy Inverse Problems	Hyungjin Chung	Zero-shot	ICLR 2023	Multi-task Restoration
DiracDiffusion	DiracDiffusion: Denoising and incremental reconstruction with assured data-consistency	Zalan Fabian	Zero-shot	Preprint' 23	Multi-task Restoration
IIGDM	Pseudoinverse-guided diffusion models for inverse problems	Jiaming Song	Zero-shot	ICLR 2023	Multi-task Restoration
GDP	Generative diffusion prior for unified image restoration and enhancement	Ben Fei	Zero-shot	CVPR2023	Multi-task Restoration
DiffPIR	Denoising diffusion models for plug-and-play image restoration	Yuanzhi Zhu	Zero-shot	CVPR 2023	Multi-task Restoration
RED-Diff	A variational perspective on solving inverse problems with diffusion models	Morteza Mardani	Zero-shot	PrePrint'23	Multi-task Restoration
DriftRec	DriftRec: Adapting diffusion models to blind image restoration tasks	Simon Welker	Supervised	Preprint'22	Blind Restoration
DR2	DR2: Diffusion-based robust degradation remover for blind face restoration	Zhixin Wang	Zero-shot	CVPR2023	Blind Face Restoration
DiffBFR	DiffBFR: Bootstrapping diffusion model towards blind face restoration	Xinmin Qiu	Supervised	Preprint'23	Blind Face Restoration
DifFace	DifFace: Blind face restoration with diffused error contraction	Zongsheng Yu	Zero-shot	PrePrint'23	Blind Restoration
GibbsDDRM	GibbsDDRM: A partially collapsed gibbs sampler for solving blind inverse problems with denoising diffusion restoration	Naoki Murata	Zero-shot	ICML 2023	Blind Restoration
ADIR	ADIR: Adaptive diffusion for image reconstruction	Shady Abu-Hussein	Zero-shot	PrePrint'22	Blind Restoration
PSLD	Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Models	Litu Rout	Zero-shot	PrePrint'23	Image Restoration
IDM	Towards Authentic Face Restoration with Iterative Diffusion Models and Beyond	Yang Zhao	Supervised	ICCV 2023	Face Restoration
DiffBIR	DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior	Xinqi Lin	Supervised	Preprint' 2023	Image Restoration
RDDM	Residual Denoising Diffusion Models	Jiawei Liu	Supervised	Preprint' 2023	Image Restoration
C2F-DFT	Learning A Coarse-to-Fine Diffusion Transformer for Image Restoration	Liyan Wang	Supervised	Preprint' 2023	Image Restoration
MPerceiver	Multimodal Prompt Perceiver: Empower Adaptiveness, Generalizability and Fidelity for All-in-One Image Restoration	Yuang Ai	Supervised	Preprint'23	All-in-one Restoration
MCGdiff	Monte Carlo guided Diffusion for Bayesian linear inverse problems	Gabriel Cardoso	Zero-shot	Preprint' 2023	Image Restoration
AutoDIR	AutoDIR: Automatic All-in-One Image Restoration with Latent Diffusion	Yitong Jiang	Supervised	Preprint' 2023	Image Restoration
FastDiffusionEM	Fast Diffusion EM: a diffusion model for blind inverse problems with application to deconvolution	Charles Laroche	Supervised	Preprint' 2023	Blind Image Restoration
P2L	PROMPT-TUNING LATENT DIFFUSION MODELS FOR INVERSE PROBLEMS	Hyungjin Chung	Supervised	Preprint' 2023	Image Restoration
--	Multiscale Structure Guided Diffusion for Image Deblurring	Mengwei Ren	Supervised	Preprint'22	Blind Restoration
--	Synthesizing realistic image restoration training pairs: A diffusion approach	Tao Yang	Supervised	Preprint'22	Blind Restoration

Image Inpainting

Model	Paper	First Author	Training Way	Venue	Topic	Project
RePaint	RePaint: Inpainting using denoising diffusion probabilistic models	Andreas Lugmayr	Zero-shot	CVPR2022	Image Inpainting
CoPaint	Towards coherent image inpainting using denoising diffusion implicit models	Guanhua Zhang	Zero-shot	PrePrint'23	Image Inpainting

Image Shadow Removal

Model	Paper	First Author	Training Way	Venue	Topic
ShadowDiffusion	Shadowdiffusion: When degradation prior meets diffusion model for shadow removal	Lanqing Guo	Supervised	CVPR2023	Image Shadow Removal
DeS3	DeS3: Attention-driven Self and Soft Shadow Removal using ViT Similarity and Color Convergence	Yeying Jin	Supervised	Preprint'23	Image Shadow Removal
Deshadow-Anything	Deshadow-Anything: When Segment Anything Model Meets Zero-shot shadow removal	Xiao Feng Zhang	Supervised	Preprint'23	Image Shadow Removal

Image Denoising

Paper	First Author	Training Way	Venue	Topic
Diffusion model for generative image denoising	Yutong Xie	Supervised	Preprint'23	Image Denoising
Score-Based Diffusion Models as Principled Priors for Inverse Imaging	Berthy T. Feng	Zero-shot	ICCV 2023	Image Denoising, Image Deblurring
Real-World Denoising via Diffusion Model	Cheng Yang	Supervised	PrePrint'23	Image Denoising

Image Dehazing

Model	Paper	First Author	Training Way	Venue	Topic	Project
HazeDDPM	High-quality Image Dehazing with Diffusion Model	Hu Yu	Supervised	Preprint'23	Image Dehazing
--	Frequency Compensated Diffusion Model for Real-scene Dehazing	Jing Wang	Supervised	Preprint'23	Image Dehazing

Image Deblurring

Paper	First Author	Training Way	Venue	Topic
Deblurring via stochastic refinement	Jay Whang	Supervised	CVPR2022	Image Deblurring
Score-Based Diffusion Models as Principled Priors for Inverse Imaging	Berthy T. Feng	Zero-shot	ICCV 2023	Image Denoising, Image Deblurring
BlindDPS:Parallel diffusion models of operator and image for blind inverse problems	Hyungjin Chung	Zero-shot	CVPR2023	Blind Deblurring
HI-Diff:Hierarchical Integration Diffusion Model for Realistic Image Deblurring	Zheng Chen	Supervised	Prepreint'23	Image Deblurring

Medical Restoration (MRI,CT)

Model	Paper	First Author	Training Way	Venue	Topic
MCG	Improving diffusion models for inverse problems using manifold constraints	Hyungjin Chung	Zero-shot	NeurIPS 2022	CT Reconstruction
--	Solving inverse problems in medical imaging with score-based generative models	Yang Song	Zero-shot	ICLR 2022	CT Reconstruction
AdaDiff	Adaptive diffusion priors for accelerated mri reconstruction	Alper Güngör	Supervised	Preprint'23	MRI Reconstruction
HFS-SDE	High-Frequency Space Diffusion Models for Accelerated MRI	Chentao Cao	Supervised	Preprint'22	MRI Reconstruction
SWORD	Stage-by-stage Wavelet Optimization Refinement Diffusion Model for Sparse-View CT Reconstruction	Kai Xu	Supervised	Preprint'23	MRI Reconstruction
FDB	Learning Fourier-Constrained Diffusion Bridges for MRI Reconstruction	Muhammad U. Mirza	Supervised	Preprint'23	MRI Reconstruction

Low-Light Enchancement

Model	Paper	First Author	Training Way	Venue	Topic
PyDiff	Pyramid Diffusion Models For Low-light Image Enhancement	Dewei Zhou	Supervised	Preprint'23	Low-light Enhancemnent
DiffLL	Lowlight image enhancement with wavelet-based diffusion models	Hai Jiang	Supervised	Preprint'23	Low-light Enhancemnent
ExploreDiffusion	ExposureDiffusion: Learning to Expose for Low-light Image Enhancement	Yufei Wang	Supervised	Preprint'23	Low-light Enhancemnent
Diff-Retinex	Rethinking Low-light Image Enhancement with A Generative Diffusion Model	Xunpeng Yi	Supervised	ICCV 2023	Low-light Enhancemnent
CLE Diffusion	CLE Diffusion: Controllable Light Enhancement Diffusion Model	Yuyang Yin	Supervised	Preprint'23	Low-light Enhancemnent
DiffLLE	DiffLLE: Diffusion-guided Domain Calibration for Unsupervised Low-light Image Enhancement	Shuzhou Yang	Supervised	Preprint'23	Low-light Enhancemnent
LLDiffusion	LLDiffusion: Learning Degradation Representations in Diffusion Models for Low-Light Image Enhancement	Tao Wang	Supervised	Preprint'23	Low-light Enhancemnent

Other tasks

Model	Paper	First Author	Training Way	Venue	Topic
RainDiffusion	RainDiffusion: When Unsupervised Learning Meets Diffusion Models for Real-world Image Deraining	Mingqiang Wei	Supervised	Preprint'23	Image Deraining
DiffGAR	DiffGAR: Model-agnostic restoration from generative artifacts using image-to-image diffusion models	Yueqin Yin	Supervised	Preprint'22	Generative Artifacts Removal
HSR-Diff	HSR-Diff: Hyperspectral Image Super-Resolution via Conditional Diffusion Models	Chanyue Wu	Supervised	Preprint'23	hyperspectral images super-resolution
DRUS	Ultrasound Image Reconstruction with Denoising Diffusion Restoration Models	Yuxin Zhang	Zero-shot	Preprint'23	Ultrasound Image Reconstruction
--	Jpeg artifact correction using denoising diffusion restoration models	Bahjat Kawar	Zero-shot	NeurIPS 2022	JPEG Artifacts Removal
--	Improved Diffusion-based Image Colorization via Piggybacked Models	Hanyuan Liu	Zero-shot	PrePrint'23	Image Colorization

Benchmark Datasets

Dataset	Task	Usage	Year
DIV2K	Image Super-resolution	Training,Testing	2017
Flickr2K	Image Super-resolution	Training	2017
Set5	Image Super-resolution	Testing	2012
Set14	Image Super-resolution	Testing	2012
BSD100	Image Super-resolution	Testing	2012
Manga109	Image Super-resolution	Testing	2015
Urban100	Image Super-resolution	Testing	2015
OST300	Image Super-resolution	Testing	2018
DIV8K	Image Super-resolution	Training,Testing	2019
RealSR	Image Super-resolution	Training,Testing	2019
DRealSR	Image Super-resolution	Training,Testing	2020
GoPro	Image Deblurring	Training,Testing	2017
HIDE	Image Deblurring	Training,Testing	2019
RealBlur	Image Deblurring	Training,Testing	2020
Kodak	Image Denoising	Testing	1999
CBSD68	Image Denoising	Testing	2001
McMaster	Image Denoising	Testing	2011
ImageNet	Image Classification	Training,Testing	2010
ImageNet1k	Image Classification	Testing	2020
LSUN	Image Classification	Training,Testing	2015
Places365	Image Classification	Training,Testing	2019
LFW	Face Generation	Training	2008
FFHQ	Face Generation	Training	2019
Celeba-HQ	Face Generation	Training	2018
AFHQ	Face Generation	Training	2020
CelebA	Face Generation	Training	2015
ISTD	Image Shadow Removal	Training,Testing	2018
SRD	Image Shadow Removal	Training,Testing	2017
CSD	Image Desnowing	Training,Testing	2021
Snow100k	Image Desnowing	Training,Testing	2017
SRRS	Image Desnowing	Training,Testing	2020
RainDrop	Image Deraining	Training	2018
Outdoor-Rain	Image Deraining	Training,Testing	2019
DDN-data	Image Deraining	Training,Testing	2017
SPA-data	Image Deraining	Training,Testing	2019
Rain100H	Image Deraining	Training,Testing	2017
Rain100L	Image Deraining	Training,Testing	2017
Haze-4K	Image Dehazing	Training	2021
Dense-Haze	Image Dehazing	Training	2019
RESIDE	Image Dehazing	Training	2019

Diffusion model-based Image/Video Compression

Model	Paper	First Author	Venue	Topic
DiffC	Lossy Compression with Gaussian Diffusion	Lucas Theisl	Preprint'22	Lossy Image Compression
CDC	Lossy Image Compression with Conditional Diffusion Models	Ruihan Yang	ICLR 2023	Lossy Image Compression
--	Extreme Generative Image Compression by Learning Text Embedding from Diffusion Models	Zhihong Pan	Preprint'22	Image Compression
DIRAC	A Residual Diffusion Model for High Perceptual Quality Codec Augmentation	Noor Fathima Ghouse	Preprint'23	Lossy Image Compression
HFD	High-Fidelity Image Compression with Score-based Generative Models	Emiel Hoogeboom	Preprint'23	Lossy Image Compression
PerCo	TOWARDS IMAGE COMPRESSION WITH PERFECT REALISM AT ULTRA-LOW BITRATES	Marl` ene Careil	Preprint'23	Lossy Image Compression
PIC	Text + Sketch: Image Compression at Ultra Low Rates	Eric Lei	Preprint'23	Lossy Image Compression
--	TOWARDS IMAGE COMPRESSION WITH PERFECT REALISM AT ULTRA-LOW BITRATES	Marlène Careil	PrePrint'23	Lossy Image Compression

Diffusion model-based Image quality assessment

Model	Paper	First Author	Venue	Topic	Project
DifFIQA	DifFIQA: Face Image Quality Assessment Using Denoising Diffusion Probabilistic Models	Žiga Babnik	Preprint'23	Image quality assessment

Cite US

If this work is helpful to you, we expect you can cite this work and star this repo. Thanks.

@article{li2023diffusion,
  title={Diffusion Models for Image Restoration and Enhancement--A Comprehensive Survey},
  author={Li, Xin and Ren, Yulin and Jin, Xin and Lan, Cuiling and Wang, Xingrui and Zeng, Wenjun and Wang, Xinchao and Chen, Zhibo},
  journal={arXiv preprint arXiv:2308.09388},
  year={2023}
}

anhuipl2010/Awesome-diffusion-model-for-image-processing