[CVPR 2023] ShadowDiffusion: When Degradation Prior Meets Diffusion Model for Shadow Removal

Paper

ShadowDiffusion: When Degradation Prior Meets Diffusion Model for Shadow Removal
Lanqing Guo, Chong Wang, Wenhan Yang, Siyu Huang, Yufei Wang, Hanspeter Pfister, Bihan Wen
In CVPR'2023

News

Aug 08, 2023: Release the pretrained model for SRD and evaluation results.
Aug 05, 2023: Release the training and testing codes.

Introduction

Our work addresses shadow removal by proposing a unified diffusion framework, dubbed ShadowDiffusion, that integrates both the image and degradation priors for highly effective shadow removal. ShadowDiffusion progressively refines the estimated shadow mask as an auxiliary task of the diffusion generator, which leads to more accurate and robust shadow-free image generation. For more details, please refer to our original paper.

Requirement

Python 3.7
Pytorch 1.7
CUDA 11.1

pip install -r requirements.txt

Datasets

ISTD [link]
ISTD+ [link]
SRD [Training][Testing] [Mask] (detected by DHAN)

Pretrained models

SRD | ISTD

Please download the corresponding pretrained model and modify the resume_state in shadow.json.

Test

You can directly test the performance of the pre-trained model as follows

Modify the paths to dataset and pre-trained model. You need to modify the following path in the shadow.json

resume_state # pretrain model or training state -- Line 12
dataroot # validation dataset path -- Line 30

Test the model

python sr.py -p val -c config/shadow.json

We use the DDIM sampling to speed up the inference stage. The number of steps can be set as 5 or 25.

skip = self.num_timesteps // 5
# skip = self.num_timesteps // 25

Train

Download datasets and set the following structure

|-- SRD_Dataset
    |-- train
        |-- train_A # shadow image
        |-- train_B # shadow mask
        |-- train_C # shadow-free GT
    |-- test
        |-- test_A # shadow image
        |-- test_B # shadow mask
        |-- test_C # shadow-free GT

You need to modify the following terms in option.py

"resume_state": null # if train from scratch
"dataroot"   # training and testing set path
"gpu_ids": [0] # Our model can be trained using a single RTX A5000 GPU. You can also train the model using multiple GPUs by adding more GPU ids in it.

Train the network

python sr.py -p train -c config/shadow.json

Evaluation

The results reported in the paper are calculated by the matlab script used in previous method. Details refer to evaluation/measure_shadow.m.

Testing results

The testing results on dataset ISTD, ISTD+, SRD are: results

References

Our implementation is based on SR3 and WeatherDiffusion. We would like to thank them.

Citation

Preprint available here.

In case of use, please cite our publication:

L. Guo, C. Wang, W. Yang, S. Huang, Y. Wang, P. Hanspeter, and B. Wen, "Shadowdiffusion: When degradation prior meets diffusion model for shadow removal," CVPR 2023.

Bibtex:

@inproceedings{guo2023shadowdiffusion,
  title={Shadowdiffusion: When degradation prior meets diffusion model for shadow removal},
  author={Guo, Lanqing and Wang, Chong and Yang, Wenhan and Huang, Siyu and Wang, Yufei and Pfister, Hanspeter and Wen, Bihan},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={14049--14058},
  year={2023}
}

Contact

If you have any questions, please contact lanqing001@e.ntu.edu.sg

TolgaAktas/ShadowDiffusion