DDPS: Denoising Diffusion Prior for Segmentation

Zeqiang Lai*, Yuchen duan*, Jifeng Dai, Ziheng Li, Ying Fu, Hongsheng Li, Yu Qiao, and Wenhai Wang

Denoising Diffusion Semantic Segmentation with Mask Prior Modeling.

DDPS explore the mask prior modeled by a recently developed denoising diffusion generative model for ameliorating the semantic segmentation quality of existing discriminative approaches. DDPS focuses on a discrete instantiation of a unified architecture that adapts diffusion models for mask prior modeling, with two novel training and inference strategies, i.e., Diffusion-on-First-Prediction diffusion strategy and Free-Re-noising denoising strategy.

Working in progress.

News

[TBD] Integrate an online demo into InternGPT.
[2023/7/26] Release preview code and pretrained models.
[2023/6/2] Release paper to arXiv.

DDPS Model

Semantic segmentation performance on ADE20K.

Method	Backbone	Params (M)	mIoU (ss)	mIoU (ms)	mBIoU	Download
DDPS-DeeplabV3+	MobibleNetV2	27.9	38.07	39.07	23.47	TBD
DDPS-Segformer	MiT-B0	15.1	41.67	41.95	26.77	TBD
DDPS-DeeplabV3+	ResNet50	54.2	45.26	46.01	30.39	checkpoint
DDPS-Segformer	MiT-B2	65.5	49.73	49.96	33.95	checkpoint
DDPS-DeeplabV3+	ResNet101	104.3	46.32	46.99	31.82	checkpoint
DDPS-Segformer	MiT-B5	122.8	51.11	51.71	35.66	TBD

Semantic segmentation performance on Cityscapes.

Method	Backbone	Params (M)	mIoU (ss)	mIoU (ms)	mBIoU	Download
DDPS-DeeplabV3+	MobibleNetV2	27.9	78.11	79.73	61.14	TBD
DDPS-Segformer	MiT-B0	15.1	78.19	79.62	64.07	checkpoint
DDPS-DeeplabV3+	ResNet50	54.2	81.39	82.27	65.76	checkpoint
DDPS-Segformer	MiT-B2	65.5	81.77	82.22	67.75	checkpoint
DDPS-DeeplabV3+	ResNet101	104.3	81.68	82.51	65.96	checkpoint
DDPS-Segformer	MiT-B5	122.8	82.42	82.94	66.45	TBD

Installation

See environment.yaml for details.

Usage

Evaluation

sh tools/dist_test_diffusion_origin.sh configs/ade20k/segformer_b2_ade20k_multistep.py checkpoints/ade20k/segformer_b2_multistep/best_mIoU_iter_144000.pth  8 --eval "mIoU"

We are still cleaning up the code and checkpoints, so the results might slightly differ from the reported ones, but you should expect the following results with the above command.

+-------------------+
|    mIoU 0,10,19   |
+-------------------+
| 49.35,49.55,49.66 |
+-------------------+

Visualization

Mask Prior Modeling

Citation

If you find this work helpful, please consider cite our paper.

@misc{lai2023ddps,
  title = {Denoising Diffusion Semantic Segmentation with Mask Prior Modeling},
  author = {Zeqiang Lai and Yuchen duan and Jifeng Dai and Ziheng Li and Ying Fu and Hongsheng Li and Yu Qiao and Wenhai Wang},
  publisher = {arXiv},
  year = {2023},
}

Acknowledgement

MMSegmentation VQDiffusion Segformer Diffusers

OpenGVLab/DDPS