/UniMatch

[CVPR 2023] Revisiting Weak-to-Strong Consistency in Semi-Supervised Semantic Segmentation

Primary LanguagePythonMIT LicenseMIT

UniMatch

This codebase contains a strong re-implementation of FixMatch in the scenario of semi-supervised semantic segmentation, as well as the official PyTorch implementation of our paper:

Revisiting Weak-to-Strong Consistency in Semi-Supervised Semantic Segmentation
Lihe Yang, Lei Qi, Litong Feng, Wayne Zhang, Yinghuan Shi
In Conference on Computer Vision and Pattern Recognition (CVPR), 2023

We have another semi-supervised semantic segmentation work accepted by CVPR 2022: ST++

Abstract. In this work, we revisit the weak-to-strong consistency framework, popularized by FixMatch from semi-supervised classification, where the prediction of a weakly perturbed image serves as supervision for its strongly perturbed version. Intriguingly, we observe that such a simple pipeline already achieves competitive results against recent advanced works, when transferred to our segmentation scenario. Its success heavily relies on the manual design of strong data augmentations, however, which may be limited and inadequate to explore a broader perturbation space. Motivated by this, we propose an auxiliary feature perturbation stream as a supplement, leading to an expanded perturbation space. On the other, to sufficiently probe original image-level augmentations, we present a dual-stream perturbation technique, enabling two strong views to be simultaneously guided by a common weak view. Consequently, our overall Unified Dual-Stream Perturbations approach (UniMatch) surpasses all existing methods significantly across all evaluation protocols on the Pascal, Cityscapes, and COCO benchmarks. We also demonstrate the superiority of our method in remote sensing interpretation and medical image analysis.

Results

Pascal

Labeled images are sampled from the original high-quality training set. Results are obtained by DeepLabv3+ based on ResNet-101 with training size 321.

Method 1/115 (92) 1/57 (183) 1/28 (366) 1/14 (732) 1/7 (1464)
SupOnly 45.1 55.3 64.8 69.7 73.5
U2PL 68.0 69.2 73.7 76.2 79.5
ST++ 65.2 71.0 74.6 77.3 79.1
PS-MT 65.8 69.6 76.6 78.4 80.0
UniMatch (Ours) 75.2 77.2 78.8 79.9 81.2

Cityscapes

Results are obtained by DeepLabv3+ based on ResNet-50/101. We reproduce U2PL results on ResNet-50.

ResNet-50 1/30 1/8 1/4 ResNet-101 1/16 1/8 1/4
SupOnly 56.8 70.8 73.7 SupOnly 67.9 73.5 75.4
U2PL 59.8 73.0 76.3 U2PL 74.9 76.5 78.5
UniMatch (Ours) 64.5 75.6 77.4 UniMatch (Ours) 75.7 77.3 78.7

COCO

Results are obtained by DeepLabv3+ based on Xception-65.

Method 1/512 (232) 1/256 (463) 1/128 (925) 1/64 (1849) 1/32 (3697)
SupOnly 22.9 28.0 33.6 37.8 42.2
PseudoSeg 29.8 37.1 39.1 41.8 43.6
PC2Seg 29.9 37.5 40.1 43.7 46.1
UniMatch (Ours) 32.2 40.4 46.2 48.7 51.2

Getting Started

Installation

cd UniMatch
conda create -n unimatch python=3.6.9
conda activate unimatch
pip install -r requirements.txt
pip install pip install torch==1.8.1+cu102 torchvision==0.9.1+cu102 -f https://download.pytorch.org/whl/torch_stable.html

Pretrained Backbone:

ResNet-50 | ResNet-101 | Xception-65

├── ./pretrained
    ├── resnet50.pth
    ├── resnet101.pth
    └── xception.pth

Dataset:

Please modify the dataset path in configuration files.

The groundtruth mask ids have already been pre-processed. You may use them directly.

├── [Your Pascal Path]
    ├── JPEGImages
    └── SegmentationClass
    
├── [Your Cityscapes Path]
    ├── leftImg8bit
    └── gtFine
    
├── [Your COCO Path]
    ├── train2017
    ├── val2017
    └── masks

Usage

UniMatch

# use torch.distributed.launch
sh tools/train.sh <num_gpu> <port>

# or use slurm
# sh tools/slurm_train.sh <num_gpu> <port> <partition>

To run on different labeled data partitions or different datasets, please modify:

config, labeled_id_path, unlabeled_id_path, and save_path in train.sh.

FixMatch

Modify the unimatch.py to fixmatch.py in train.sh.

Supervised Baseline

Modify the unimatch.py to supervised.py in train.sh, and double the batch_size in configuration file if you use the same number of GPUs as semi-supervised setting (no need to change lr).

Citation

If you find these projects useful, please consider citing:

@inproceedings{unimatch,
  title={Revisiting Weak-to-Strong Consistency in Semi-Supervised Semantic Segmentation},
  author={Yang, Lihe and Qi, Lei and Feng, Litong and Zhang, Wayne and Shi, Yinghuan},
  booktitle={CVPR},
  year={2023}
}

@inproceedings{st++,
  title={ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentation},
  author={Yang, Lihe and Zhuo, Wei and Qi, Lei and Shi, Yinghuan and Gao, Yang},
  booktitle={CVPR},
  year={2022}
}

Acknowledgement

We thank AEL, CPS, CutMix-Seg, DeepLabv3Plus, PseudoSeg, PS-MT, SimpleBaseline, U2PL and other relevant works (see below) for their amazing open-sourced projects!

Semi-Supervised Semantic Segmentation Projects