SeCo

Semantic Connectivity-Driven Pseudo-labeling for Cross-domain Segmentation.

📔 For more information, please see our paper at ARXIV

💬 Pixel-driven PL. v.s. Connectivity-driven PL

💬 Pipeline

💬 Requirements

Python 3.8.0
pytorch 1.10.1
torchvision 0.11.2
einops  0.3.2

Please see requirements.txt for all the other requirements.

You can use PSA and SCC to obtain high-purity connectivity-based pseudo-labels. These pseudo-labels can then be exploited and embedded into existing unsupervised domain adaptative semantic segmentation methods.

💬 Pixel Semantic Aggregation

First, you can obtain pixel-level pseudo-labels by pixel thresholding (e.g.cbst ) from a UDA method (e.g. ProDA ) or a source-free UDA method (e.g. DTST), or a UDG method (e.g. SHADE ).
And organize them in the following format.

"""
├─image
├─pixel-level pseudo-label
└─list
"""
list (XXX.txt) records the image names (XXX.png) and their corresponding pixel-level pseudo-labels.

Then, run the PSA as follows:

${exp_name}="HRDA_seco"
CUDA_VISIBLE_DEVICES="1"  nohup python seco_sam.py --id-list-path  ./splits/cityscapes/${exp_name}/all.txt --class-num ${class_name}  > logs/${exp_name} 2>&1 &

Afterward, you can find the aggregated pseudo-labels in root_path/${exp_name}_vit_{B/H}.

💬 Semantic Connectivity Correction

After PSA, the noise is also amplified, and then you can use SCC to denoise the connected regions. Refer to (SCC ) part for specific instructions.

💬 Embedded SeCo into UDA semantic segmentation

In the paper, we leverage the pseudo-labels generated by SeCo across multiple codebases. Due to the diversity in code structures, we provide a unified implementation for utilizing SeCo's pseudo-labels. This involves partitioning the unlabeled data into two subsets (one as a labeled subset and the other as an unlabeled subset) and employing a semi-supervised method (Unimatch ) for further adaptation, as mentioned in the paper.

Adaptation task	model	Before adaptation	After adaptation	Training logs
GTA → Cityscapes	deeplab-r101	55.1 DTST	64.6	`training_logs/seco_gta`
Synthia → Cityscapes	deeplab-r101	52.3 DTST	59.2	`training_logs/seco_synthia`
GTA → BDD100K	deeplab-r101	37.9 SFOCDA	44.3	`training_logs/seco_bdd`

This table shows the source-free setting, which is more competitive than the report results in Tables 1 and 2 in the paper. More adaptation scripts will be updated later.

💬 License

Code is released for non-commercial and research purposes only. For commercial purposes, please contact the authors.

Acknowledgement

Many thanks to those wonderful work and the open-source code.

💬 Citation

If you use this code for your research, please cite our paper:

@misc{zhao2023semantic,
      title={Semantic Connectivity-Driven Pseudo-labeling for Cross-domain Segmentation}, 
      author={Dong Zhao and Ruizhi Yang and Shuang Wang and Qi Zang and Yang Hu and Licheng Jiao and Nicu Sebe and Zhun Zhong},
      year={2023},
      eprint={2312.06331},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

DZhaoXd/SeCo