/VBLC

[AAAI 2023 Oral] VBLC: Visibility Boosting and Logit-Constraint Learning for Domain Adaptive Semantic Segmentation under Adverse Conditions

Primary LanguagePythonOtherNOASSERTION

VBLC for Domain Adaptive Semantic Segmentation under Adverse Conditions

VBLC: Visibility Boosting and Logit-Constraint Learning for Domain Adaptive Semantic Segmentation under Adverse Conditions. AAAI 2023 Oral.

by Mingjia Li, Binhui Xie, Shuang Li, Chi Harold Liu and Xinjing Cheng

Project Page   arXiv   Bilibili   YouTube   Dropbox  

Update on 2023/01/31: Project page available.

Update on 2023/01/16: We are happy to announce that VBLC was accepted for Oral Presentation at AAAI 2023. 🎉🎉🎉

Update on 2023/01/12: Video and poster available.

Overview

We propose VBLC to tackle the problem of domain adaptive semantic segmentation under adverse conditions. VBLC explores the potential of getting rid of reference images and resolving the mixture of adverse conditions simultaneously.

The visibility boost module is introduced in the input space to narrow the gap brought about by poor visibility, while the logit constraint learning loss is devised to relieve the pain of overconfidence common in self-training schemes.

Extensive analysis and visualizations can be found in our paper.

↥

Installation

This code is implemented with Python 3.8.5 and PyTorch 1.10.2 on CUDA 11.3.

To try out this project, it is recommended to set up a virtual environment first:

# create and activate the environment
conda create --name vblc -y python=3.8.5
conda activate vblc

# install the right pip and dependencies for the fresh python
conda install -y ipython pip

Then, the dependencies can be installed by:

# install required packages
pip install -r requirements.txt

# install mmcv-full, this command compiles mmcv locally and may take some time
pip install mmcv-full==1.5.0  # requires other packeges to be installed first

Alternatively, the mmcv-full package can be installed faster with official pre-built packages, for instance:

# another way to install mmcv-full, faster
pip install mmcv-full==1.5.0 -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.10.0/index.html

The environment is now fully prepared.

↥

Datasets Preparation

Download Datasets

  • Cityscapes: Download leftImg8bit_trainvaltest.zip and gtFine_trainvaltest.zip from here and extract them to a custom directory.
  • ACDC: Download rgb_anon_trainvaltest.zip and gt_trainval.zip from here and extract them to a custom directory.
  • FoggyCityscapes: Download leftImg8bit_trainvaltest_foggy.zip from here and extract them to the same directory as Cityscapes dataset.
  • RainCityscapes: Download leftImg8bit_trainval_rain.zip from here and extract them to the same directory as Cityscapes dataset.

Setup Datasets

Symlink the required datasets:

ln -s /path/to/cityscapes/dataset data/cityscapes
ln -s /path/to/acdc/dataset data/acdc

Perform preprocessing to convert label IDs to the train IDs and gather dataset statistics:

python tools/convert_datasets/cityscapes.py data/cityscapes --nproc 8

Ultimately, the data structure should look like this:

VBLC
├── ...
├── data
│   ├── acdc
│   │   ├── gt
│   │   └── rgb_anon
│   ├── cityscapes
│   │   ├── gtFine
│   │   ├── leftImg8bit
│   │   ├── leftImg8bit_foggy
│   │   ├── leftImg8bit_rain
├── ...
↥

Model Zoo

We provide pretrained models of both Domain Adaptive Semantic Segmentation tasks through Google Drive and Baidu Netdisk (access code: VBLC).

Task 1: Cityscapes → ACDC

model name backbone mIoU (test) mIoU (val) checkpoint download
vblc_city2acdc_dlv2.pth ResNet-101 47.8 46.0 Google / Baidu (acc: VBLC)
vblc_city2acdc_daformer.pth MiT-B5 64.2 63.7 Google / Baidu (acc: VBLC)

Task 2: Cityscapes → FoggyCityscapes + RainCityscapes

model name backbone mIoU (val) checkpoint download
vblc_city2foggyrain_dlv2.pth ResNet-101 74.9 Google / Baidu (acc: VBLC)
vblc_city2foggyrain_daformer.pth MiT-B5 79.8 Google / Baidu (acc: VBLC)
↥

VBLC Evaluation

To evaluate the pretrained models on the validation set, please run as follows:

python -m tools.test /path/to/config /path/to/checkpoint --eval mIoU --val
Example

For example, if you download vblc_city2acdc_daformer.pth along with its config json file vblc_city2acdc_daformer.json into folder ./checkpoints/vblc_city2acdc_daformer/, then the evaluation script should be like:

python -m tools.test ./checkpoints/vblc_city2acdc_daformer/vblc_city2acdc_daformer.json ./checkpoints/vblc_city2acdc_daformer/vblc_city2acdc_daformer.pth --eval mIoU --val

Additionally, for ACDC Dataset, you can also obtain the test results through:

python -m tools.test /path/to/config /path/to/checkpoint --format-only --eval-options imgfile_prefix=/path/prefix/labelTrainIds do_palette=False
Example

For the aforementioned example, the corresponding script is:

python -m tools.test ./checkpoints/vblc_city2acdc_daformer/vblc_city2acdc_daformer.json ./checkpoints/vblc_city2acdc_daformer/vblc_city2acdc_daformer.pth --format-only --eval-options imgfile_prefix=./test_results/labelTrainIds do_palette=False

The test results should be under folder /path/prefix/labelTrainIds, and can then be zipped and submitted to ACDC test server for the test score.

↥

VBLC Training

To begin with, download SegFormer's official MiT-B5 weights (i.e., mit_b5.pth) pretrained on ImageNet-1k from here and put it into a new folder ./pretrained.

The training entrance is at run_experiments.py. To examine the setting for a specific task, please take a look at experiments.py for more details. Generally, the training script is given as:

python run_experiments.py --exp <exp_id>

The mapping between <exp_id> and tasks:

<exp_id> task backbone
0 Test the setup of the environment All
1 Cityscapes → ACDC MiT-B5
2 Cityscapes → ACDC ResNet-101
3 Cityscapes → FoggyCityscapes + RainCityscapes MiT-B5
4 Cityscapes → FoggyCityscapes + RainCityscapes ResNet-101

After training, the models can be validated and tested following VBLC Evaluation. Note that the training results are located in ./work_dirs. The config filename should look like: 220824_2303_city2acdc_vblc_rcs001_daformer_sepaspp_logit_constraint_mitb5_poly10warm_s0_4b56d.json, and the model file has suffix .pth.

↥

How to Add VBLC to your Self-Training UDA Code

  • To include the visibility boost module, take a look at night_fog_filter method in mmseg/models/utils/night_fog_filter.py.
  • To include the logit constraint learning loss, take a look at mmseg/models/losses/logit_constraint_loss.py for implementation details.
↥

Acknowledgments

This project is based on the following open-source projects. We thank their authors for making the source code publicly available.

⭐ Also see our SePiCo ⭐

↥

Citation

If you find this work helpful to your research, please consider citing the paper:

@inproceedings{li2023vblc,
  title={VBLC: Visibility Boosting and Logit-Constraint Learning for Domain Adaptive Semantic Segmentation under Adverse Conditions},
  author={Li, Mingjia and Xie, Binhui and Li, Shuang and Liu, Chi Harold and Cheng, Xinjing},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  pages={},
  year={2023}
}
↥