- 06/2021: check out our domain generalization paper FSDR: Frequency Space Domain Randomization for Domain Generalization (accepted to CVPR 2021). We design a domain adaptive panoptic segmentation network that exploits inter-style consistency and inter-task regularization for optimal domain adaptation in panoptic segmentation. Pytorch code and pre-trained models are coming soon.
Cross-View Regularization for Domain Adaptive Panoptic Segmentation
Jiaxing Huang, Dayan Guan, Xiao Aoran, Shijian Lu
School of Computer Science Engineering, Nanyang Technological University, Singapore
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021 (Oral)
If you find this code useful for your research, please cite our paper:
@InProceedings{Huang_2021_CVPR,
author = {Huang, Jiaxing and Guan, Dayan and Xiao, Aoran and Lu, Shijian},
title = {Cross-View Regularization for Domain Adaptive Panoptic Segmentation},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2021},
pages = {10133-10144}
}
Panoptic segmentation unifies semantic segmentation and instance segmentation which has been attracting increasing attention in recent years. However, most existing research was conducted under a supervised learning setup whereas unsupervised domain adaptive panoptic segmentation which is critical in different tasks and applications is largely neglected. We design a domain adaptive panoptic segmentation network that exploits inter-style consistency and inter-task regularization for optimal domain adaptive panoptic segmentation. The inter-style consistency leverages semantic invariance across the same image of the different styles which fabricates certain self-supervisions to guide the network to learn domain-invariant features. The inter-task regularization exploits the complementary nature of instance segmentation and semantic segmentation and uses it as a constraint for better feature alignment across domains. Extensive experiments over multiple domain adaptive panoptic segmentation tasks (e.g. synthetic-to-real and real-to-real) show that our proposed network achieves superior segmentation performance as compared with the state-of-the-art.
- Python 3.7
- Pytorch >= 0.4.1
- CUDA 9.0 or higher
- Clone the repo:
$ git clone https://github.com/jxhuang0508/CVRN.git
$ cd CVRN
- Creat conda environment:
$ conda env create -f environment.yaml
- Clone UPSNet:
$ git clone https://github.com/uber-research/UPSNet.git
- Initialization:
$ cd UPSNet
$ sh init.sh
$ cp -r lib/dataset_devkit/panopticapi/panopticapi/ .
$ git clone https://github.com/yzou2/CRST.git
$ cd UPSNet
$ sh init_cityscapes.sh
$ cd ..
$ python cvrn/init_citiscapes_19cls_to_16cls.py
$ cp cvrn/models/* UPSNet/upsnet/models
$ cp cvrn/dataset/* UPSNet/upsnet/dataset
$ cp cvrn/upsnet/* UPSNet/upsnet
Pre-trained models can be downloaded here and put in CVRN/pretrained_models
$ cd UPSNet
$ python upsnet/test_cvrn_upsnet.py --cfg ../config/cvrn_upsnet.yaml --weight_path ../pretrained_models/cvrn_upsnet.pth
$ 2021-06-10 14:20:09,688 | base_dataset.py | line 499: | PQ SQ RQ N
$ 2021-06-10 14:20:09,688 | base_dataset.py | line 500: --------------------------------------
$ 2021-06-10 14:20:09,688 | base_dataset.py | line 505: All | 34.0 68.2 43.4 16
$ 2021-06-10 14:20:09,688 | base_dataset.py | line 505: Things | 27.9 73.6 37.3 6
$ 2021-06-10 14:20:09,688 | base_dataset.py | line 505: Stuff | 37.7 65.0 47.1 10
$ python upsnet/test_cvrn_pfpn.py --cfg ../config/cvrn_pfpn.yaml --weight_path ../pretrained_models/cvrn_pfpn.pth
$ 2021-06-10 14:27:36,841 | base_dataset.py | line 361: | PQ SQ RQ N
$ 2021-06-10 14:27:36,842 | base_dataset.py | line 362: --------------------------------------
$ 2021-06-10 14:27:36,842 | base_dataset.py | line 367: All | 31.4 66.4 40.0 16
$ 2021-06-10 14:27:36,842 | base_dataset.py | line 367: Things | 20.7 68.1 28.2 6
$ 2021-06-10 14:27:36,842 | base_dataset.py | line 367: Stuff | 37.9 65.4 47.0 10
$ python upsnet/test_cvrn_psn.py --cfg ../config/cvrn_psn.yaml --weight_path ../pretrained_models/cvrn_psn_maskrcnn_branch.pth
$ 2021-06-10 23:18:22,662 | test_cvrn_psn.py | line 240: combined pano result:
$ 2021-06-10 23:20:32,259 | base_dataset.py | line 361: | PQ SQ RQ N
$ 2021-06-10 23:20:32,261 | base_dataset.py | line 362: --------------------------------------
$ 2021-06-10 23:20:32,261 | base_dataset.py | line 367: All | 32.1 66.6 41.1 16
$ 2021-06-10 23:20:32,261 | base_dataset.py | line 367: Things | 21.6 68.7 30.2 6
$ 2021-06-10 23:20:32,261 | base_dataset.py | line 367: Stuff | 38.4 65.3 47.6 10
This codebase is heavily borrowed from UPSNet and CRST.
If you have any questions, please contact: jiaxing.huang@ntu.edu.sg