/awesome-panoptic-segmentation

Panoptic Segmentation Resources List

Awesome-Panoptic-Segmentation Awesome

This repo is a collection of the challenging panoptic segmentation, including papers, codes, and benchmark results, etc.

Outline

Panoptic Segmentation

Summarize in one sentence : Panoptic Segmentation proposes to solve the semantic segmentation(*Stuff*) and instance segmentation(*Thing*) in a unified and general manner.

Structure Overview

from UPSNet.

Datasets

Generally, the datasets which contains both semantic and instance annotations can be used to solve the challenging panoptic task.

Evaluation

Metrics

  • PC are the standard metrics described in DeeperLab.

Evaluation Code

Benchmark Results

COCO val Benchmark

Method Backbone PQ PQ-Thing PQ-Stuff SQ RQ mIoU AP-Mask PC e2e
SOGNet ResNet-50 43.7 50.6 33.2 78.7 53.5 54.56 34.2 - ✅
UPSNet ResNet-50 42.5 48.6 33.4 - - 54.3 34.3 - ✅
OANet ResNet-101 41.3 50.4 27.7 - - - - - ✅
OCFusion ResNet-50 41.0 49.0 29.0 77.1 50.6 - - - ✅
Panoptic FPN ResNet-101 40.9 48.3 29.7 - - - - - ✅
AUNet ResNet-50 39.6 49.1 25.2 - - 45.1 34.7 - ✅
AdaptIS ResNet-101 37.0 41.8 29.9 - - - - - ✅
DeeperLab Xception-71 34.3 37.5 29.6 77.1 43.1 - - 56.8 ✅

Cityscapes valBenchmark

Method Backbone PQ PQ-Thing PQ-Stuff SQ RQ mIoU AP-Mask PC e2e
Panoptic(Merge) - 61.2 66.4 54.0 80.9 74.4 - - - ❎
AdaptIS ResNet-101 60.6 58.7 64.4 - - 79.2 36.3 - ✅
SOGNet ResNet-50 60.0 56.7 62.5 - - - - - ✅
Seamless ResNet-50 59.8 53.4 64.5 - - 75.4 31.9 - ✅
UPSNet ResNet-50 59.3 54.6 62.7 79.7 73.0 75.2 33.3 - ✅
TASCNet ResNet-101 59.2 56 61.5 - - 77.8 37.6 - ✅
AUNet ResNet-101 59.0 54.8 62.1 - - 75.6 34.4 - ✅
Panoptic FPN ResNet-101 58.1 52.0 62.5 - - 75.7 33.0 - ✅
DeeperLab Xception-71 56.5 - - - - - - 75.6 ✅

Mapillary val Benchmark

Method Backbone PQ PQ-Thing PQ-Stuff SQ RQ mIoU AP-Mask PC e2e
Panoptic(Merge) - 38.3 41.8 35.7 73.6 47.7 - - - ❎
Seamless ResNet-50 37.2 33.2 42.5 - - 50.2 16.3 - ✅
AdaptIS ResNet-101 33.4 28.3 40.3 - - - - - ✅
TASCNet ResNet-101 32.6 31.3 34.4 - - 35.0 18.5 - ✅
DeeperLab Xception-71 32.0 - - - - - - 55.3 ✅

Papers

CVPR2020

  • Hou, Rui, et al. "Real-Time Panoptic Segmentation from Dense Detections." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020. [paper]

  • Dundar, Aysegul, et al. "Panoptic-based Image Synthesis." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020. [paper]

  • Wang, Haochen, et al. "Pixel Consensus Voting for Panoptic Segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020. [paper]

  • Cheng, Bowen, et al. "Panoptic-deeplab: A simple, strong, and fast baseline for bottom-up panoptic segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020. [paper]

  • Lazarow, Justin, et al. "Learning instance occlusion for panoptic segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020. [[paper] (http://openaccess.thecvf.com/content_CVPR_2020/html/Lazarow_Learning_Instance_Occlusion_for_Panoptic_Segmentation_CVPR_2020_paper.html)]

  • Chen, Yifeng, et al. "BANet: Bidirectional Aggregation Network with Occlusion Handling for Panoptic Segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020. [paper]

AAAI2020

  • SOGNet: Yibo Yang, Hongyang Li, Xia Li, Qijie Zhao, Jianlong Wu, Zhouchen Lin.
    "SOGNet: Scene Overlap Graph Network for Panoptic Segmentation." AAAI (2020). [paper]

ICCV2019

  • AdaptIS: Konstantin Sofiiuk, Olga Barinova, Anton Konushin.
    "AdaptIS: Adaptive Instance Selection Network." ICCV (2019). [paper]

  • Cheng-Yang Fu, Tamara L. Berg, Alexander C. Berg.
    "IMP: Instance Mask Projection for High Accuracy Semantic Segmentation of Things." ICCV (2019). [paper]

  • Bowen Cheng, Maxwell D. Collins, Yukun Zhu, Ting Liu, Thomas S. Huang, Hartwig Adam, Liang-Chieh Chen.
    "Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation Bowen." ICCVW (2019). [paper]

CVPR2019

  • Panoptic Segmentation: Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, Piotr Dollár.
    "Panoptic Segmentation." CVPR (2019). [paper]

  • Panoptic FPN: Alexander Kirillov, Ross Girshick, Kaiming He, Piotr Dollár.
    "Panoptic Feature Pyramid Networks." CVPR (2019 oral). [paper] [unofficial code][detectron2]

  • AUNet: Yanwei Li, Xinze Chen, Zheng Zhu, Lingxi Xie, Guan Huang, Dalong Du, Xingang Wang.
    "Attention-guided Unified Network for Panoptic Segmentation." CVPR (2019). [paper]

  • UPSNet: Yuwen Xiong, Renjie Liao, Hengshuang Zhao, Rui Hu, Min Bai, Ersin Yumer, Raquel Urtasun.
    "UPSNet: A Unified Panoptic Segmentation Network." CVPR (2019 oral). [paper] [code]

  • DeeperLab: Tien-Ju Yang, Maxwell D. Collins, Yukun Zhu, Jyh-Jing Hwang, Ting Liu, Xiao Zhang, Vivienne Sze, George Papandreou, Liang-Chieh Chen.
    "DeeperLab: Single-Shot Image Parser." CVPR (2019 oral). [paper] [project] [code]

  • OANet: Huanyu Liu, Chao Peng, Changqian Yu, Jingbo Wang, Xu Liu, Gang Yu, Wei Jiang.
    "An End-to-End Network for Panoptic Segmentation." CVPR (2019). [paper]

  • Eirikur Agustsson, Jasper R. R. Uijlings, Vittorio Ferrari .
    "Interactive Full Image Segmentation by Considering All Regions Jointly." CVPR (2019). [paper]

  • Seamless: Lorenzo Porzi, Samuel Rota Bulo, Aleksander Colovic, Peter Kontschieder.
    "Seamless Scene Segmentation." CVPR (2019) (Extended Version). [paper][code]

ECCV2018

  • Qizhu Li, Anurag Arnab, Philip H.S. Torr.
    "Weakly- and Semi-Supervised Panoptic Segmentation." ECCV (2018). [paper] [code]

ArXiv

  • Behley, Jens, Andres Milioto, and Cyrill Stachniss. "A Benchmark for LiDAR-based Panoptic Segmentation based on KITTI." arXiv preprint arXiv:2003.02371 (2020). [paper]

  • Carion, Nicolas, et al. "End-to-End Object Detection with Transformers." arXiv preprint arXiv:2005.12872 (2020). [paper]

  • Meletis, Panagiotis, et al. "Cityscapes-Panoptic-Parts and PASCAL-Panoptic-Parts datasets for Scene Understanding." arXiv preprint arXiv:2004.07944 (2020). [paper]

  • Li, Qizhu, Xiaojuan Qi, and Philip HS Torr. "Unifying Training and Inference for Panoptic Segmentation." arXiv preprint arXiv:2001.04982 (2020). [paper]

  • Chang, Chia-Yuan, et al. "EPSNet: Efficient Panoptic Segmentation Network with Cross-layer Attention Fusion." arXiv preprint arXiv:2003.10142 (2020). [paper]

  • Wang, Huiyu, et al. "Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation." arXiv preprint arXiv:2003.07853 (2020). [paper]

  • Rui Hou, Jie Li, Arjun Bhargava, Allan Raventos, Vitor Guizilini, Chao Fang, Jerome Lynch, Adrien Gaidon.
    "Real-Time Panoptic Segmentation from Dense Detections." arXiv (2019). [paper]

  • Mark Weber, Jonathon Luiten, Bastian Leibe.
    "Single-Shot Panoptic Segmentation." arXiv (2019). [paper]

  • Qiang Chen, Anda Cheng, Xiangyu He, Peisong Wang, Jian Cheng.
    "SpatialFlow: Bridging All Tasks for Panoptic Segmentation." arXiv (2019). [paper]

  • Sagi Eppel, Alan Aspuru-Guzik.
    "Generator evaluator-selector net: a modular approach for panoptic segmentation." arXiv (2019). [paper]

  • Jasper R. R. Uijlings, Mykhaylo Andriluka, Vittorio Ferrari.
    "Panoptic Image Annotation with a Collaborative Assistant." arXiv (2019). [paper]

  • OCFusion: Justin Lazarow, Kwonjoon Lee, Zhuowen Tu.
    "Learning Instance Occlusion for Panoptic Segmentation." arXiv (2019). [paper]

  • PEN: Yuan Hu, Yingtian Zou, Jiashi Feng.
    "Panoptic Edge Detection." arXiv (2019). [paper]

  • TASCNet: Jie Li, Allan Raventos, Arjun Bhargava, Takaaki Tagawa, Adrien Gaidon.
    "Learning to Fuse Things and Stuff." arXiv (2018). [paper]

  • Daan de Geus, Panagiotis Meletis, Gijs Dubbelman.
    "Panoptic Segmentation with a Joint Semantic and Instance Segmentation Network." arXiv (2018). [paper]

  • Daan de Geus, Panagiotis Meletis, Gijs Dubbelman.
    "Single Network Panoptic Segmentation for Street Scene Understanding." arXiv (2019). [paper]

  • David Owen, Ping-Lin Chang.
    "Detecting Reflections by Combining Semantic and Instance Segmentation." arXiv (2019). [paper]

  • Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji.
    "PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things." arXiv (2019, IROS). [paper]

Tutorials

  • CVPR 2019 Tutorial on Visual Recognition and Beyond. [slides] [homepage]
  • COCO 2017 Workshop. [slides]

Blogs