A curated publication list on open vocabulary semantic segmentation.
If you find this project helpful, please consider giving it a star ⭐.
The model is trained on fully-supervised semantic segmentation datasets with pixel-level annotations (e.g., COCO Stuff dataset).
- [LSeg] | ICLR'22 | Language-driven Semantic Segmentation |
[pdf]
|[code]
- [ZegFormer] | CVPR'22 | ZegFormer: Decoupling Zero-Shot Semantic Segmentation |
[pdf]
|[code]
- [OpenSeg] | ECCV'22 | Scaling Open-vocabulary Image Segmentation with Image-level Labels |
[pdf]
|[code]
- [Xu et al.] | ECCV'22 | A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model |
[pdf]
|[code]
- [SegCLIP] | ICML'23 | SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentation |
[pdf]
|[code]
- [OVSeg] | CVPR'23 | Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP |
[pdf]
|[code]
- [X-Decoder] | CVPR'23 | Generalized Decoding for Pixel, Image, and Language |
[pdf]
|[code]
- [SAN] | CVPR'23(Highlight) | Side Adapter Network for Open-Vocabulary Semantic Segmentation |
[pdf]
|[code]
- [SAN] | TAPMI'23 | SAN: Side Adapter Network for Open-vocabulary Semantic Segmentation |
[pdf]
|[code]
- [ODISE] | CVPR'23 | Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models |
[pdf]
|[code]
- [FreeSeg] | CVPR'23 | FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentation |
[pdf]
|[code]
- [CAT-Seg] | Arxiv'23 | CAT-Seg : Cost Aggregation for Open-Vocabulary Semantic Segmentation |
[pdf]
|[code]
- [ADA] | Arxiv'23 | Open-Vocabulary Semantic Segmentation via Attribute Decomposition-Aggregation |
[pdf]
- [OpenSeeD] | ICCV'23 | A Simple Framework for Open-Vocabulary Segmentation and Detection |
[pdf]
|[code]
- [GKC] | ICCV'23 | Global Knowledge Calibration for Fast Open-Vocabulary Segmentation |
[pdf]
- [OPSNet] | ICCV'23 | Open-vocabulary Panoptic Segmentation with Embedding Modulation |
[pdf]
|[code]
- [MasQCLIP] | ICCV'23 | MasQCLIP for Open-Vocabulary Universal Image Segmentation |
[pdf]
- [DeOP] | ICCV'23 | Open Vocabulary Semantic Segmentation with Decoupled One-Pass Network |
[pdf]
|[code]
- [HIPIE] | NeurIPS'23 | Hierarchical Open-vocabulary Universal Image Segmentation |
[pdf]
|[code]
- [FC-CLIP] | NeurIPS'23 | Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP |
[pdf]
|[code]
The model is trained on weakly supervised datasets with only image-level annotations/captions (e.g., CC12M dataset).
- [GroupViT] | CVPR'22 | GroupViT: Semantic Segmentation Emerges from Text Supervision |
[pdf]
|[code]
- [ViL-Seg] | ECCV'22 | Open-world Semantic Segmentation via Contrasting and Clustering Vision-Language Embedding |
[pdf]
- [MaskCLIP+] | ECCV'22(Oral) | Extract Free Dense Labels from CLIP |
[pdf]
|[code]
- [ViewCo] | ICLR'23 | Viewco: Discovering Text-supervised Segmentation Masks via Multi-view semantic Consistency |
[pdf]
- [SegCLIP] | ICML'23 | SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentation |
[pdf]
|[code]
- [CLIP-S4] | CVPR'23 | CLIP-S4: Language-Guided Self-Supervised Semantic Segmentation |
[pdf]
- [PACL] | CVPR'23 | Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning |
[pdf]
- [OVSegmentor] | CVPR'23 | Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervision |
[pdf]
|[code]
- [SimSeg] | CVPR'23 | A Simple Framework for Text-Supervised Semantic Segmentation |
[pdf]
|[code]
- [TCL] | CVPR'23 | Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairs |
[pdf]
|[code]
- [ZeroSeg] | CVPR'23 | Exploring Open-Vocabulary Semantic Segmentation from CLIP Vision Encoder Distillation Only |
[pdf]
- [CLIPpy] | ICCV'23 | Perceptual Grouping in Contrastive Vision-Language Models |
[pdf]
|[code]
- [MixReorg] | ICCV'23 | MixReorg: Cross-Modal Mixed Patch Reorganization is a Good Mask Learner for Open-World Semantic Segmentation |
[pdf]
- [Zhang et al.] | Arxiv'23 | Associating Spatially-Consistent Grouping with Text-supervised Semantic Segmentation |
[pdf]
- [SimCon] | Arxiv'23 | SimCon Loss with Multiple Views for Text Supervised Semantic Segmentation |
[pdf]
- [CoCu] | NeurIPS'23 | Bridging Semantic Gaps for Language-Supervised Semantic Segmentation |
[pdf]
|[code]
The model is modified from the off-the-shelf large models (e.g., CLIP, Diffusion models) without an additional training phase.
- [MaskCLIP] | ECCV'22(Oral) | Extract Free Dense Labels from CLIP |
[pdf]
|[code]
- [CLIP Surgery] | Arxiv'23 | CLIP Surgery for Better Explainability with Enhancement in Open-Vocabulary Tasks |
[pdf]
|[code]
- [OVDiff] | Arxiv'23 | Diffusion Models for Zero-Shot Open-Vocabulary Segmentation |
[pdf]
- [CLIP-DIY] | Arxiv'23 | CLIP-DIY: CLIP Dense Inference Yields Open-Vocabulary Semantic Segmentation For-Free |
[pdf]
- [DiffSegmenter] | Arxiv'23 | Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmenter |
[pdf]
- [RO-ViT] | CVPR'23(Highlight) | Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformers |
[pdf]
|[code]
- [CAT] | CVPR'23 | CAT: LoCalization and IdentificAtion Cascade Detection Transformer for Open-World Object Detection |
[pdf]
|[code]
- [DetCLIPv2] | CVPR'23 | DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-training via Word-Region Alignment |
[pdf]
- [CondHead] | CVPR'23 | Learning to Detect and Segment for Open Vocabulary Object Detection |
[pdf]
- [CORA] | CVPR'23 | CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matching |
[pdf]
|[code]
- [ovdet] | CVPR'23 | Aligning Bag of Regions for Open-Vocabulary Object Detection |
[pdf]
|[code]
- [OADP] | CVPR'23 | Object-Aware Distillation Pyramid for Open-Vocabulary Object Detection |
[pdf]
|[code]
- [F-VLM] | ICLR'23 | F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models |
[pdf]
|[code]
- [MMC-Det] | Arxiv'23 | Exploring Multi-Modal Contextual Knowledge for Open-Vocabulary Object Detection |
[pdf]
- [IPL] | Arxiv'23 | Improving Pseudo Labels for Open-Vocabulary Object Detection |
[pdf]
- [mm-ovod] | ICML 2023 | Multi-Modal Classifiers for Open-Vocabulary Object Detection |
[pdf]
|[code]
- [EdaDet] | ICCV'23 | EdaDet: Open-Vocabulary Object Detection Using Early Dense Alignment |
[pdf]
|[code]
- [SGDN] | Arxiv'23 | Open-Vocabulary Object Detection via Scene Graph Discovery |
[pdf]
- Towards Open Vocabulary Learning: A Survey |
[pdf]
- A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Future |
[pdf]
If you have any suggestions or find missing papers, please don't hesitate to contact me.