Recent Advances in Video Object Segmentation (VOS). VOS works before 2022 can be found in our survey paper:
Deep Learning for Video Object Segmentation: A Review / paper / project page
BibTex
@article{gao2023deep, title={Deep learning for video object segmentation: a review}, author={Gao, Mingqi and Zheng, Feng and Yu, James JQ and Shan, Caifeng and Ding, Guiguang and Han, Jungong}, journal={Artificial Intelligence Review}, volume={56}, number={1}, pages={457--531}, year={2023}, publisher={Springer} }
🧸 We indicate different VOS tasks with coloured squares:
| Indicator | VOS Task |
|---|---|
🟦 SVOS |
Semi-Supervised VOS (also termed as One-Shot VOS) |
🟩 UVOS |
Un-Supervised VOS (also termed as Zero-Shot VOS) |
🟧 RVOS |
Referring VOS (also termed as Language-Guided VOS) |
🟥 AVOS |
Audio-guided VOS (also termed as Audio-Visual Video Segmentation) |
💠 VMAT |
Video Matting (VOS with detailed edge handling) |
⬜ XVOS |
Other types of VOS |
🧸 Please feel free to send us pull requests to add VOS works.
Links for a quick jump: ArXiv 2024, 🔥ECCV 2024🔥, CVPR 2024, AAAI 2024, EMNLP 2023, NeurIPS 2023, ACMMM 2023, ICCV 2023, CVPR 2023, IJCAI 2023, AAAI 2023, Journals 2023, Earlier ArXiv 2023, NeurIPS 2022, ECCV 2022, CVPR 2022, AAAI 2022, Journals 2022
🟦 SVOS 🟧 RVOS - LSVOS @ECCV 2024 (Finished)
🟦 SVOS - VOTS @ECCV 2024 (Finished)
🟦 SVOS 🟧 RVOS - PVUW @CVPR 2024 (Finished)
🟥 AVOS 🟧 RVOS Aug - paper / code - Unleashing the Temporal-Spatial Reasoning Capacity of GPT for Training-Free Audio and Language Referenced Video Object Segmentation
🟥 AVOS Aug - paper / code - AVESFormer: Efficient Transformer Design for Real-Time Audio-Visual Segmentation
🟦 SVOS Jul - paper / code - SAM 2: Segment Anything in Images and Videos
🟧 RVOS Jul - paper / code - ViLLa: Video Reasoning Segmentation with Large Language Model
🟩 UVOS Jul - paper / code - Improving Unsupervised Video Object Segmentation via Fake Flow Generation
🟦 SVOS Jul - paper / code - Learning Spatial-Semantic Features for Robust Video Object Segmentation
🟥 AVOS Jun - paper / code - Progressive Confident Masking Attention Network for Audio-Visual Segmentation
🟥 AVOS Jun - paper / code - Extending Segment Anything Model into Auditory and Temporal Dimensions for Audio-Visual Segmentation
🟧 RVOS Jun - paper / code - GroPrompt: Efficient Grounded Prompting and Adaptation for Referring Video Object Segmentation
🟦 SVOS May - paper / code and dataset - LVOS (v2, with more data): A Benchmark for Large-scale Long-term Video Object Segmentation
🟧 RVOS May - paper / code - Driving Referring Video Object Segmentation with Vision-Language Pre-trained Models
🟦 SVOS May - paper / code - One-shot Training for Video Object Segmentation
🟦 SVOS May - paper / code - Global Motion Understanding in Large-Scale Video Object Segmentation
🟦 SVOS May - paper / code - DeVOS: Flow-Guided Deformable Transformer for Video Object Segmentation
⬜ XVOS Apr - paper / code - Dynamic in Static: Hybrid Visual Correspondence for Self-Supervised Video Object Segmentation
🟩 UVOS Apr - paper / code - Moving Object Segmentation: All You Need Is SAM (and Flow)
🟧 RVOS Mar - paper / code - Towards Temporally Consistent Referring Video Object Segmentation
🟥 AVOS Mar - paper / code - Unsupervised Audio-Visual Segmentation with Modality Alignment
🟦 SVOS Mar - paper / code - Efficient Video Object Segmentation via Modulated Cross-Attention Memory
🟦 SVOS Mar - paper / code - OneVOS: Unifying Video Object Segmentation with All-in-One Transformer Framework
⬜ XVOS Mar - paper / code - ClickVOS: Click Video Object Segmentation
🟦 SVOS - paper / code - Spatial-Temporal Multi-level Association for Video Object Segmentation
🟧 RVOS - paper / code - VISA: Reasoning Video Object Segmentation via Large Language Model
🟧 RVOS - paper / code - Exploring Pre-trained Text-to-Video Diffusion Models for Referring Video Object Segmentation
🟧 RVOS - paper / code - ActionVOS: Actions as Prompts for Video Object Segmentation
🟧 RVOS 🟥 AVOS - paper / code & dataset - R2-Bench: Benchmarking the Robustness of Referring Perception Models under Perturbations
🟧 RVOS 🟥 AVOS - paper / code - Ref-AVS: Refer and Segment Objects in Audio-Visual Scenes
🟥 AVOS - paper / code - Stepping Stones: A Progressive Training Strategy for Audio-Visual Semantic Segmentation
💠 VMAT - paper / code - MaGGIe: Masked Guided Gradual Human Instance Matting
⬜ XVOS - paper / code - Point-VOS: Pointing Up Video Object Segmentation
🟥 AVOS - paper / code - Towards Robust Audiovisual Segmentation in Complex Environments with Quantization-based Semantic Decomposition
🟥 AVOS - paper / code - Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentation
🟥 AVOS - paper / code - A Closer Look at Audio-Visual Segmentation
🟩 UVOS - paper / code - Depth-aware Test-Time Training for Zero-shot Video Object Segmentation
🟩 UVOS - paper / code - Dual Prototype Attention for Unsupervised Video Object Segmentation
🟩 UVOS - paper / code - Guided Slot Attention for Unsupervised Video Object Segmentation
🟧 RVOS Apr - paper / code - Decoupling Static and Hierarchical Motion Perception for Referring Video Segmentation
🟧 RVOS - paper / code - LoSh: Long-Short Text Joint Prediction Network for Referring Video Object Segmentation
🟧 RVOS Dec - paper / code - Universal Segmentation at Arbitrary Granularity with Language Instruction
🟦 SVOS 🟧 RVOS - paper / code - UniVS: Unified and Universal Video Segmentation with Prompts as Queries
🟦 SVOS 🟩 UVOS - paper / code - Matching Anything By Segmenting Anything
🟦 SVOS - paper / code - RMem: Restricted Memory Banks Improve Video Object Segmentation
🟦 SVOS - paper / code - Event-assisted Low-Light Video Object Segmentation
🟦 SVOS - paper / code - Putting the Object Back into Video Object Segmentation
🟧 RVOS 🟥 AVOS - paper / code - Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentation
🟩 UVOS - paper / code - Generalizable Fourier Augmentation for Unsupervised Video Object Segmentation
🟧 RVOS - paper / code - Towards Noise-Tolerant Speech-Referring Video Object Segmentation: Bridging Speech and Text (Spoken language as referring guidance)
🟧 RVOS - paper / code - SOC: Semantic-Assisted Object Cluster for Referring Video Object Segmentation
🟦 SVOS - paper / code - Test-time Training for Matching-based Video Object Segmentation
🟦 SVOS - paper / code - From ViT Features to Training-free Video Object Segmentation via Streaming-data Mixture Models
🟩 UVOS - paper / code - SimulFlow: Simultaneously Extracting Feature and Identifying Target for Unsupervised Video Object Segmentation
🟩 UVOS - paper / code - Temporally Efficient Gabor Transformer for Unsupervised Video Object Segmentation
🟦 SVOS - paper / code - Exploring the Adversarial Robustness of Video Object Segmentation via One-shot Adversarial Attacks
🟥 AVOS - paper / code - CATR: Combinatorial-Dependence Audio-Queried Transformer for Audio-Visual Video Segmentation
🟥 AVOS - paper / code - Audio-Visual Segmentation by Exploring Cross-Modal Mutual Semantics
⬜ XVOS - paper / code - Multi-grained Temporal Prototype Learning for Few-shot Video Object Segmentation
🟩 UVOS - paper / code - Time Does Tell: Self-Supervised Time-Tuning of Dense Image Representations (self-supervised learning for UVOS)
🟩 UVOS - paper / code - Isomer: Isomerous Transformer for Zero-Shot Video Object Segmentation
🟩 UVOS - paper / code - Unsupervised Video Object Segmentation with Online Adversarial Self-Tuning
🟩 UVOS 🟧 RVOS - paper / code - DEVA: Tracking Anything with Decoupled Video Segmentation
🟧 RVOS - paper / code - Temporal Collection and Distribution for Referring Video Object Segmentation
🟧 RVOS - paper / code - Robust Referring Video Object Segmentation with Cyclic Structural Consensus
🟧 RVOS - paper / code - Spectrum-guided Multi-granularity Referring Video Object Segmentation
🟧 RVOS - paper / code - OnlineRefer: A Simple Online Baseline for Referring Video Object Segmentation
🟧 RVOS - paper / code - Learning Cross-Modal Affinity for Referring Video Object Segmentation Targeting Limited Samples
🟧 RVOS - paper / code - HTML: Hybrid Temporal-scale Multimodal Learning Framework for Referring Video Object Segmentation
🟧 RVOS - paper / code & dataset - MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressions
🟦 SVOS - paper / code - Integrating Boxes and Masks: A Multi-Object Framework for Unified Visual Tracking and Segmentation
🟦 SVOS - paper / code - XMem++: Production-level Video Segmentation From Few Annotated Frames
🟦 SVOS - paper / code - Scalable Video Object Segmentation with Simplified Framework
🟦 SVOS - paper / code - Alignment Before Aggregation: Trajectory Memory Retrieval Network for Video Object Segmentation
🟦 SVOS - paper / code - SegGPT: Segmenting Everything In Context
🟦 SVOS - paper / code & dataset - LVOS: A Benchmark for Long-term Video Object Segmentation
🟦 SVOS - paper / code & dataset - MOSE: A New Dataset for Video Object Segmentation in Complex Scenes
💠 VMAT - paper / code - Adaptive Human Matting for Dynamic Videos
🟩 UVOS - paper / code - MED-VT: Multiscale Encoder-Decoder Video Transformer with Application to Object Segmentation
🟦 SVOS - paper / code - Boosting Video Object Segmentation via Space-time Correspondence Learning
🟦 SVOS 🟧 RVOS - paper / code - Universal Instance Perception as Object Discovery and Retrieval
🟦 SVOS - paper / code - TarViS: A Unified Approach for Target-Based Video Segmentation
🟦 SVOS - paper / code - Two-shot Video Object Segmetnation
🟦 SVOS - paper / code - MobileVOS: Real-Time Video Object Segmentation Contrastive Learning meets Knowledge Distillation
🟦 SVOS - paper / code - Look Before You Match: Instance Understanding Matters in Video Object Segmentation
⬜ XVOS - paper / code & dataset - Breaking the “Object” in Video Object Segmentation
🟥 AVOS - paper / code - Discovering Sounding Objects by Audio Queries for Audio Visual Segmentation
🟦 SVOS - paper / code & dataset - Video Object Segmentation in Panoptic Wild Scenes
🟦 SVOS - paper / code - Learning to Learn Better for Video Object Segmentation
🟩 UVOS - paper / code - TIP Hierarchical Graph Pattern Understanding for Zero-Shot Video Object Segmentation
🟩 UVOS - paper / code - TCSVT Online Unsupervised Video Object Segmentation via Contrastive Motion Clustering
🟦 SVOS - paper / code - TIP Hierarchical Co-Attention Propagation Network for Zero-Shot Video Object Segmentation
🟧 RVOS - paper / code - TPAMI VLT: Vision-Language Transformer and Query Generation for Referring Segmentation
🟧 RVOS - paper / code - TPAMI Local-Global Context Aware Transformer for Language-Guided Video Segmentation
🟥 AVOS - paper / code - Bootstrapping Audio-Visual Segmentation by Strengthening Audio Cues
⬜ XVOS - paper / code - Vivim: a Video Vision Mamba for Medical Video Object Segmentation
⬜ XVOS - paper / code - Self-supervised Video Object Segmentation with Distillation Learning of Deformable Attention
⬜ XVOS - paper / code - Explore Synergistic Interaction Across Frames for Interactive Video Object Segmentation
🟧 RVOS - paper / code - Tracking with Human-Intent Reasoning
🟦 SVOS 🟧 RVOS - paper / code - UniRef++: Segment Every Reference Object in Spatial and Temporal Spaces
🟩 UVOS Dec - paper / code - Appearance-based Refinement for Object-Centric Motion Segmentation
🟦 SVOS 🟧 RVOS - paper / code - General Object Foundation Model for Images and Videos at Scale
🟦 SVOS - paper / code - M3T: Multi-Scale Memory Matching for Video Object Segmentation and Tracking
🟥 AVOS - paper / code - Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentation
🟧 RVOS - paper / code - VIDiff: Translating Videos via Multi-Modal Instructions with Diffusion Models
🟩 UVOS - paper / code - Betrayed by Attention: A Simple yet Effective Approach for Self-supervised Video Object Segmentation
⬜ XVOS - paper / code - Sketch-based Video Object Segmentation: Benchmark and Analysis
⬜ XVOS - paper / code - Learning the What and How of Annotation in Video Object Segmentation
🟦 SVOS - paper / code - Sub-token ViT Embedding via Stochastic Resonance Transformers (support svos)
🟦 SVOS - paper / code & dataset - PanoVOS: Bridging Non-panoramic and Panoramic Views with Transformer for Video Segmentation
🟩 UVOS - paper / code - Treating Motion as Option with Output Selection for Unsupervised Video Object Segmentation
🟥 AVOS - paper / code - Rethinking Audiovisual Segmentation with Semantic Quantization and Decomposition
🟦 SVOS - paper / code - Joint Modeling of Feature, Correspondence, and a Compressed Memory for Video Object Segmentation
🟧 RVOS 🟥 AVOS - paper / code - EPCFormer: Expression Prompt Collaboration Transformer for Universal Referring Video Object Segmentation
🟧 RVOS - paper / code - Learning Referring Video Object Segmentation from Weak Annotation
🟩 UVOS - paper / code - UVOSAM: A Mask-free Paradigm for Unsupervised Video Object Segmentation via Segment Anything Model
⬜ XVOS - paper / code - Segment and Track Anything
⬜ XVOS - paper / code - Track Anything: Segment Anything Meets Videos
⬜ XVOS - paper / code - Reliability-Hierarchical Memory Network for Scribble-Supervised Video Object Segmentation
🟦 SVOS - paper / code - Tracking Anything in High Quality
🟧 RVOS - paper / code - Referring Video Object Segmentation with Inter-Frame Interaction and Cross-Modal Correlation
🟧 RVOS - paper / code - RefSAM: Efficiently Adapting Segmenting Anything Model for Referring Video Object Segmentation
⬜ XVOS - paper / code - Segment Anything Meets Point Tracking
🟦 SVOS - paper / code - Decoupling Features in Hierarchical Propagation for Video Object Segmentation
⬜ XVOS - paper / code - Self-supervised Amodal Video Object Segmentation
🟦 SVOS - paper / code - XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model
🟦 SVOS - paper / code - BATMAN: Bilateral Attention Transformer in Motion-Appearance Neighboring Space for Video Object Segmentation
🟦 SVOS - paper / code - Learning Quality-aware Dynamic Memory for Video Object Segmentation
🟦 SVOS - paper / code - Tackling Background Distraction in Video Object Segmentation
🟦 SVOS - paper / code - Global Spectral Filter Memory Network for Video Object Segmentation
🟩 UVOS - paper / code - Hierarchical Feature Alignment Network for Unsupervised Video Object Segmentation
🟧 RVOS - paper / code - End-to-End Referring Video Object Segmentation With Multimodal Transformers
🟧 RVOS - paper / code - Language As Queries for Referring Video Object Segmentation
🟧 RVOS - paper / code - Language-Bridged Spatial-Temporal Interaction for Referring Video Object Segmentation
🟧 RVOS - paper / code - Multi-Level Representation Learning With Semantic Alignment for Referring Video Object Segmentation
🟦 SVOS - paper / code - Recurrent Dynamic Embedding for Video Object Segmentation
🟦 SVOS - paper / code - Accelerating Video Object Segmentation With Compressed Video
🟦 SVOS - paper / code - SWEM: Towards Real-Time Video Object Segmentation With Sequential Weighted Expectation-Maximization
🟦 SVOS - paper / code - Per-Clip Video Object Segmentation
⬜ XVOS - paper / code - Wnet: Audio-Guided Video Object Segmentation via Wavelet-Based Cross-Modal Denoising Networks
⬜ XVOS - paper / code & dataset - YouMVOS: An Actor-Centric Multi-Shot Video Object Segmentation Dataset
🟦 SVOS - paper / code - Siamese Network with Interactive Transformer for Video Object Segmentation
🟦 SVOS - paper / code - Reliable Propagation-Correction Modulation for Video Object Segmentation
🟧 RVOS - paper / code - You Only Infer Once: Cross-Modal Meta-Transfer for Referring Video Object Segmentation
🟩 UVOS - paper / code - Iteratively Selecting an Easy Reference Frame Makes Unsupervised Video Object Segmentation Easier
🟦 SVOS - paper / code - TPAMI Video Object Segmentation Using Kernelized Memory Network With Multiple Kernels
🟦 SVOS - paper / code - TIP From Pixels to Semantics: Self-Supervised Video Object Segmentation With Multiperspective Feature Mining
🟦 SVOS - paper / code - TIP Delving Deeper Into Mask Utilization in Video Object Segmentation
🟦 SVOS - paper / code - TIP Adaptive Online Mutual Learning Bi-Decoders for Video Object Segmentation
End of the list. 🌱
VOS papers and datasets before 2022 could be found below:
Deep Learning for Video Object Segmentation: A Review / paper / project page