/SemanticSegmentation_DL

Resources of semantic segmantation based on Deep Learning model

Primary LanguagePython

Semantic-Segmentation

A list of all papers and resoureces on Semantic Segmentation.

Dataset importance

Dataset importance plot

SemanticSegmentation_DL

Some implementation of semantic segmantation for DL model

Dataset

Resources

Survey papers

Online demos

2D Semantic Segmentation

Papers:

  • AAAI-2018 Mix-and-Match Tuning for Self-Supervised Semantic Segmentation [Paper]
  • NIPS-2017-Learning Affinity via Spatial Propagation Networks [Paper]
  • AAAI-2018-Spatial As Deep: Spatial CNN for Traffic Scene Understanding [Paper]
  • Stacked Deconvolutional Network for Semantic Segmentation-2017 [Paper]
  • Deeplab v3: Rethinking Atrous Convolution for Semantic Image Segmentation-2017(DeeplabV3) [Paper]
  • Learning Object Interactions and Descriptions for Semantic Image Segmentation-2017 [Paper]
  • Pixel Deconvolutional Networks-2017 [Code-Tensorflow] [Paper]
  • Dilated Residual Networks-2017 [Paper]
  • A Review on Deep Learning Techniques Applied to Semantic Segmentation-2017 [Paper]
  • BiSeg: Simultaneous Instance Segmentation and Semantic Segmentation with Fully Convolutional Networks [Paper]
  • ICNet for Real-Time Semantic Segmentation on High-Resolution Images-2017 [Project] [Code] [Paper] [Video]
  • Not All Pixels Are Equal: Difficulty-Aware Semantic Segmentation via Deep Layer Cascade-2017 [Paper]
  • Loss Max-Pooling for Semantic Image Segmentation-2017 [Paper]
  • Annotating Object Instances with a Polygon-RNN-2017 [Project] [Paper]

3D Semantic Segmentation

Papers

  • PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation [Paper]
  • PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space (2017) [Paper]
  • Learning 3D Mesh Segmentation and Labeling (2010) [Paper]
  • Unsupervised Co-Segmentation of a Set of Shapes via Descriptor-Space Spectral Clustering (2011) [Paper]
  • Single-View Reconstruction via Joint Analysis of Image and Shape Collections (2015) [Paper]
  • 3D Shape Segmentation with Projective Convolutional Networks (2017) [Paper]
  • Learning Hierarchical Shape Segmentation and Labeling from Online Repositories (2017) [Paper]
  • 3D Graph Neural Networks for RGBD Semantic Segmentation (2017) [Paper]
  • 3DCNN-DQN-RNN: A Deep Reinforcement Learning Framework for Semantic Parsing of Large-scale 3D Point Clouds (2017)[Paper]
  • Multi-view deep learning for consistent semantic mapping with rgb-d cameras [Paper]

Robotics

  • Virtual-to-Real: Learning to Control in Visual Semantic Segmentation [Paper]
  • End-to-End Tracking and Semantic Segmentation Using Recurrent Neural Networks [Paper]
  • Semantic Segmentation using Adversarial Networks [Paper]

Adversarial Training

  • CVPR-2017-Image-to-Image Translation with Conditional Adversarial Networks [Paper]
  • ICCV-2017-Adversarial Examples for Semantic Segmentation and Object Detection [Paper]

Scene Understanding

Papers

1.Spatial As Deep: Spatial CNN for Traffic Scene Understanding [Paper]

Dataset & Resources

  • SUNRGB-D 3D Object Detection Challenge [Link] 19 object categories for predicting a 3D bounding box in real world dimension Training set: 10,355 RGB-D scene images, Testing set: 2860 RGB-D images
  • SceneNN (2016) [Link] 100+ indoor scene meshes with per-vertex and per-pixel annotation.
  • ScanNet (2017) [Link] An RGB-D video dataset containing 2.5 million views in more than 1500 scans, annotated with 3D camera poses, surface reconstructions, and instance-level semantic segmentations.
  • Matterport3D: Learning from RGB-D Data in Indoor Environments (2017) [Link]
    10,800 panoramic views (in both RGB and depth) from 194,400 RGB-D images of 90 building-scale scenes of private rooms. Instance-level semantic segmentations are provided for region (living room, kitchen) and object (sofa, TV) categories.
  • SUNCG: A Large 3D Model Repository for Indoor Scenes (2017) [Link]
    The dataset contains over 45K different scenes with manually created realistic room and furniture layouts. All of the scenes are semantically annotated at the object level.
  • MINOS: Multimodal Indoor Simulator (2017) [Link] MINOS is a simulator designed to support the development of multisensory models for goal-directed navigation in complex indoor environments. MINOS leverages large datasets of complex 3D environments and supports flexible configuration of multimodal sensor suites. MINOS supports SUNCG and Matterport3D scenes.
  • Facebook House3D: A Rich and Realistic 3D Environment (2017) [Link]
    House3D is a virtual 3D environment which consists of 45K indoor scenes equipped with a diverse set of scene types, layouts and objects sourced from the SUNCG dataset. All 3D objects are fully annotated with category labels. Agents in the environment have access to observations of multiple modalities, including RGB images, depth, segmentation masks and top-down 2D map views.
  • HoME: a Household Multimodal Environment (2017) [Link]
    HoME integrates over 45,000 diverse 3D house layouts based on the SUNCG dataset, a scale which may facilitate learning, generalization, and transfer. HoME is an open-source, OpenAI Gym-compatible platform extensible to tasks in reinforcement learning, language grounding, sound-based navigation, robotics, multi-agent learning.
  • AI2-THOR: Photorealistic Interactive Environments for AI Agents [Link]
    AI2-THOR is a photo-realistic interactable framework for AI agents. There are a total 120 scenes in version 1.0 of the THOR environment covering four different room categories: kitchens, living rooms, bedrooms, and bathrooms. Each room has a number of actionable objects.

Medical Image Semantic Segmentation

Papers

Codes

Weakly-Supervised-Segmentatio

  • Weakly Supervised Structured Output Learning for Semantic Segmentation [Paper]
  • ICCV-2011 Weakly supervised semantic segmentation with a multi-image model [Paper]
  • ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016[Paper]
  • Constrained convolutional neural networks for weakly supervised segmentation. Proceedings of the IEEE International Conference on Computer Vision. 2015.[Paper]
  • Weakly-and semi-supervised learning of a DCNN for semantic image segmentation. arXiv preprint arXiv:1502.02734 (2015).[Paper]
  • Learning to segment under various forms of weak supervision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015.[Paper]
  • STC: A Simple to Complex Framework for Weakly-supervised Semantic Segmentation 2017 TPAMI [Paper] [Project]
  • [Paper]
  • CVPR-2017-Simple Does It: Weakly Supervised Instance and Semantic Segmentation [Paper]
  • CVPR-2017-Weakly Supervised Semantic Segmentation using Web-Crawled Videos [Paper]
  • AAAI-2017-Weakly Supervised Semantic Segmentation Using Superpixel Pooling Network [Paper]
  • ICCV-2015-Weakly supervised graph based semantic segmentation by learning communities of image-parts [Paper]
  • Towards Weakly Supervised Semantic Segmentation by Means of Multiple Instance and Multitask Learning [Paper]
  • Weakly-Supervised Semantic Segmentation using Motion Cues [Paper] [Project]
  • Weakly Supervised Semantic Segmentation Based on Web Image Co-segmentation [Paper]
  • Learning to Re ne Object Segments [Paper]

Video Semantic Segmentation

  • Feature Space Optimization for Semantic Video Segmentation[Paper][Slides]
  • The Basics of Video Object Segmentation [Blog]
  • ICCV2017----SegFlow_Joint Learning for Video Object Segmentation and Optical Flow
  • OSVOS:One-Shot Video Object Segmentation
  • Surveillance Video Parsing with Single Frame Supervision
  • The 2017 DAVIS Challenge on Video Object Segmentation
  • Video Propagation Networks
  • OnAVOS: Online Adaptation of Convolutional Neural Networks for Video Object Segmentation. P. Voigtlaender, B. Leibe, BMVC 2017. [Project Page] [Precomputed results]
  • OSVOS: One-Shot Video Object Segmentation. S. Caelles*, K.K. Maninis*, J. Pont-Tuset, L. Leal-Taixé, D. Cremers, L. Van Gool, CVPR 2017. [Project Page] [Precomputed results]
  • MSK: Learning Video Object Segmentation from Static Images. F. Perazzi*, A. Khoreva*, R. Benenson, B. Schiele, A. Sorkine-Hornung, CVPR 2017. [Project Page] [Precomputed results]
  • SFL: SegFlow: Joint Learning for Video Object Segmentation and Optical Flow. J. Cheng, Y.-H. Tsai, S. Wang, M.-H. Yang, ICCV 2017. [Project Page] [Precomputed results]
  • CTN: Online Video Object Segmentation via Convolutional Trident Network. W.-D. Jang, C.-S. Kim, CVPR 2017. [Project Page] [Precomputed results]
  • VPN: Video Propagation Networks. V. Jampani, R. Gadde, P. V. Gehler, CVPR 2017. [Project Page] [Precomputed results]
  • PLM: Pixel-level Matching for Video Object Segmentation using Convolutional Neural Networks. J. Shin Yoon, F. Rameau, J. Kim, S. Lee, S. Shin, I. So Kweon, ICCV 2017. [Project Page] [Precomputed results]
  • OFL: Video Segmentation via Object Flow. Y.-H. Tsai, M.-H. Yang, M. Black, CVPR 2016. [Project Page] [Precomputed results]
  • BVS: Bilateral Space Video Segmentation. N. Marki, F. Perazzi, O. Wang, A. Sorkine-Hornung, CVPR 2016. [Project Page] [Precomputed results]
  • FCP: Fully Connected Object Proposals for Video Segmentation. F. Perazzi, O. Wang, M. Gross, A. Sorkine-Hornung, ICCV 2015. [Project Page] [Precomputed results]
  • JMP: JumpCut: Non-Successive Mask Transfer and Interpolation for Video Cutout. Q. Fan, F. Zhong, D. Lischinski, D. Cohen-Or, B. Chen, SIGGRAPH 2015. [Project Page] [Precomputed results]
  • HVS: Efficient hierarchical graph-based video segmentation. M. Grundmann, V. Kwatra, M. Han, I. A. Essa, CVPR 2010. [Project Page] [Precomputed results]
  • SEA: SeamSeg: Video Object Segmentation Using Patch Seams. S. Avinash Ramakanth, R. Venkatesh Babu, CVPR 2014. [Project Page] [Precomputed results]
  • ARP: Primary Object Segmentation in Videos Based on Region Augmentation and Reduction. Y.J. Koh, C.-S. Kim, CVPR 2017. [Project Page] [Precomputed results]
  • LVO: Learning Video Object Segmentation with Visual Memory. P. Tokmakov, K. Alahari, C. Schmid, ICCV 2017. [Project Page] [Precomputed results]
  • FSEG: FusionSeg: Learning to combine motion and appearance for fully automatic segmentation of generic objects in videos. S. Jain, B. Xiong, K. Grauman, CVPR 2017. [Project Page] [Precomputed results]
  • LMP: Learning Motion Patterns in Videos. P. Tokmakov, K. Alahari, C. Schmid, CVPR 2017. [Project Page] [Precomputed results]
  • SFL: SegFlow: Joint Learning for Video Object Segmentation and Optical Flow. J. Cheng, Y.-H. Tsai, S. Wang, M.-H. Yang, ICCV 2017. [Project Page] [Precomputed results] FST: Fast Object Segmentation in Unconstrained Video. A. Papazoglou, V. Ferrari, ICCV 2013. [Project Page] [Precomputed results]
  • CUT: Motion Trajectory Segmentation via Minimum Cost Multicuts. M. Keuper, B. Andres, T. Brox, ICCV 2015. [Project Page] [Precomputed results]
  • NLC: Video Segmentation by Non-Local Consensus voting. A. Faktor, M. Irani, BMVC 2014. [Project Page] [Precomputed results]
  • MSG: Object segmentation in video: A hierarchical variational approach for turning point trajectories into dense regions. P. Ochs, T. Brox, ICCV 2011. [Project Page] [Precomputed results]
  • KEY: Key-segments for video object segmentation. Y. Lee, J. Kim, K. Grauman, ICCV 2011. [Project Page] [Precomputed results]
  • CVOS: Causal Video Object Segmentation from Persistence of Occlusions. B. Taylor, V. Karasev, S. Soatto, CVPR 2015. [Project Page] [Precomputed results]
  • TRC: Video segmentation by tracing discontinuities in a trajectory embedding. K. Fragkiadaki, G. Zhang, J. Shi, CVPR 2012. [Project Page] [Precomputed results]
  • Instance Embedding Transfer to Unsupervised Video Object Segmentation [Paper]
  • Result of DAVIS-Challenge 2017
  • Benchmark 2016----A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation
    2016----Clockwork Convnets for Video Semantic Segmentation
    2016----MaskTrack ----Learning Video Object Segmentation from Static Images
    2017----DAVIS-Challenge-1st----Video Object Segmentation with Re-identification
    2017----DAVIS-Challenge-2nd----Lucid Data Dreaming for Multiple Object Tracking
    2017----DAVIS-Challenge-3rd----Instance Re-Identification Flow for Video Object Segmentation
    2017----DAVIS-Challenge-4th----Multiple-Instance Video Segmentation with Sequence-Specific Object Proposals
    2017----DAVIS-Challenge-5th Online Adaptation of Convolutional Neural Networks for the 2017 DAVIS Challenge on Video Object Segmentation
    2017----DAVIS-Challenge-6th ----Learning to Segment Instances in Videos with Spatial Propagation Network
    2017----DAVIS-Challenge-7th----Some Promising Ideas about Multi-instance Video Segmentation
    2017----DAVIS-Challenge-8th----One-Shot Video Object Segmentation with Iterative Online Fine-Tuning
    2017----DAVIS-Challenge-9th----Video Object Segmentation using Tracked Object Proposals

Road Segmentation

Papers:

  • MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving [Paper]
  • self-driving-car-road-segmentation [Link]
  • Efficient Deep Models for Monocular Road Segmentation[Paper]
  • Semantic Road Segmentation via Multi-scale Ensembles of Learned Features [Paper]
  • Distantly Supervised Road Segmentation [Paper]
  • Deep Fully Convolutional Networks with Random Data Augmentation for Enhanced Generalization in Road Detection [Paper]
  • Real-time category-based and general obstacle detection for autonomous driving [Paper]
  • Road Scene Segmentation from a Single Image [Paper]
  • FoveaNet: Perspective-aware Urban Scene Parsing [Paper]

Codes

Transferable Semantic Segmentation

  • Weakly Supervised Object Localization Using Things and Stuff Transfer [Paper]
  • Semi and Weakly Supervised Semantic Segmentation Using Generative Adversarial Network [Paper]
  • Weakly- and Semi-Supervised Learning of a Deep Convolutional Network for Semantic Image Segmentation [Paper]

Real-Time Semantic Segmentation

  • LinkNet: Exploiting Encoder Representations for Efficient Semantic Segmentation [Paper]
  • ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation-2016 [Code-Caffe1][Code-Caffe2] [Paper] [Blog]
  • Efficient Deep Models for Monocular Road Segmentation[Paper]
  • Real-Time Coarse-to-fine Topologically Preserving Segmentation[Paper]
  • ICNet [Paper]
  • Efficient and robust deep networks for semantic segmentation [Paper]

Part Semantic Segmentation

  • Look into Person: Self-supervised Structure-sensitive Learning and A New Benchmark for Human Parsing-2017 [Project] [Code-Caffe] [Paper]
  • Deep Learning for Human Part Discovery in Images-2016 [Code-Chainer] [Paper]
    -A CNN Cascade for Landmark Guided Semantic Part Segmentation-2016 [Project] [Paper]
  • Deep Learning for Semantic Part Segmentation With High-level Guidance-2015 [Paper]
  • Neural Activation Constellations-Unsupervised Part Model Discovery with Convolutional Networks-2015 [Paper]
  • Human Parsing with Contextualized Convolutional Neural Network-2015 [Paper]
  • Part detector discovery in deep convolutional neural networks-2014 [Code] [Paper]
  • Hypercolumns for object segmentation and fine-grained localization [Paper]

Clothes Parsing

Instance Segmentation

  • Pixelwise Instance Segmentation with a Dynamically Instantiated Network-2017 [Paper]
  • Semantic Instance Segmentation via Deep Metric Learning-2017 [Paper]
  • Mask R-CNN-2017 [Code-Tensorflow] [Paper]
  • Pose2Instance: Harnessing Keypoints for Person Instance Segmentation-2017 [Paper]
  • Pixelwise Instance Segmentation with a Dynamically Instantiated Network-2017 [Paper]
  • Fully Convolutional Instance-aware Semantic Segmentation-2016 [Code] [Paper]
  • Instance-aware Semantic Segmentation via Multi-task Network Cascades-2015 [Code] [Paper]
  • Recurrent Instance Segmentation-2015 [Project] [Code-Torch7] [Paper] [Poster] [Video]
  • Annotating Object Instances with a Polygon-RNN [Paper]
  • MaskLab: Instance Segmentation by Refining Object Detection with Semantic and Direction Features [Paper]
  • BlitzNet: A Real-Time Deep Network for Scene Understanding [Paper]
  • FCIS [Paper]Code
  • MNC:Instance-aware Semantic Segmentation via Multi-task Network Cascades [Paper]Code
  • DeepMask:Learning to Segment Object Candidates [Paper] Code
  • SharpMask [Paper]Code
  • RIS:Recurrent Instance Segmentation [Paper]Code
  • FastMask: Segment Multi-scale Object Candidates in One Shot [Paper]Code
  • Proposal-free network for instance-level object segmentation [Paper]

Segment Object Candidates

Foreground Object Segmentation

Popular Methods and Implementations

Annotation Tools:

Distinguished Researchers & Teams:

Results:

Reference

https://github.com/nightrome/really-awesome-semantic-segmentation

https://github.com/mrgloom/awesome-semantic-segmentation