Learning to Segment Instances in Videos with Spatial Propagation Network

This paper is available at the 2017 DAVIS Challenge website.

Check our results in this video.

Contact: Jingchun Cheng (chengjingchun at gmail dot com)

Cite the Paper

If you find that our method is useful in your research, please cite:

@article{DAVIS2017-6th,
  author = {J. Cheng and S. Liu and Y.-H. Tsai and W.-C. Hung and S. Gupta and J. Gu and J. Kautz and S. Wang and M.-H. Yang}, 
  title = {Learning to Segment Instances in Videos with Spatial Propagation Network}, 
  journal = {The 2017 DAVIS Challenge on Video Object Segmentation - CVPR Workshops}, 
  year = {2017}
}

About the Code

The code released here mainly consistes of two parts in the paper: foreground segmentation and instance recognition.
It contains the parent net for foreground segmentation and training codes for instance recognition networks.
The matlab_code folder contains a simple version of our CRAF step for segmentation refinement.

Requirements

Install caffe and pycaffe at http://caffe.berkeleyvision.org/.
Download the DAVIS 2017 dataset and put it in the data folder.
Download the pre-trained foreground/background model here and put it in the pretrained folder.

Training

Train the per-object recognition model.
cd training
python solve.py PATH_OF_MODEL PATH_OF_SOLVER
Foe example, on the 'choreography' video for the 1st object, run:
python solve.py ../pretrained/PN_ResNetF.caffemodel ../ResNetF/testnet_per_obj/choreography/solver_1.prototxt

Testing

Test the general foreground/backgroung model.
python infer_test_fgbg.py PATH_OF_MODEL PATH_OF_RESULT VIDEO_NAME
Foe example, on the 'lions' video, run:
python infer_test_fgbg.py pretrained/PN_ResNetF.caffemodel results/fgbg lions
Test the object instance model.
python infer_test_perobj.py MODEL_ITERATION VIDEO_NAME OBJECT_ID
For example, on the 'lions' video for the 2nd object, run:
python infer_test_perobj.py 3000 lions 2
Run example_CRAF.m in the matlab_code folder for a demo on CRAF segmentation refinement.

Download Our Segmentation Results on 2017 DAVIS Challenge

General foreground/background segmentation here
Instance-level object segmentation without refinement here
Final instance-level object segmentation with refinement here

Note

The model and code are available for non-commercial research purposes only.

09/2017: code and model released
03/2018: pre-trained model updated

JingchunCheng/Seg-with-SPN