Recurrent Neural Networks for Semantic Instance Segmentation

Code supporting the paper:

Amaia Salvador, Miriam Bellver, Manel Baradad, Ferran Marques, Jordi Torres and Xavier Giro-i-Nieto. Recurrent Neural Networks for Semantic Instance Segmentation

Download the pdf here.

If you find it useful, please consider citing !

Installation

Clone the repo:

git clone https://github.com/imatge-upc/rsis.git

Install requirements pip install -r requirements.txt
Install PyTorch 0.2 (choose the whl file according to your setup):

pip install http://download.pytorch.org/whl/cu80/torch-0.2.0.post3-cp27-cp27mu-manylinux1_x86_64.whl  
pip install torchvision

Compile COCO Python API and add it to your PYTHONPATH:

cd src/coco/PythonAPI;
make

# Run from the root directory of this project
export PYTHONPATH=$PYTHONPATH:./src/coco/PythonAPI

Data

Pascal VOC 2012

Download Pascal VOC 2012:

wget http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar
tar -xvf VOCtrainval_11-May-2012.tar

Download additional VOC annotations from Hariharan et al. Semantic contours from inverse detectors. ICCV 2011:

# berkeley augmented Pascal VOC
wget http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/semantic_contours/benchmark.tgz # 1.3 GB
tar zxvf benchmark.tgz

Create a merged dataset out of the two sets of images and annotations:

python src/dataloader/pascalplus_gen.py --voc_dir /path/to/pascal --contours_dir /path/to/additional/dataset --vocplus_dir /path/to/merged

Precompute instance and semantic segmentation masks & ground truth files in COCO format:

python src/dataloader/pascal_precompute.py --split train --pascal_dir /path/to/merged

You must run this three times for the different splits (train, val and test).

Point args.pascal_dir to /path/to/merged.

CVPPP

Download the training CVPPP dataset from their website. In our case we just worked with the A1 dataset. Extract the A1 package and point args.leaves_dir to this folder. To obtain the test set for evaluation you will have to contact the organizers.

Cityscapes

Download the Cityscapes dataset from their website. Extract the images and the labels into the same directory and point args.cityscapes_dir to it.

Training

Train the model with python train.py -model_name model_name. Checkpoints and logs will be saved under ../models/model_name.
Other arguments can be passed as well. For convenience, scripts to train with typical parameters are provided under scripts/.
Visdom can be enabled to monitor training losses and outputs:
- First run the visdom server withpython -m visdom.server.
- Run training with the --visdom flag. Navigate to localhost:8097 to visualize training curves.
Plot loss curves at any time with python plot_curves.py -model_name model_name.

Evaluation

We provide bash scripts to display results and evaluate models for the three datasets. You can find them under the scripts folder.

In the case of cityscapes, the evaluation bash script will generate the results in the appropiate format to use the official evaluation code.

For CVPPP, the evaluation bash script will generate the results in the appropiate format to use the evaluation scripts that are provided with the dataset.

Pretrained models

Download weights for models trained with:

Extract and place the obtained folder under models directory. You can then run evaluation scripts with the downloaded model by setting args.model_name to the name of the folder.

Contact

For questions and suggestions use the issues section or send an e-mail to amaia.salvador@upc.edu

Additional notes to GPI users

Helpful commands to train on the GPI cluster and get visualizations in your browser:

Start server: with srun --tunnel $UID:$UID python -m visdom.server -port $UID.
Check the node where the server launched and (eg. c3).
Run training with:

srun --gres=gpu:1,gmem:12G --mem=10G python train.py --visdom -port $UID -server http://c3

Notice that the port and the server must match the ones used in the previous run.

echo $UID to know which port you are using.
ssh tunnel (run this in local machine): ssh -L 8889:localhost:YOUR_UID -p2222 user@imatge.upc.edu.
Navigate to localhost:8889 in your browser locally.

mohuamuyu/rsis