SubCNN: Subcategory-aware Convolutional Neural Networks for Object Proposals and Detection

Created by Yu Xiang at CVGL at Stanford University.

Part of this code is based on the Fast R-CNN created by Ross Girshick at Microsoft Research, Redmond.

Introduction

We introduce a new region proposal network that uses subcategory information to guide the proposal generating process, and a new detection network for joint detection and subcategory classification. By using subcategories related to object pose, we achieve state-of-the-art performance on both detection and pose estimation on commonly used benchmarks, such as KITTI and PASCAL3D+.

This package supports

Subcategory-aware region proposal network
Subcategory-aware detection network
Region proposal network in Faster R-CNN (Ren et al. NIPS 2015)
Detection network in Faster R-CNN (Ren et al. NIPS 2015)
Experiments on the following datasets: KITTI Detection, PASCAL VOC, PASCAL3D+, KITTI Tracking sequences, MOT sequences

License

SubCNN is released under the MIT License (refer to the LICENSE file for details).

Citing SubCNN

If you find SubCNN useful in your research, please consider citing:

@incollection{xiang2016subcategory,
    author = {Xiang, Yu and Choi, Wongun and Lin, Yuanqing and Savarese, Silvio},
    title = {Subcategory-aware Convolutional Neural Networks for Object Proposals and Detection},
    booktitle = {arXiv:1604.04693},
    year = {2016}
}

Installation

Clone the SubCNN repository

# Make sure to clone with --recursive
git clone --recursive https://github.com/yuxng/SubCNN.git

We'll call the directory that you cloned SubCNN into ROOT

Ignore notes 1 and 2 if you followed step 1 above.

Note 1: If you didn't clone SubCNN with the --recursive flag, then you'll need to manually clone the caffe-fast-rcnn submodule:
```
git submodule update --init --recursive
```
Note 2: The caffe-fast-rcnn submodule needs to be on the fast-rcnn branch (or equivalent detached state). This will happen automatically if you follow these instructions.
Build the Cython modules
```
cd $ROOT/fast-rcnn/lib
make
```

Build our modified Caffe and pycaffe

cd $ROOT/caffe-fast-rcnn
# Now follow the Caffe installation instructions here:
#   http://caffe.berkeleyvision.org/installation.html

# If you're experienced with Caffe and have all of the requirements installed
# and your Makefile.config in place, then simply do:
make -j8 && make pycaffe

Download pre-trained ImageNet models
```
cd $ROOT/fast-rcnn
./data/scripts/fetch_imagenet_models.sh
```
This will populate the $ROOT/fast-rcnn/data folder with imagenet_models.

Running with the KITTI detection dataset

Download the KITTI detection dataset from here.

Create symlinks for the KITTI detection dataset

cd $ROOT/fast-rcnn/data/KITTI
ln -s $data_object_image_2 data_object_image_2

Unzip the voxel_exemplars.zip in $ROOT/fast-rcnn/data/KITTI. These are subcategories from 3D voxel patterns (Xiang et al. CVPR'15).

Run the region proposal network to generate region proposals

cd $ROOT/fast-rcnn

# subcategory-aware RPN for validation
./experiments/scripts/kitti_val_caffenet_rpn.sh $GPU_ID

# subcategory-aware RPN for testing
./experiments/scripts/kitti_test_caffenet_rpn_6k8k.sh $GPU_ID

# Faster RCNN RPN for validation
./experiments/scripts/kitti_val_caffenet_rpn_msr.sh $GPU_ID

# Faster RCNN RPN for testing
./experiments/scripts/kitti_test_caffenet_rpn_msr_6k8k.sh $GPU_ID

Copy the region proposals to $ROOT/fast-rcnn/data/KITTI/region_proposals/RPN_*:

# validation (125 subcategories for car)
$ROOT/fast-rcnn/data/KITTI/region_proposals/RPN_125/training   # a directory contains region proposals for training images: 000000.txt, ..., 007480.txt

# testing (227 subcategories for car)
$ROOT/fast-rcnn/data/KITTI/region_proposals/RPN_227/training   # a directory contains region proposals for training images: 000000.txt, ..., 007480.txt
$ROOT/fast-rcnn/data/KITTI/region_proposals/RPN_227/testing    # a directory contains region proposals for testing  images: 000000.txt, ..., 007517.txt

Run the detection network

cd $ROOT/fast-rcnn

# subcategory-aware detection network for validation
./experiments/scripts/kitti_val_caffenet_rcnn_multiscale.sh $GPU_ID

# subcategory-aware detection network for testing
./experiments/scripts/kitti_test_caffenet_rcnn_multiscale_6k8k.sh $GPU_ID

# subcategory-aware detection network for testing with VGG16
./experiments/scripts/kitti_test_vgg16_rcnn_multiscale_6k8k.sh $GPU_ID

# subcategory-aware detection network for testing with GoogleNet
./experiments/scripts/kitti_test_googlenet_rcnn.sh $GPU_ID

# Faster RCNN detection network for validation
./experiments/scripts/kitti_val_caffenet_rcnn_msr.sh $GPU_ID

# Faster RCNN detection network for testing
./experiments/scripts/kitti_test_caffenet_rcnn_original_msr.sh $GPU_ID

Running with the PASCAL3D+ dataset

Download the PASCAL3D+ dataset from here.

Create symlinks for the PASCAL VOC 2012 dataset

cd $ROOT/fast-rcnn/data/PASCAL3D
ln -s $PASCAL3D+_release1.1/PASCAL/VOCdevkit VOCdevkit2012

Unzip the voxel_exemplars.zip in $ROOT/fast-rcnn/data/PASCAL3D. These are subcategories from 3D voxel patterns (Xiang et al. CVPR'15).

Run the region proposal network to generate region proposals

cd $ROOT/fast-rcnn

# subcategory-aware RPN
./experiments/scripts/pascal3d_vgg16_rpn_6k8k.sh $GPU_ID

Copy the region proposals to $ROOT/fast-rcnn/data/PASCAL3D/region_proposals/RPN_6k8k:

# training set and validation set
$ROOT/fast-rcnn/data/PASCAL3D/region_proposals/RPN_6k8k/training     # a directory contains region proposals for training images
$ROOT/fast-rcnn/data/PASCAL3D/region_proposals/RPN_6k8k/validation   # a directory contains region proposals for validation images

Run the detection network

cd $ROOT/fast-rcnn

# subcategory-aware detection network
./experiments/scripts/pascal3d_vgg16_rcnn_multiscale.sh $GPU_ID

Running with other datasets

The package also supports running experiments on the PASCAL VOC detection dataset, the KITTI Tracking dataset and the MOT Tracking dataset. Please see the scripts in $ROOT/fast-rcnn/experiments/scripts.

Running with the NTHU dataset (internal usage)

The NTHU dataset should have a directory named 'data', under which it has the following structure:

$data/                           # the directory contains all the data
$data/71                         # a directory for video 71: 000001.jpg, ..., 002956.jpg
$data/71.txt                     # a txt file contains the frame names: 000001 \n 000002 \n ... 002956
# ... and several other directories and txt files ...

Create symlinks for the NTHU dataset

cd $ROOT/fast-rcnn/data/NTHU
ln -s $data data

Run the region proposal network to generate region proposals, modify the script to run with different videos
```
cd $ROOT/fast-rcnn
./experiments/scripts/nthu_caffenet_rpn_6k8k.sh $GPU_ID
```

Copy the region proposals to $ROOT/fast-rcnn/data/NTHU/region_proposals/RPN_6k8k:

$ROOT/fast-rcnn/data/NTHU/region_proposals/RPN_6k8k/71    # a directory contains region proposals for video 71: 000001.txt, ..., 002956.txt

Run the detection network, modify the script to run with different videos

cd $ROOT/fast-rcnn
./experiments/scripts/nthu_caffenet_rcnn_multiscale_6k8k.sh $GPU_ID

ConnorSiXiong/SubCNN