py-faster-rcnn has been deprecated. Please see Detectron, which includes an implementation of Mask R-CNN.

Disclaimer

The official Faster R-CNN code (written in MATLAB) is available here. If your goal is to reproduce the results in our NIPS 2015 paper, please use the official code.

This repository contains a Python reimplementation of the MATLAB code. This Python implementation is built on a fork of Fast R-CNN. There are slight differences between the two implementations. In particular, this Python port

is ~10% slower at test-time, because some operations execute on the CPU in Python layers (e.g., 220ms / image vs. 200ms / image for VGG16)
gives similar, but not exactly the same, mAP as the MATLAB version
is not compatible with models trained using the MATLAB code due to the minor implementation differences
includes approximate joint training that is 1.5x faster than alternating optimization (for VGG16) -- see these slides for more information

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

By Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun (Microsoft Research)

This Python implementation contains contributions from Sean Bell (Cornell) written during an MSR internship.

Please see the official README.md for more details.

Faster R-CNN was initially described in an arXiv tech report and was subsequently published in NIPS 2015.

License

Faster R-CNN is released under the MIT License (refer to the LICENSE file for details).

Citing Faster R-CNN

If you find Faster R-CNN useful in your research, please consider citing:

@inproceedings{renNIPS15fasterrcnn,
    Author = {Shaoqing Ren and Kaiming He and Ross Girshick and Jian Sun},
    Title = {Faster {R-CNN}: Towards Real-Time Object Detection
             with Region Proposal Networks},
    Booktitle = {Advances in Neural Information Processing Systems ({NIPS})},
    Year = {2015}
}

Requirements: software
Requirements: hardware
Basic installation
Demo
Beyond the demo: training and testing
Usage

Requirements: Software

NOTE: If you are having issues compiling and you are using a recent version of CUDA/cuDNN, please consult this issue for a workaround

Requirements for Caffe and pycaffe (see: Caffe installation instructions)

Note: Caffe must be built with support for Python layers!

# In your Makefile.config, make sure to have this line uncommented
WITH_PYTHON_LAYER := 1
# Unrelatedly, it's also recommended that you use CUDNN
USE_CUDNN := 1

You can download my Makefile.config for reference.
2. Python packages you need to install additionally: cython, python-opencv, easydict
3. [Optional] MATLAB is required for official PASCAL VOC evaluation only. The code now includes unofficial Python evaluation code.

Requirements: Hardware

For training smaller networks (ZF, VGG_CNN_M_1024) a good GPU (e.g., Titan, K20, K40, ...) with at least 3G of memory suffices
For training Fast R-CNN with VGG16, you'll need a K40 (~11G of memory)
For training the end-to-end version of Faster R-CNN with VGG16, 3G of GPU memory is sufficient (using CUDNN)

Installation (Sufficient for the demo)

Clone the Faster R-CNN repository

# Make sure to clone with --recursive
git clone --recursive https://github.com/foss-for-synopsys-dwc-arc-processors/py-faster-rcnn.git

We'll call the directory that you cloned Faster R-CNN into FRCN_ROOT

Ignore notes 1 and 2 if you followed step 1 above.

Note 1: If you didn't clone Faster R-CNN with the --recursive flag, then you'll need to manually clone the caffe-fast-rcnn submodule:
```
git submodule update --init --recursive
```
Note 2: The caffe-fast-rcnn submodule needs to be on the faster-rcnn branch (or equivalent detached state). This will happen automatically if you followed step 1 instructions.
Build the Cython modules
```
cd $FRCN_ROOT/lib
make
```
- Note: You may need to align the CUDA and gcc versions in the environment first to avoid possible build errors.

Build Caffe and pycaffe

cd $FRCN_ROOT/caffe-fast-rcnn
# Now follow the Caffe installation instructions here:
#   http://caffe.berkeleyvision.org/installation.html

# If you're experienced with Caffe and have all of the requirements installed
# and your Makefile.config in place, then simply do:
make -j8 && make pycaffe

Download pre-computed Faster R-CNN detectors
```
cd $FRCN_ROOT
./data/scripts/fetch_faster_rcnn_models.sh
```
This will populate the $FRCN_ROOT/data folder with faster_rcnn_models. See data/README.md for details. These models were trained on VOC 2007 trainval.

Demo

After successfully completing basic installation, you'll be ready to run the demo.

To run the demo

cd $FRCN_ROOT
./tools/demo.py

The demo performs detection using a VGG16 network trained for detection on PASCAL VOC 2007.

Beyond the demo: Installation for training and testing models

The following steps 1-6 are used to download VOC2007 and VOC2012 datasets, and merge them together to create VOC0712 dataset mainly for Faster-RCNN pruning dataset preparation.

$VOCdevkit points to where you put the VOC 2007, 2012 and the combined 0712 datasets
$FRCN_ROOT points to where this py-faster-rcnn located

Download the training, validation, test data and VOCdevkit

wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtrainval_06-Nov-2007.tar
wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtest_06-Nov-2007.tar
wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCdevkit_08-Jun-2007.tar
wget http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar

Extract all of these tars into one directory named VOCdevkit

tar xvf VOCtrainval_06-Nov-2007.tar
tar xvf VOCtest_06-Nov-2007.tar
tar xvf VOCdevkit_08-Jun-2007.tar
tar xvf VOCtrainval_11-May-2012.tar

It should have this basic structure

$VOCdevkit/                           # development kit
$VOCdevkit/VOCcode/                   # VOC utility code
$VOCdevkit/VOC2007                    # image sets, annotations, etc.
$VOCdevkit/VOC2012                    # image sets, annotations, etc.
# ... and several other directories ...

Merge VOC 2007 data and VOC 2012 data manually

Create a new directory named VOC0712 in VOCdevkit, put all subfolders of VOC2007 and VOC2012 (Annotations, JPEGImages, SegmentationClass, SegmentationObject) except ImageSets into VOC0712. Now the combined Annotations and JPEGImages folder should contain 27088 xmls and images and the SegmentationClass and SegmentationObject should contain 3545 images each. For ImageSets, you can manually merge the text files within, or just unzip the ImageSets.zip to VOC0712 folder.
Create results folder in VOCdevkit for VOC0712: mkdir -p VOCdevkit/results/VOC0712/Main
Create symlinks for the PASCAL VOC dataset
```
cd $FRCN_ROOT/data
ln -s $VOCdevkit VOCdevkit0712
```
Using symlinks is a good idea because you will likely want to share the same PASCAL dataset installation between multiple projects.

Update cocoAPI

The scripts under lib/pycocotools could be updated according to cocoapi

Faster-RCNN Pruning

Prepare the merged VOC0712 dataset successfully by following steps above.
Please check the following packages are installed: cython, python-opencv, easydict, python-tk.

Build the Cython modules

cd $FRCN_ROOT/lib
make clean # clean previous built modules, if you don't want to remove, please don't this command
make

Make sure that your Synopsys Caffe is already built and install, and PYTHONPATH is set correctly.
Set PYTHONPATH to $FRCNN_ROOT/lib:$PYTHONPATH
cd to $FRCN_ROOT/pruning folder, download the faster-rcnn-resnet without ohem pretrained model from faster-rcnn-resnet
You can check the md5sum of the caffemodel we used for testing in the pruning folder.
Pruning the faster-rcnn model with an accuracy drop tolerance of 0.01: evprune --model_path . --faster_rcnn_path ../ --config_file prune.cfg --accuracy_tolerance 0.01 --output_path drop0.01
Notice:
- When you changed the train or val list, you need to manually remove the cached data in $VOCdevkit/annotations_cache and $FRCN_ROOT/data/cache
- If you enter to an error, please also check the <prune_output>/default_accuracy_log.txt file, sometimes it will contains error message dumped by faster-rcnn tool itself.
- Test is done using TiTan XP GPU with 12G RAM, pruning cost about 50 hours, it can achieve compression factor of 1.43, Mean AP of 0.7867, which original Mean AP is 0.7871.

Download pre-trained ImageNet models

Pre-trained ImageNet models can be downloaded for the three networks described in the paper: ZF and VGG16.

cd $FRCN_ROOT
./data/scripts/fetch_imagenet_models.sh

VGG16 comes from the Caffe Model Zoo, but is provided here for your convenience. ZF was trained at MSRA.

Usage

To train and test a Faster R-CNN detector using the alternating optimization algorithm from our NIPS 2015 paper, use experiments/scripts/faster_rcnn_alt_opt.sh. Output is written underneath $FRCN_ROOT/output.

cd $FRCN_ROOT
./experiments/scripts/faster_rcnn_alt_opt.sh [GPU_ID] [NET] [--set ...]
# GPU_ID is the GPU you want to train on
# NET in {ZF, VGG_CNN_M_1024, VGG16} is the network arch to use
# --set ... allows you to specify fast_rcnn.config options, e.g.
#   --set EXP_DIR seed_rng1701 RNG_SEED 1701

("alt opt" refers to the alternating optimization training algorithm described in the NIPS paper.)

To train and test a Faster R-CNN detector using the approximate joint training method, use experiments/scripts/faster_rcnn_end2end.sh. Output is written underneath $FRCN_ROOT/output.

cd $FRCN_ROOT
./experiments/scripts/faster_rcnn_end2end.sh [GPU_ID] [NET] [--set ...]
# GPU_ID is the GPU you want to train on
# NET in {ZF, VGG_CNN_M_1024, VGG16} is the network arch to use
# --set ... allows you to specify fast_rcnn.config options, e.g.
#   --set EXP_DIR seed_rng1701 RNG_SEED 1701

This method trains the RPN module jointly with the Fast R-CNN network, rather than alternating between training the two. It results in faster (~ 1.5x speedup) training times and similar detection accuracy. See these slides for more details.

Artifacts generated by the scripts in tools are written in this directory.

Trained Fast R-CNN networks are saved under:

output/<experiment directory>/<dataset name>/

Test outputs are saved under:

output/<experiment directory>/<dataset name>/<network snapshot name>/

foss-for-synopsys-dwc-arc-processors/py-faster-rcnn