By Thanh-Toan Do*, Anh Nguyen*, Ian Reid (* equal contribution)
-
Caffe
- Caffe must be built with support for Python layers.
-
Hardware
- To train a full AffordanceNet, you'll need a GPU with ~11GB (e.g. Titan, K20, K40, Tesla, ...).
- To test a full AffordanceNet, you'll need ~6GB GPU.
-
[Optional] Robot Demo - This option is from the original repo. Since this repo is now used only as a submodule in the
rail_part_affordance_detection
repo, check out the main repo for more up-to-date robot demo.
This package has been tested on Ubuntu 18.04 with CUDA 9.2 and CUDA 10.1. Installing these versions of CUDA should ensure successful installation.
Caffe needs to be built from source since this package uses customized caffe functions.
-
Clone the AffordanceNet repository into your
$AffordanceNet_ROOT
folder if you haven't already done so. -
Build
Caffe
andpycaffe
from source:cd $AffordanceNet_ROOT/caffe-affordance-net
- Now follow the Caffe installation instructions to
make sure you have all of the requirements installed.
- Check out the step-by-step instructions for
Ubuntu installation
and make sure the dependencies underGeneral dependencies
,BLAS
, andPython
forUbuntu (< 17.04)
are installed. ForBLAS
, just useATLAS
. - Go back to the main page and follow the instructions to install necessary python packages using pip.
- Check out the step-by-step instructions for
- Copy an existing Makefile.config from
$AffordanceNet_ROOT/caffe-affordance-net/makefile_config_template
to$AffordanceNet_ROOT/caffe-affordance-net
make -j8
- If run into problems, this webpage usually provides solutions.
make pycaffe
export PYTHONPATH=$AffordanceNet_ROOT/caffe-affordance-net/python:$PYTHONPATH
-
Build the Cython modules:
cd $AffordanceNet_ROOT/lib
make
-
Download pretrained weights (Google Drive, One Drive). This weight is trained on the training set of the IIT-AFF dataset:
- Extract the file you downloaded to
$AffordanceNet_ROOT
- Make sure you have the caffemodel file like this:
'$AffordanceNet_ROOT/pretrained/AffordanceNet_200K.caffemodel
- Extract the file you downloaded to
After successfully completing installation, you'll be ready to run the demo.
-
Demo on static images:
cd $AffordanceNet_ROOT/tools
python demo_img.py
- You should see the detected objects and their affordances.
-
(Optional) Demo on depth camera (such as Asus Xtion):
- With AffordanceNet and the depth camera, you can easily select the interested object and its affordances for robotic applications such as grasping, pouring, etc.
- First, launch your depth camera with ROS, OpenNI, etc.
cd $AffordanceNet_ROOT/tools
python demo_asus.py
- You may want to change the object id and/or affordance id (line
380
,381
indemo_asus.py
). Currently, we select thebottle
and itsgrasp
affordance. - The 3D grasp pose can be visualized with rviz. You should see something like this:
-
We train AffordanceNet on IIT-AFF dataset
- We need to format IIT-AFF dataset as in Pascal-VOC dataset for training.
- For your convinience, we did it for you. Just download this file (Google Drive, One Drive) and extract it into your
$AffordanceNet_ROOT
folder. - The extracted folder should contain three sub-folders:
$AffordanceNet_ROOT/data/cache
,$AffordanceNet_ROOT/data/imagenet_models
, and$AffordanceNet_ROOT/data/VOCdevkit2012
.
-
Train AffordanceNet:
cd $AffordanceNet_ROOT
./experiments/scripts/faster_rcnn_end2end.sh [GPU_ID] [NET] [--set ...]
- e.g.:
./experiments/scripts/faster_rcnn_end2end.sh 0 VGG16 pascal_voc
- We use
pascal_voc
alias although we're training using the IIT-AFF dataset.
- AffordanceNet vs. Mask-RCNN: AffordanceNet can be considered as a general version of Mask-RCNN when we have multiple classes inside each instance.
- The current network achitecture is slightly diffrent from the paper, but it achieves the same accuracy.
- Train AffordanceNet on your data:
- Format your images as in Pascal-VOC dataset (as in
$AffordanceNet_ROOT/data/VOCdevkit2012
folder). - Prepare the affordance masks (as in
$AffordanceNet_ROOT/data/cache
folder): For each object in the image, we need to create a mask and save as a .sm file. See$AffordanceNet_ROOT/utils
for details.
- Format your images as in Pascal-VOC dataset (as in
- If
ImportError: No module named Image
occurs when running demo, change import fromimport Image
tofrom PIL import Image
in$AffordanceNet_ROOT/lib/fast_rcnn/test.py
If you find AffordanceNet useful in your research, please consider citing:
@inproceedings{AffordanceNet18,
title={AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detection},
author={Do, Thanh-Toan and Nguyen, Anh and Reid, Ian},
booktitle={International Conference on Robotics and Automation (ICRA)},
year={2018}
}
If you use IIT-AFF dataset, please consider citing:
@inproceedings{Nguyen17,
title={Object-Based Affordances Detection with Convolutional Neural Networks and Dense Conditional Random Fields},
author={Nguyen, Anh and Kanoulas, Dimitrios and Caldwell, Darwin G and Tsagarakis, Nikos G},
booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
year={2017},
}
MIT License
This repo used a lot of source code from Faster-RCNN
If you have any questions or comments, please send us an email: thanh-toan.do@adelaide.edu.au
and anh.nguyen@iit.it