In CoRL 2020 [arXiv] [Robot Video] [project page] [pdf]
Adithya Murali, Weiyu Liu, Kenneth Marino, Sonia Chernova, Abhinav Gupta
Carnegie Mellon University Robotics Institute, Georgia Institute of Technology, Facebook AI Research
This is a pytorch-based implementation for our CoRL 2020 paper on task-oriented grasping. If you find this work useful in your research, please cite:
@inproceedings{murali2020taskgrasp,
title={Same Object, Different Grasps: Data and Semantic Knowledge for Task-Oriented Grasping},
author={Murali, Adithyavairavan and Liu, Weiyu and Marino, Kenneth and Chernova, Sonia and Gupta, Abhinav},
booktitle={Conference on Robot Learning},
year={2020}
}
The code has been tested on Ubuntu 16.04 and with CUDA 10.0.
- Create a virtual env or conda environment with python3
conda create --name taskgrasp python=3.6
conda activate taskgrasp
- Make a workspace, clone the repo
mkdir ~/taskgrasp_ws && cd ~/taskgrasp_ws
git clone https://github.com/adithyamurali/TaskGrasp.git
- Install dependencies
cd TaskGrasp
conda install pytorch=1.4.0 torchvision cudatoolkit=10.0 -c pytorch
pip install -r requirements.txt
- Compile and install PointNet ops
cd ~/taskgrasp_ws
git clone https://github.com/erikwijmans/Pointnet2_PyTorch.git
cd Pointnet2_PyTorch
pip install -r requirements.txt
pip install -e .
- Install Pytorch Geometric (only tested on v1.5.0)
pip install torch-scatter==latest+cu100 -f https://pytorch-geometric.com/whl/torch-1.4.0.html && pip install torch-sparse==latest+cu100 -f https://pytorch-geometric.com/whl/torch-1.4.0.html && pip install torch-cluster==latest+cu100 -f https://pytorch-geometric.com/whl/torch-1.4.0.html && pip install torch-spline-conv==latest+cu100 -f https://pytorch-geometric.com/whl/torch-1.4.0.html && pip install torch-geometric==1.5.0
The dataset (5 GB) could be downloaded here and place it in the data
folder as shown below:
cd ~/taskgrasp_ws/TaskGrasp
unzip ~/Downloads/data.zip -d ./
rm ~/Downloads/data.zip
To run any of the demo scripts explained below, do the same with the pretrained models (589 MB) and config files (10 KB) and put them in the checkpoints
and cfg
folder respectively.
Coming Soon: We are trying to release mesh models for the objects as well.
NOTE: The stable grasps were sampled from the point cloud using the agile_grasp repo from GT-RAIL.
Point Cloud | Stable grasps |
---|---|
![]() |
![]() |
To visualize the point cloud data and stable grasps:
python visualize.py --data_and_grasps --obj_name 124_paint_roller
Add the --obj_path
argument with the absolute path to the dataset if you have placed it somewhere other than the default location (data/taskgrasp
). The object can be specified with the --obj_name
argument and the full list of objects can be found here.
To visualize the labelled task-oriented grasps in the TaskGrasp dataset:
python visualize.py --visualize_labels --visualize_labels_blacklist_object 124_paint_roller
To visualize a specific grasp:
python visualize.py --obj_name 124_paint_roller --visualize_grasp --grasp_id 10
To run any of the training or evaluation scripts, download the config files from here and put them in the cfg folder. To train a model:
python gcngrasp/train.py --cfg_file cfg/train/gcngrasp/gcngrasp_split_mode_t_split_idx_3_.yml
To run pretrained models, download the models from here and unzip them into the checkpoints folder. To evaluate trained model on test set:
python gcngrasp/eval.py cfg/eval/gcngrasp/gcngrasp_split_mode_t_split_idx_3_.yml --save
Here's how to run evaluation on a single point cloud (e.g. sample data of an unknown object captured from a depth sensor) and assuming you already have sampled stable grasps. Make sure you have downloaded the pretrained models and data from the previous step, and run the following:
python gcngrasp/infer.py cfg/eval/gcngrasp/gcngrasp_split_mode_t_split_idx_3_.yml --obj_name pan --obj_class pan.n.01 --task pour