In Tac-Man: Tactile-Informed Prior-Free Manipulation of Articulated Objects, we present a prior-free strategy for articulated object manipulation leveraging tactile sensors. This approach has been validated through extensive real-world experiments, and here we present our simulation studies for further large-scale validation.
This repository hosts the code for the simulation studies as detailed in our paper. For a comprehensive understanding, please consider reading the full paper.
Installation of Omniverse Isaac SimOur simulations run on NVIDIA Isaac Sim, which is part of the NVIDIA Omniverse platform. Ensure that you install the Isaac Sim 2022.2.1 for compatibility with our code. We are actively working on supporting later versions.
Post-installation, the simulation environment can be initiated using the provided Python script. It is advisable to use an alias, as suggested in the official documentation:
alias omni_python='~/.local/share/ov/pkg/isaac_sim-2022.2.1/python.sh'Next, we need a few additional packages:
omni_python -m pip install trimesh tqdm bezier We evaluate our methods on a subset of objects from the PartNet-Mobility dataset. We also use annotations from GAPartNet for initializing grasps.
The datasets have been pre-processed to include extra details (see Appx. C of our paper.). To access the pre-processed data, please fill out the GAPartNet Dataset Application Form to comply with GAPartNet's terms and conditions. Once completed, download data from the link below and extract the data into data/gapartnet/.
Data link: Google Drive.
Please download the data from the link below and unzip the contents to data/playboards/
Data link: Google Drive.
For objects from PartNet-Mobility, please run:
omni_python run_partnet.py
--idx # Test case index, 0 by default.
--obj_scale # object scaling, 0.5 by default
--x_offset # x-axis offset of robot base, 0.75 by defualt
--y_offset # y-axis offset of robot base, 0.0 by defualt
--z_offset # z-axis offset of robot base, 0.0 by defualt
--headless # run simulation in headless mode
--capture # record the simulation with Replicator
--device # computing device for transformation and states, "cuda" by defaultFor each case, we have pre-annotated grasps and robot base positions as initialization data in data/gapartnet/grasps.json.
For playboards with Bezier trajectories, run:
omni_python run_playboard.py
--n_ctrl # amount of control points of the Bezier curve
--id # test case index (under each `n_ctrl`)
--x_offset # x-axis offset of robot base, 0.5 by defualt
--y_offset # y-axis offset of robot base, 0.0 by defualt
--z_offset # z-axis offset of robot base, 0.0 by defualt
--inverse # run the trajectory in reverse
--headless # run simulation in headless mode
--capture # record the simulation with Replicator
--device # computing device for transformation and states, "cuda" by default
We find that Correct positioning of the robot base is critical to avoid joint limits during interactions. This problem is caused by limited workspace and could be solved by using mobile manipulators rather than fixed robot arms with limited workspace. For simplicity, we provide proper robot placements in data/..., with which the robot should be able to finish the manipulation. If it is stuck at some point, retry with an adjusted base position.
Due to the some bugs in the deformable body simulation within Isaac Sim, we use a simple heuristic lock_joint_drive(...) and release_joint_drive(...) to mimic gripper compliance. Please refer to Appx. B of our paper.
While the codebase is designed with parallel evaluation in mind, the implementation is not yet complete. We aim to introduce this feature in future updates.
- Some URDF cannot be imported properly when launching
run_partnet.pyusing Isaac Sim 2023.
If you encounter issues or have questions, please open a GitHub issue or contact Yuyang (liyuyang20@mails.tsinghua.edu.cn).
If you find our research beneficial, we appreciate citations to our paper:
@article{zhao2024tac,
title={Tac-Man: Tactile-Informed Prior-Free Manipulation of Articulated Objects},
author={Zhao, Zihang and Li, Yuyang and Li, Wanlin and Qi, Zhenghao and Ruan, Lecheng and Zhu, Yixin and Althoefer, Kaspar},
journal={arXiv preprint arXiv:2403.01694},
year={2024}
}