Repository for reactive task and motion planning making use of active inference for symbolic planning, and a new multi-modal MPPI for motion planning. Rollouts are evaluated in IsaacGym, a parallelizable physics simulator.
First, clone the repo in your folder and create the conda environment.
cd <project_folder>
git clone https://github.com/tud-amr/m3p2i-aip.git
conda create -n m3p2i-aip python=3.8
conda activate m3p2i-aip
This project requires the source code of IsaacGym. Check for the prerequisites and troubleshooting. Download it from https://developer.nvidia.com/isaac-gym, unzip and paste it in the thirdparty
folder. Move to IsaacGym and install the package.
cd <project_folder>/m3p2i-aip/thirdparty/IsaacGym_Preview_4_Package/isaacgym/python
pip install -e.
Then install the current package by:
cd <project_folder>/m3p2i-aip
pip install -e.
Now you are ready to test an example file, where you can drive the robot around with ASDW keys.
cd <project_folder>/m3p2i-aip/examples
python3 example_key.py
If you want to test the TAMP framework, you will need two instances of Isaac Gym, one for throwing the rollouts and deriving the optimal solution, and one for updating the "real system". Please run the commands below in two terminals from the scripts
folder with activated python environment.
Run this terminal first:
cd <project_folder>/m3p2i-aip/scripts
conda activate m3p2i-aip
python3 reactive_tamp.py
Then run the second terminal:
cd <project_folder>/m3p2i-aip/scripts
conda activate m3p2i-aip
python3 sim.py
Specifically, you can test the following:
python3 reactive_tamp.py task=navigation goal="[-3, 3]"
python3 sim.py
Note
Feel free to change the goal position.
python3 reactive_tamp.py task=push goal="[-1, -1]"
python3 sim.py
Note
Feel free to change the goal position. Pushing will always fail if the initial position is in the corner.
python3 reactive_tamp.py task=pull goal="[0, 0]"
python3 sim.py task=pull
Note
Feel free to change the goal position. Pulling will always fail if the goal is in the corner.
python3 reactive_tamp.py task=push_pull multi_modal=True goal="[-3.75, -3.75]"
python3 sim.py task=push_pull
Note
Feel free to change the goal position. The corner positions are [-3.75, -3.75], [3.75, 3.75], [3.75, -3.75], [-3.75, 3.75].
Pick the cube from the table:
python3 reactive_tamp.py -cn config_panda
python3 sim.py -cn config_panda
Note
You can play with the cubes using ASDW keys and keyup, keydown, keyleft and keyright.
Pick the cube from the shelf:
python3 reactive_tamp.py -cn config_panda multi_modal=True cube_on_shelf=True
python3 sim.py -cn config_panda multi_modal=True cube_on_shelf=True
Note
You can play with the cubes using ASDW keys and keyup, keydown, keyleft and keyright.
If you find the code useful, please cite:
@article{zhang2024multi,
title={Multi-Modal MPPI and Active Inference for Reactive Task and Motion Planning},
author={Zhang, Yuezhe and Pezzato, Corrado and Trevisan, Elia and Salmi, Chadi and Corbato, Carlos Hern{\'a}ndez and Alonso-Mora, Javier},
journal={IEEE Robotics and Automation Letters},
year={2024},
publisher={IEEE}
}
- Isaac-mppi: an MPPI implementation that uses IsaacGym as a dynamic model (paper, website, code).
- Biased-mppi: an MPPI implementation whose sampling distribution is informed with ancillary controllers (paper, website, code).
- AIP: an Active Inference planner for decision making (paper, video, code).
We thank the pioneers (Grady Williams, pytorch_mppi, storm) who have paved the way and the future newcomers who will propel MPPI forward!