Multi-Modal MPPI and Active Inference for Reactive Task and Motion Planning

Paper | Website | Video

Repository for reactive task and motion planning making use of active inference for symbolic planning, and a new multi-modal MPPI for motion planning. Rollouts are evaluated in IsaacGym, a parallelizable physics simulator.

Installation

First, clone the repo in your folder and create the conda environment.

cd <project_folder>
git clone https://github.com/tud-amr/m3p2i-aip.git

conda create -n m3p2i-aip python=3.8
conda activate m3p2i-aip

This project requires the source code of IsaacGym. Check for the prerequisites and troubleshooting. Download it from https://developer.nvidia.com/isaac-gym, unzip and paste it in the thirdparty folder. Move to IsaacGym and install the package.

cd <project_folder>/m3p2i-aip/thirdparty/IsaacGym_Preview_4_Package/isaacgym/python
pip install -e.

Then install the current package by:

cd <project_folder>/m3p2i-aip
pip install -e.

Now you are ready to test an example file, where you can drive the robot around with ASDW keys.

cd <project_folder>/m3p2i-aip/examples
python3 example_key.py

Run the scripts

If you want to test the TAMP framework, you will need two instances of Isaac Gym, one for throwing the rollouts and deriving the optimal solution, and one for updating the "real system". Please run the commands below in two terminals from the scripts folder with activated python environment.

Run this terminal first:

cd <project_folder>/m3p2i-aip/scripts
conda activate m3p2i-aip
python3 reactive_tamp.py

Then run the second terminal:

cd <project_folder>/m3p2i-aip/scripts
conda activate m3p2i-aip
python3 sim.py

Specifically, you can test the following:

Navigation among obstacles

python3 reactive_tamp.py task=navigation goal="[-3, 3]"

python3 sim.py

Note

Feel free to change the goal position.

Push only using MPPI

python3 reactive_tamp.py task=push goal="[-1, -1]"

python3 sim.py

Note

Feel free to change the goal position. Pushing will always fail if the initial position is in the corner.

Pull only using MPPI

python3 reactive_tamp.py task=pull goal="[0, 0]"

python3 sim.py task=pull

Note

Feel free to change the goal position. Pulling will always fail if the goal is in the corner.

Multi-modal push pull using M3P2I

python3 reactive_tamp.py task=push_pull multi_modal=True goal="[-3.75, -3.75]"

python3 sim.py task=push_pull

Note

Feel free to change the goal position. The corner positions are [-3.75, -3.75], [3.75, 3.75], [3.75, -3.75], [-3.75, 3.75].

Reactive pick using MPPI

Pick the cube from the table:

python3 reactive_tamp.py -cn config_panda

python3 sim.py -cn config_panda

Note

You can play with the cubes using ASDW keys and keyup, keydown, keyleft and keyright.

Multi-modal pick using M3P2I

Pick the cube from the shelf:

python3 reactive_tamp.py -cn config_panda multi_modal=True cube_on_shelf=True

python3 sim.py -cn config_panda multi_modal=True cube_on_shelf=True

Note

You can play with the cubes using ASDW keys and keyup, keydown, keyleft and keyright.

Cite

If you find the code useful, please cite:

@article{zhang2024multi,
  title={Multi-Modal MPPI and Active Inference for Reactive Task and Motion Planning},
  author={Zhang, Yuezhe and Pezzato, Corrado and Trevisan, Elia and Salmi, Chadi and Corbato, Carlos Hern{\'a}ndez and Alonso-Mora, Javier},
  journal={IEEE Robotics and Automation Letters},
  year={2024},
  publisher={IEEE}
}

Related works

Isaac-mppi: an MPPI implementation that uses IsaacGym as a dynamic model (paper, website, code).
Biased-mppi: an MPPI implementation whose sampling distribution is informed with ancillary controllers (paper, website, code).
AIP: an Active Inference planner for decision making (paper, video, code).

Acknowledgment

We thank the pioneers (Grady Williams, pytorch_mppi, storm) who have paved the way and the future newcomers who will propel MPPI forward!