Iterative Residual Policy

This repository contains the source code for the paper Iterative Residual Policy for Goal-Conditioned Dynamic Manipulation of Deformable Objects. This paper has been accepted to RSS 2022.

Cite this work

@inproceedings{chi2022irp,
  title={Iterative Residual Policy for Goal-Conditioned Dynamic Manipulation of Deformable Objects},
  author={Chi, Cheng and Burchfiel, Benjamin and Cousineau, Eric and Feng, Siyuan and Song, Shuran},
  booktitle={Proceedings of Robotics: Science and Systems (RSS)},
  year={2022}
}

Datasets

IRP Rope Dataset [required for eval] (7.63GB)

IRP Cloth Dataset [training only] (938MB)

Pretrained Models

IRP Rope [action + tracking] (914MB)

IRP Cloth [action] (450MB)

Usage

Installation (🖥️ Basic)

A conda environment.yml for python=3.8, pytorch=1.9.0 and cudatoolkit=11.2 is provided.

conda env create --file environment.yml

Please try mambaforge for better dependency conflict resolution.

mamba env create --file environment.yml

Installation (👾 Simulation)

Install Mujoco 2.1.0
Install mujoco-py 2.0.2 and carefully follow instructions.
Install abr_control with Mujoco.

Installation (🦾 Real Robot)

Hardware

Stereolabs ZED 2i Camera
UR5-CB3 or UR5e (RTDE Interface is required)
Millibar Robotics Manual Tool Changer (only need robot side)
3D Print Quick Change Plate to mount the wooden extension stick to EEF.
3/8 inch Square Wooden Dowel
8mm Cotton Rope
Wood screws
Duct tape 😛

Software

Install Zed SDK and pyzed
Install ur_rtde

Evaluation (🖥️ Basic)

Under project root (i.e. irp/), create data folder and download IRP Rope Dataset as well as Pretrained Models. Extract tar files

$ cd data
$ tar -xvf checkpoints.tar
$ tar -xvf irp_rope.zarr.tar

Activate environment

$ conda activate irp
(irp) $

Run dataset evaluation script. Use action.gpu_id to select GPU on multi-GPU systems.

(irp) $ python eval_irp_rope_dataset.py action.gpu_id=0

A log.pkl file will be saved to the ad-hoc output directory created by hydra. Add command-line argument offline=False to enable wandb logging (recommended).

Numbers reported in our paper is generated using this method.

Evaluation (👾 Simulation)

Extract data and checkpoints following basic (required). Install dependencies following sim installation.

Run simulation evaluation script.

(irp) $ python eval_irp_rope_dataset.py action.gpu_id=0

Note that this method has not been extensively tested and is mainly provided for future development. Mujoco might crash due to numerical instability (i.e. Nan), which is better handled in the dataset.

Evaluation (🦾 Real Robot)

Extract data and checkpoints following basic (required). Install dependencies following real installation.

Initialize UR5 robot and write down <robot_ip>. Move robot in teach mode close to joint configuration [-90,-70,150,-170,-90,-90] to prevent unexpected movement.

Run ur5_camera_calibration_app to create homography calibration file to data/calibration.pkl. Use A/D keys to move j2 and W/S keys to move j3. Click on the tip of the wooden extension for calibration.

(irp) $ python ur5_camera_calibration_app.py --ip <robot_ip> -o data/calibration.pkl

Example for calibration result, the red and blue crosses should be fairly close.

Run real eval script.

(irp) $ python eval_irp_rope_real.py

Result and videos will be saved to the hydra ad-hoc output directory.

Training

To train IRP model from scratch:

(irp) $ python train_irp.py

In case of inaccurate tracking, use video_labeler.py to generate tracking labels and train_tracker.py to train tracking model.

real-stanford/irp