Authors: Sridhar Pandian Arunachalam, Irmak Guzey, Soumith Chintala, Lerrel Pinto
This repository contains the official implementation of Holo-Dex including the unity scripts for the VR application, robot demonstration collection pipeline, training scripts and deployment modules. The VR application's APK can be found here.
Holo-Dex consists of two phases: demonstration colleciton, which is performed in real-time with visual feedback to VR Headset, and demonstration-based policy learning, which can learn to solve dexterous tasks from a limited number of demonstrations.
The pipeline requires ROS for Server-Robot communication. This Package uses the Allegro Hand and Kinova Arm controllers from DIME-Controllers. This implementation uses Realsense cameras and which require the librealsense API. Also, OpenCV is required for image compression and other purposes.
After installing all the prerequisites, you can install this pipeline as a package with pip:
pip install -e .
You can test if it has installed correctly by running import holodex from a python shell.
Install the VR Application in your Oculus Headset using the APK provided here. To setup the VR Application in your Oculus Headset and enter the robot server's IP address (should be in the same network). The following stream border color codes indicate the following:
- Green - the right hand keypoints are being streamed
- Blue - the left hand keypoints are being stream.
- Red - the stream is paused and gives access to the menu.
To use the Holo-Dex teleop module, open the VR Application in your Oculus Headset. On the robot server side, start the controllers first followed by the following command to start the teleop:
python teleop.py
The Holo-Dex teleop configurations can be adjusted in configs/tracker/oculus.yaml. The robot camera configurations can be adjusted in configs/robot_camera.yaml.
The package also contains an 30 Hz teleop implementation of DIME and you can run it using the following command:
python teleop.py tracker=mediapipe tracker/cam_serial_num=<realsense_camera_serial_number>
All our data can be found in this URL: https://drive.google.com/drive/folders/1PiuqYkG7O1sIxE7YewVkni6ohLuNY7vF?usp=sharing
To collect demonstrations using this framework, run the following command:
python record_data.py -n <demonstration_number>
To filter and process data from the raw demonstrations run the following command:
python extract_data.py
You can change the data extraction configurations in configs/demo_extract.yaml.
You can train encoders using Self-Supervised methods such as BYOL, VICReg, SimCLR and MoCo. Use the following command to train a resnet encoder using the above mentioned SSL methods:
python train_ssl.py ssl_method=<byol|vicreg|simclr|mocov3>
The training configurations can be changed in configs/train_ssl.yaml.
You can also train:
- Behavior Cloning:
python train_bc.py encoder_gradient=true
python train_bc.py encoder_gradient=false
The training configurations can be changed in configs/train_bc.yaml.
This implementation can deploy BC, BC-Rep and INN (all visual) on the robot. To deploy BC or BC-Rep, use the following command:
python deploy.py model=BC task/bc.model_weights=<bc_model-weights>
To deploy INN use the following command:
python deploy.py model=VINN task/vinn.encoder_weights_path=<ssl_encoder_weights_path>
You can set a control loop instead of pressing the Enter key to get actions using the following command:
python deploy.py model=<BC|VINN> run_loop=true loop_frequency=<action_loop_frequency>
To use Holo-Dex's VR side source code vr/Holo-Dex, you need to install Unity along with other dependencies which can be downloaded through Nuget:
- Oculus SDK
- NetMQ
- TextMeshPro
If you use this repo in your research, please consider citing the paper as follows:
@article{arunachalam2022holodex,
title={Holo-Dex: Teaching Dexterity with Immersive Mixed Reality},
author={Sridhar Pandian Arunachalam and Irmak Guzey and Soumith Chintala and Lerrel Pinto},
journal={arXiv preprint arXiv:2210.06463},
year={2022}
}