JiaheXu/dex_retarget_ros1

Dex Retargeting

Installation

pip3 install -e ".[example]"
# If you do not need to run the examples:
# pip install -e .

Examples

Retargeting from human video

1. Generate the robot joint pose trajectory from our pre-recorded video.

export PYTHONPATH=$PYTHONPATH:`pwd`
python3 example/detect_from_video.py \
  --robot-name allegro \
  --video-path example/data/human_hand_video.mp4 \
  --retargeting-type vector \
  --hand-type right \
  --output-path example/data/allegro_joints.pkl 

This command will output the joint trajectory as a pickle file at the output_path.

The pickle file is a python dictionary with two keys: meta_data and data. meta_data, a dictionary, includes details about the robot, while data, a list, contains the robotic joint positions for each frame. For additional options, refer to the help information. Note that the time cost here includes both the hand pose detection from video, and the hand pose retargeting in single process mode.

python3 example/detect_from_video.py --help

2. Utilize the pickle file to produce a video of the robot

export PYTHONPATH=$PYTHONPATH:`pwd`
python3 example/render_robot_hand.py \
  --pickle-path example/data/allegro_joints.pkl \
  --output-video-path example/data/retargeted_allegro.mp4 \
  --headless

This command uses the data saved from the previous step to create a rendered video.

3. Record a video of your own hand

export PYTHONPATH=$PYTHONPATH:`pwd`
python3 example/capture_webcam.py --video-path example/data/my_human_hand_video.mp4

This command will access your webcam (which should be connected to your computer) and record the video stream in mp4 format. To end video recording, press q on the keyboard.