/aloha

Primary LanguagePythonMIT LicenseMIT

ALOHA: A Low-cost Open-source Hardware System for Bimanual Teleoperation

Project Website: https://tonyzhaozh.github.io/aloha/

This codebase contains implementation for teleoperation and data collection with the ALOHA hardware. To build ALOHA, follow the Hardware Assembly Tutorial and the quick start guide below. To train imitation learning algorithms, you would also need to install ACT.

Repo Structure

  • config: a config for each robot, designating the port they should bind to, more details in quick start guide.
  • launch: a ROS launch file for all 4 cameras and all 4 robots.
  • aloha_scripts: python code for teleop and data collection

Quick start guide

Hardware selection

We suggest using a "heavy-duty" computer if possible.

In particular, at least 6 USB3 ports are needed. 4 ports for robot connections and 2 ports for cameras. We have seen cases that a machine was not able to stably connect to all 4 robot arms simultaneously over USB, especially when USB hubs are used.

Software selection -- OS:

Currently tested and working configurations:

  • ✅ Ubuntu 18.04 + ROS 1 noetic
  • ✅ Ubuntu 20.04 + ROS 1 noetic

Ongoing testing (compatibility effort underway):

  • 🚧 ROS 2

Software installation - ROS:

  1. Install ROS and interbotix software following https://docs.trossenrobotics.com/interbotix_xsarms_docs/
  2. This will create the directory ~/interbotix_ws which contains src.
  3. git clone this repo inside ~/interbotix_ws/src
  4. source /opt/ros/noetic/setup.sh && source ~/interbotix_ws/devel/setup.sh
  5. sudo apt-get install ros-noetic-usb-cam && sudo apt-get install ros-noetic-cv-bridge
  6. run catkin_make inside ~/interbotix_ws, make sure the build is successful
  7. go to ~/interbotix_ws/src/interbotix_ros_toolboxes/interbotix_xs_toolbox/interbotix_xs_modules/src/interbotix_xs_modules/arm.py, find function publish_positions. Change self.T_sb = mr.FKinSpace(self.robot_des.M, self.robot_des.Slist, self.joint_commands) to self.T_sb = None. This prevents the code from calculating FK at every step which delays teleoperation.

Hardware installation:

The goal of this section is to run roslaunch aloha 4arms_teleop.launch, which starts communication with 4 robots and 4 cameras. It should work after finishing the following steps:

Step 1: Connect 4 robots to the computer via USB, and power on. Do not use extension cable or usb hub.

  • To check if the robot is connected, install dynamixel wizard here

  • Dynamixel wizard is a very helpful debugging tool that connects to individual motors of the robot. It allows things such as rebooting the motor (very useful!), torque on/off, and sending commands. However, it has no knowledge about the kinematics of the robot, so be careful about collisions. The robot will collapse if motors are torque off i.e. there is no automatically engaged brakes in joints.

  • Open Dynamixel wizard, go into options and select:

    • Protocal 2.0
    • All ports
    • 1000000 bps
    • ID range from 0-10

  • Note: repeat above everytime before you scan.

  • Then hit Scan. There should be 4 devices showing up, each with 9 motors.

  • One issue that arises is the port each robot binds to can change over time, e.g. a robot that is initially ttyUSB0 might suddenly become ttyUSB5. To resolve this, we bind each robot to a fixed symlink port with the following mapping:

    • ttyDXL_master_right: right master robot (master: the robot that the operator would be holding)
    • ttyDXL_puppet_right: right puppet robot (puppet: the robot that performs the task)
    • ttyDXL_master_left: left master robot
    • ttyDXL_puppet_left: left puppet robot

  • Take ttyDXL_master_right: right master robot as an example:

    1. Find the port that the right master robot is currently binding to, e.g. ttyUSB0

    2. run udevadm info --name=/dev/ttyUSB0 --attribute-walk | grep serial to obtain the serial number. Use the first one that shows up, the format should look similar to FT6S4DSP.

    3. sudo vim /etc/udev/rules.d/99-fixed-interbotix-udev.rules and add the following line:

      SUBSYSTEM=="tty", ATTRS{serial}=="<serial number here>", ENV{ID_MM_DEVICE_IGNORE}="1", ATTR{device/latency_timer}="1", SYMLINK+="ttyDXL_master_right"

    4. This will make sure the right master robot is always binding to ttyDXL_master_right

    5. Repeat with the rest of 3 arms.

  • To apply the changes, run sudo udevadm control --reload && sudo udevadm trigger

  • If successful, you should be able to find ttyDXL* in your /dev

Step 2: Set max current for gripper motors

  • Open Dynamixel Wizard, and select the wrist motor for puppet arms. The name of it should be [ID:009] XM430-W350
  • Tip: the LED on the base of robot will flash when it is talking to Dynamixel Wizard. This will help determine which robot is selected.
  • Find 38 Current Limit, enter 200, then hit save at the bottom.
  • Repeat this for both puppet robots.
  • This limits the max current through gripper motors, to prevent overloading errors.

Step 3: Setup 4 cameras

  • You may use usb hub here, but maximum 2 cameras per hub for reasonable latency.

  • To make sure all 4 cameras are binding to a consistent port, similar steps are needed.

  • Cameras are by default binding to /dev/video{0, 1, 2...}, while we want to have symlinks {CAM_RIGHT_WRIST, CAM_LEFT_WRIST, CAM_LOW, CAM_HIGH}

  • Take CAM_RIGHT_WRIST as an example, and let's say it is now binding to /dev/video0. run udevadm info --name=/dev/video0 --attribute-walk | grep serial to obtain it's serial. Use the first one that shows up, the format should look similar to 0E1A2B2F.

  • Then sudo vim /etc/udev/rules.d/99-fixed-interbotix-udev.rules and add the following line

    SUBSYSTEM=="video4linux", ATTRS{serial}=="<serial number here>", ATTR{index}=="0", ATTRS{idProduct}=="085c", ATTR{device/latency_timer}="1", SYMLINK+="CAM_RIGHT_WRIST"

  • Repeat this for {CAM_LEFT_WRIST, CAM_LOW, CAM_HIGH} in additional to CAM_RIGHT_WRIST

  • To apply the changes, run sudo udevadm control --reload && sudo udevadm trigger

  • If successful, you should be able to find {CAM_RIGHT_WRIST, CAM_LEFT_WRIST, CAM_LOW, CAM_HIGH} in your /dev

At this point, have a new terminal

conda deactivate # if conda shows up by default
source /opt/ros/noetic/setup.sh && source ~/interbotix_ws/devel/setup.sh
roslaunch aloha 4arms_teleop.launch

If no error message is showing up, the computer should be successfully connected to all 4 cameras and all 4 robots.

Trouble shooting

  • Make sure Dynamixel Wizard is disconnected, and no app is using webcam's stream. It will prevent ROS from connecting to these devices.

Software installation - Conda:

conda create -n aloha python=3.8.10
conda activate aloha
pip install torchvision
pip install torch
pip install pyquaternion
pip install pyyaml
pip install rospkg
pip install pexpect
pip install mujoco==2.3.7
pip install dm_control==1.0.14
pip install opencv-python
pip install matplotlib
pip install einops
pip install packaging
pip install h5py

Testing teleoperation

Notice: Before running the commands below, be sure to place all 4 robots in their sleep positions, and open master robot's gripper. All robots will rise to a height that is easy for teleoperation.

# ROS terminal
conda deactivate
source /opt/ros/noetic/setup.sh && source ~/interbotix_ws/devel/setup.sh
roslaunch aloha 4arms_teleop.launch

# Right hand terminal
conda activate aloha
cd ~/interbotix_ws/src/aloha/aloha_scripts
python3 one_side_teleop.py right

# Left hand terminal
conda activate aloha
cd ~/interbotix_ws/src/aloha/aloha_scripts
python3 one_side_teleop.py left

The teleoperation will start when the master side gripper is closed.

Example Usages

To set up a new terminal, run:

conda activate aloha
cd ~/interbotix_ws/src/aloha/aloha_scripts

The one_side_teleop.py we ran is for testing teleoperation and has no data collection. To collect data for an episode, run:

python3 record_episodes.py --dataset_dir <data save dir> --episode_idx 0

This will store a hdf5 file at <data save dir>. To change episode length and other params, edit constants.py directly.

To visualize the episode collected, run:

python3 visualize_episodes.py --dataset_dir <data save dir> --episode_idx 0

To replay the episode collected with real robot, run:

python3 replay_episodes.py --dataset_dir <data save dir> --episode_idx 0

To lower 4 robots before e.g. cutting off power, run:

python3 sleep.py