This repository contains code to perform bin picking with a RealSense camera, a Dobot Magician robotic arm, and ChatGPT.
To set up the project, follow these steps:
-
Clone the repository and submodules.
git clone --recursive https://github.com/Spritaro/bin_picking_llm.git
or
git clone https://github.com/Spritaro/bin_picking_llm.git git submodule update --init --recursive
-
Create and activate virtual environment using venv.
cd bin_picking_llm python3 -m venv venv source venv/bin/activate
-
Install PyTorch.
pip3 install --upgrade pip setuptools wheel pip3 install torch torchvision
If you are using Jetson, see Installing PyTorch for Jetson Platform and PyTorch for Jetson. Only PyTorch binary is available, so you may need to build torchvision yourself.
-
Install dependencies.
pip3 install --no-build-isolation -e .
If you are using Jetson with JetPack 4.x, install librealsense2 with apt and pyrealsense2 with pip. If JetPack 5.x, you will need to build librealsense2 and pyrealsense2.
-
Download Detic model and install Detic dependencies.
cd third_party/Detic mkdir models cd models wget https://dl.fbaipublicfiles.com/detic/Detic_LCOCOI21k_CLIP_R18_640b32_4x_ft4x_max-size.pth
pip3 install -r requirements.txt cd third_party/Deformable-DETR/models/ops ./make.sh
-
Connect the camera and the robot to the computer.
Permission for the serial device may need to be modified.
sudo chmod a+rw /dev/ttyUSB0
Hand-eye calibration is necessary for bin picking. Follow these steps.
-
Place a checkerboard in front of the camera and the robot. It should be placed in a way that the robotic arm is able to reach all corners of the checkerboard.
-
Run calibration script and launch calibration window. Adjust parameters to reflect the actual checkerboard size.
python3 scripts/calibrate_camera_robot_base.py --column 10 --row 7 --square-size 19.09
-
Calibrate camera base pose by pressing
c
key. A calibration file should be saved tooutputs/camera_base.npy
-
Calibrate robot base pose by following these steps.
- Move the arm to the origin of the checkerboard (shown by axis on the window), then press
Enter
. - From the current position, move the arm in the X axis (red) direction, touch the outmost corner, then press
Enter
. - From the current position, move the arm in the Y axis (green) direction, touch the outmost corner, then press
Enter
. - From the current position, move the arm in the negative X axis direction, touch the outmost corner, then press
Enter
. - A calibration file should be automatically saved to
outputs/robot_base.py
.
- Move the arm to the origin of the checkerboard (shown by axis on the window), then press
-
Quit the script by pressing
q
key.
-
Create OpenAI API Key and set it to the environment variable.
export OPENAI_API_KEY="your_api_key"
-
Run the following script.
python3 scripts/move_robot_with_llm.py
-
Enter prompts such as "Find a Toothbrush and move your arm to its position".