A vision pipeline for detecting objects from RGBD images and instructing a Franka Emika Research 3 to pick them up through Simulink.
Run this command to create an anaconda environment with the needed dependencies from the environment.yml file. Make sure that the matlabengine package in the environment.yml has the corrct version for the installed Matlab version. The version can be checked here. Anaconda can be installed here.
conda env create -f environment.yml
The Intel RealSense SDK is needed for the communication with the 3D camera. Install from this repo.
For GPU acceleration CUDA needs to be installed. To install it follow this guide.
Run the following command to activate the conda environment created earlier. If you chose another name for the environment replace 'pandavision' with your chosen name.
conda activate pandavision
Run the following commands in the Matlab console.
matlab.engine.shareEngine
matlab.engine.engineName
The last command will return the identifier of the Matlab engine. Copy it over to config.YAML found in the home directory next to 'ENGINE_NAME' in the 'SIMULINK' section.
To perform hand-eye calibration the robot must be active and the Simulink environment running. Make sure that the robot holds an AruCo marker with its end-effector. Also the calibration flag in one of the constant blocks in Simulink must be set to true. Also when closing the end-effector to Simulink check that the constant block responsible for opening and closing it is actually connected since in operating mode opening and closing is handled by the state flow chart. Now, run the following command from the root directory:
python capture_calibration_points.py
The script will move the robotic arm to the predefined coordinates in ./calibration/captures/calibration_points.py asking if the robot arrived after each iteration. Press enter to take a picture with the connected RealSense camera and move on to the next point. The script will terminate after the last point was reached. To calculate the calibration matrix run the following script:
python ./calibration/calibrate.py
If the program throws an error because the marker isn't visible in a picture you can enter the number of the picture to the skip array in the program itself. If too many images are corrupted, calibration won't work. After successful calibration the calibration matrices will automatically be saved and used by the vision pipeline later, no more is needed at this point.
To grab one of the four classes (pen, knife, fork, spoon) run the following command and move through the options displayed in the command line with the arrow keys.
python pipeline.py
You will get the option to use a stock or finetuned model. The finetuned model must be located in the root directory and called 'finetuned_post_own_labels.pt'. If the stock option is used the yolov8x model will automatically be downloaded if it isn't found in the root directory.