panda-deep-grasping

Linux Ubuntu Focal (20.04) | ROS Noetic

Installation

Simulation and evaluation are using different verisons of franka_ros and libfranka, so you need two separate workspaces. The simulation workspace requires some extra steps. The location to save and load the replay buffer files, model files and training or evaluation data has to be set manually.

Step 1: Installation for both Evaluation and Simulation

Install ROS Noetic
Create two ROS-Workspaces. One for the simulated environment, one for the real environment. Recommended file structure and workspace names from user folder:
```
panda_deep_grasping
|
+--- simulation_ws
|
+--- real_ws
```
Install MoveIt from packages
sudo apt install ros-noetic-moveit ros-noetic-moveit-visual-tools
Install Gazebo from packages
Install gazebo_ros_packages from packages
sudo apt-get install ros-noetic-gazebo-ros-pkgs ros-noetic-gazebo-ros-control
Install OpenCV from packages
sudo apt install libopencv-dev python3-opencv
Install realsense_ros
sudo apt-get install ros-noetic-realsense2-camera
Install franka_ros from packages
sudo apt install ros-noetic-libfranka ros-noetic-franka-ros
Install tensorflow
pip install tensorflow
Install panda_deep_grasping package by cloning in simulation_ws/src and real_ws/src:
git clone https://git.scc.kit.edu/uhksn/panda-deep-grasping.git

Step 2: Additional steps for Simulation

Following additional steps are necessary for the simulation workspace:

Install panda_moveit_config from source: In simulation_ws/src:
git clone -b noetic-devel https://github.com/vincentmllr/panda_moveit_config.git
Install franka_ros 9.0.0 from source: In simulation_ws/src:
git clone -b 0.9.0 https://github.com/vincentmllr/franka_ros.git
Add Camera to urdf description: Exchange panda_gazebo.xacro file from "franka_ros/franka_description/robots/" with the version in "panda_deep_grasping/description/": In simulation_ws/src:
cd panda-deep-grasping/description && cp panda_gazebo.xacro ../../franka_ros/franka_description/robots/panda_gazebo.xacro && cd ../..

Step 3: Finish Installation

Following commands should be done once after the previous steps:

In workspace folders:
catkin config --extend /opt/ros/${ROS_DISTRO} --cmake-args -DCMAKE_BUILD_TYPE=Release
In src folders:
rosdep install -y --from-paths . --ignore-src --rosdistro noetic
In one workspace folder:
catkin_make
In one workspace folder:
source devel/setup.bash
Has to be done in every new terminal and after catkin_make. If you are only working with one workspace you can automatically source in every new terminal with:
echo "source ~/catkin_ws/devel/setup.bash" >> ~/.bashrc

Startup

If you didnt use the recommended file structure, you have to set the file paths for the replay-buffer-files, the model-files and the training-data-file. You can change those filepaths by changing the according variables at the top of the agent_main, agent_model, agent_buffer, environment_sim and environment_real-files. Additionally, using those variables you can choose whether you want to load an existing model or replay buffer or which version name the files should be saved under.

Training in simulation

cd ~/panda_deep_grasping/simulation_ws ; source devel/setup.bash
roslaunch panda_deep_grasping training.launch

Evaluation in simulation

Change ONLY_ACTOR variable in agent_model.py to true
cd ~/panda_deep_grasping/simulation_ws ; source devel/setup.bash
roslaunch panda_deep_grasping evaluation_sim.launch

Evaluation in real environment

Real Robot black button has to be unlocked (not pressed down)
Make sure camera is connected via USB
Change ONLY_ACTOR variable in agent_model.py to true
cd ~/panda_deep_grasping/real_ws ; source devel/setup.bash
roslaunch panda_deep_grasping evaluation_real.launch

Structure

panda-deep-grasping
|
+--+ description: 3D models for simulation
|  |
|  +---panda_gazebo
|  +---realsense_d435
|  +---cube
|  +---tabletop
|
+--+ launch: files to startup training or evaluation
|  |
|  +--- evaluation_real.launch
|  +--- evaluation_sim.launch
|  +--- training.launch
|
+--- models: trained neural networks
|
+--+ msg: message description for communication over ROS
|  |
|  +--- Action.msg
|  +--- Reward.msg
|  +--- Sample.msg
|  +--- State.msg
|
+--+ replay_buffer_files: saved training samples and images
|  |
|  +--- replay_buffer.txt
|  +--- replay_images
|   
+--+ results
|  |
|  +--- training_data.txt
|
+--+ scripts
|  |
|  + agent_main.py: handles training loop
|  + agent_model.py: neural networks
|
+--+ src
|  |
|  + environment_real.cpp: interface to real environment
|  + environment_sim.cpp: interface to simulated environment
|  + agent_buffer.cpp: saving and retrieving of training samples
|
+--+ srv: service description for communication over ROS
|  |
|  +---EnvironmentStep.srv
|  +---GetAction.srv
|  +---GetBatch.srv
|  +---ResetEnvironment.srv
|  +---Train.srv
|
+--- CMakeLists.txt
+--- package.xml
+--- README_LOCAL.md
+--- README.md

vincentmllr/deep-robot-grasping