/IMR_Pick_and_Place

6D pose estimation network(DenseFusion) + Panda robot arm simulator(GAZEBO, ROS) pick and place demo

Primary LanguagePython

DenseFusion Pick and Place Demo

About

Pick and Place with 6D pose estimation network (DenseFusion) and Panda robot arm simulator (ROS+GAZEBO)

Environment

  • Ubuntu 16.04
  • ROS Kinetic
  • Gazebo 7.0

Installation: Panda Robot Arm Simulator

  • Basically follow the installation guide on this repo

Install panda simulator

  1. install libfranka ( install from source recommended )
  2. franka-ros v0.6.0 ( install from source )
  3. clone this repo, make sure using the kinetic-devel branch
cd <catkin_ws>/src
git clone https://github.com/justagist/panda_simulator -b kinetic-devel
  1. run ./build_ws.sh from <catkin_ws>/src/panda_simulator

Install python dependencies

  • panda_robot: add this repo to your <catkin_ws> and build it
  • quaternion pip install numpy-quaternion
  • pyquaternion pip install pyquaternion
  • numba (optional) 
     python -m pip install llvmlite==0.31.0
 pip install numba
  • scipy (optional) 
    pip install scipy

ROS world and launch files

  • put IMR_PICK_AND_PLACE.launch under <catkin_ws>/src/panda_simulator/panda_gazebo/launch
  • put IMR_PICK_AND_PLACE.world under your $GAZEBO_RESOURCE_PATH

Some problem you may encounted

  1. catkin build problem

    • this repo uses catkin build instead of catkin_make , you need to delete /build and /devel folder in your <catkin_ws> or simply catkin clean if you're originally using catkin_make to build the package.

  2. franka-ros problem

    • this repo uses franka-ros 0.6.0, check the version before moving on to further step, you may need to reinstall libfranka, follow build from source

    • in the install guide, use catkin build instead of catkin_make

      rosdep install --from-paths src --ignore-src --rosdistro kinetic -y --skip-keys libfranka
catkin build -DCMAKE_BUILD_TYPE=Release -DFranka_DIR:PATH=/path/to/libfranka/build

  3. libfranka problem - problem: apt-get installs a non-working version of libfranka - I use libfranka 0.7.0 build from source, make sure the /common folder is not empty before building libfranka

Demo 1: Simple Pick and Place

  • demo goal: using franka robot arm to grab linemod-dataset-objects with kinect_ros camera attach to the gripper of franka robot arm

    1. Start demo world 
      roslaunch panda_gazebo IMR_PICK_AND_PLACE.launch
    2. Start the object-robot interface 
      python object_interface_server.py
    3. Attach the camera to the robot arm 
      python attach_camera.py
    4. Start camera viewer 
      python img_saver.py
    5. Run pick and place demo 
      python pick_and_place_demo.py

Demo 2:Pick and Place using 6D pose estimation

Step 1: start densefusion inference docker

  1. Put densefusion package under /home
  1. build with dockerfile in /home/DenseFusion
sudo docker run -it -v ~/DenseFusion:/home/DenseFusion df_docker
  1. start the docker
sudo docker run -it --name df_test --runtime=nvidia -v ~/DenseFusion:/home/DenseFusion df_docker

Step 2: run densefusion pick and place demo

  1. Start demo world 
    roslaunch panda_gazebo IMR_DF_PICK_AND_PLACE.launch
  2. Start the object-robot interface 
    python object_interface_server.py
  3. Attach the camera to the robot arm 
    python attach_camera.py
  4. Start camera viewer 
    python img_saver.py
  5. Run pick and place demo 
    python df_pick_and_place_demo.py

Discussion: DenseFusion Prediction Result

  • Rotation predictions are pretty bad
  • However the xyz seems usable

Reference Frame in Gazebo

tags: GAZEBO ROS IMR Documentation