ShakingBot: Dynamic Manipulation for Bagging

Wuhan Textile University, Wuhan City, Hubei Province, China

Bag manipulation through robots is complex and challenging due to the deformability of the bag. Based on dynamic manipulation strategy, we propose the ShakingBot for the bagging tasks. Our approach utilizes a perception module to identify the key region of the plastic bag from arbitrary initial configurations. According to the segmentation, ShakingBot iteratively executes a set of actions, including Bag Adjustment, Dual-arm Shaking and One-arm Holding, to open the bag. Then, we insert the items and lift the bag for transport. We perform our method on a dual-arm robot and achieve a success rate of 21/33 for inserting at least one item across a variety of initial bag configurations. In this work, we demonstrate the performance of dynamic shaking actions compared to the quasi-static manipulation in the bagging task. We also show that our method generalizes to variations despite the bag's size, pattern, and color.

This repository contains code for training and evaluating ShakingBot in both simulation and real-world settings on a dual-UR5 robot arm setup for Ubuntu 18.04. It has been tested on machines with Nvidia GeForce RTX 2080 Ti.

1 Data Collection
- 1.1 Explanation
- 1.2 Installation
- 1.3 Show Datasets
2 Get Datasets
3 Network Training
- 3.1 Installation and Code Usage
- 3.2 Train ShakingBot
- 3.3 Evaluate ShakingBot

Data Collection

Kinect v2 based data collection code

Explanation

The folder【get_rgb_depth】is used to get the raw data of Kinect V2, RGB image size is 1920×1080,Depth image size of 512×424.
Since RGB images and depth images have different resolutions, they need to be aligned.The folder【colorized_depth】is used to align images.
【final_datasets】is used to processing of images to target resolution,Generate .png and .npy files.
【all_tools】Includes hardware kit for matlab to connect to Kinect V2.

Installation

Requirements

Ubuntu 18.04
Matlab R2020a
Kinect V2

Show Datasets

raw_rgb(1920×1080):

raw_depth(512×424):

align_rgb(512×424):

Get Datasets

Preprocess the images collected in kinect v2

The paint color can be changed into others according to the pattern color of the bag, here is the color of the paint in one of the bags.
```
cd get_datasets
python get_rgb_npy.py
```

processed_rgb(243×255):

Get datasets label
```
python color_dectect.py
```

Network Training

Installation and Code Usage

Make a new virtualenv or conda env. For example, if you're using conda envs, run this to make and then activate the environment:
```
conda create -n shakingbot python=3.6 -y
conda activate shakingbot
```
Run pip install -r requirements.txt to install dependencies.
```
cd network_training
pip install -r requirements.txt
```

Train Region Perception Model

In the repo's root, get rgb and depth map
In the configs folder modify segmentation.json
Train Region Perception model
```
    python train.py
```

Evaluate Region Perception Model

In the repo's root, download the model weights
Then validate the model from scratch with
```
python visualize.py
```

Training details can be viewed in the bag

cd network_training
tensorboard --logdir train_runs/

guningquan/ShakingBot