PBTS-2D: Pose-Based Tactile Servo Control (2D)

A Python library for tactile servo control using pose models built from convolutional neural networks for optical tactile sensors (TacTip, DigiTac, DIGIT)

Methods described in
DigiTac: A DIGIT-TacTip Hybrid Tactile Sensor for Comparing Low-Cost High-Resolution Robot Touch
N Lepora, Y Lin, B Money-Coomes, J Lloyd (2022) IEEE Robotics & Automation Letters
https://arxiv.org/abs/2206.13657.pdf
https://lepora.com/digitac

Data and models use the (x, y, z, rz) components of pose, suitable for 4DoF robot arms. Here a Dobot MG400 is used.

Installation

To install the package on Windows or Linux, clone the repository and run the setup script from the repository root directory:

pip install -e .

Code was developed and run from Visual Studio Code in Windows but has Unix compatibility.

Examples

Some examples that demonstrate how to use the library are included in the \examples directory.

E.g. to test the Dobot MG400 robot

python mg400_robot_test.py

Requirements

Needs installation of these packages

Common Robot Interface (CRI) fork for use with Dobot Desktop Robots https://github.com/nlepora/cri

Video Stream Processor (VSP) fork at https://github.com/nlepora/vsp

Also needs Tensorflow 2; tested with
tensorflow==2.30
tensorflow-gpu==2.30

Some functionality also needs hyperopt; tested with hyperopt==0.2.5

Workflow

Pose Models 2D

Check the installation with \examples
Collect the 2d edge or 2d surface tactile data with scripts in \collect
Process the data into training and test sets
Train the pose models with scripts in \train: (a) single model; (b) optimize model hyperparameters
Test the pose models with scripts in \test: (a) single model; (b) batch of models from optimizer

Note: all code requires an environment variable DATAPATH to the data directory.

Tactile Servoing 2D

Assumes you have already used Pose Models 2D to construct the models or downloaded a repository of data and models.
0. Optional: check the installation with \examples

New experiments:

Run servo_edge2d or servo_surface2d. The poses will be saved in predictions.csv. The data will be saved in frames_bw

Existing data:
2. You can replay previously collected trajectories either on the physical robot (MG400 controller) or virtually (dummy controller)
3. Optionally you can visualize the contour and tactile images during replay (saved as contour.mp4 or frames.mp4)
4. Optionally you can replay to use a camera to video the experiment.

Note: all code requires an environment variable DATAPATH to the data directory.

Repository structure

\pose_models_2d
\collect
collect_edge2d.ipynb - Notebook to collect data on 2d edge stimulus (Workflow step 1)
collect_surface2d.ipynb - Notebook to collect data on 2d surface stimulus (alternative Workflow step 1)
process.ipynb - Notebook to partition into training and test data sets (Workflow step 2)
\examples
frames_robot_test.py - Script to check sensor is collecting frames of data
mg400_robot_test.py - Script to check MG400 movement commands are working
\lib
\models
cnn_model.py - Class for training/testing/predicting using a ConvNet on tactile images
dummy_model.py - Class for test purposes
common.py - common functions used in repository
\test
test2d_cnn.ipynb - Notebook to test a trained model (Workflow step 4)
test2d_cnn_batch.py - Script to test a batch of trained models (alternative Workflow step 4)
\train
train2d_cnn.ipynb - Notebook to train a model (Workflow step 3)
train2d_cnn_opt.py - Script to optimize hyperparameters over trained models (alternative Workflow step 3)

\tactile_servoing_2D
\examples
cnn_robot_test - Script to check data-to-model throughflow is working
\lib
\visualise
plot_contour2d - Class to plot in real-time the servo control trajectory
plot_frames2d - Class to plot in real-time the captured tactile images
common_camera.py - Library of functions (home, control) customized to video the robot
common.py - Library of functions (home, control) for the robot
\servo
replay_camera.py - Replay a saved servo control run while videoing the robot (Workflow step 4)
replay.py - Replay a saved servo control run (Workflow steps 2 & 3)
servo_edge2d.ipynb - Notebook to collect data on 2d edge stimulus (Workflow step 1)
servo_surface2d.ipynb - Notebook to collect data on 2d surface stimulus (alternative Workflow step 1)
\videos
batch_run.py - Script to run multiple offline result scripts
make_video.py - Script to prepare a results video for a servo control run

Precollected data, models and servoing data

Data, models and servo control runs for use with this code are available in the data repository tactile-servoing-2d-dobot https://doi.org/10.5523/bris.110f0tkyy28pa2joru2pxxbrxd

There are 2 types of tactile servo control:

servo_edge2d: around a horizontal edge of a planar object
servo_surface2d: around a vertical wall of a planar object

Available for 4 sensors:

digit: DIGIT sensor with GelSight elastomeric skin
digitac: TacTip version of the DIGIT sensor
tactip-127: Hemispherical TacTip (127 pin version; 40mm dia.)
tactip-331: Hemispherical TacTip (331 pin version; 40mm dia.)

Note: servo_surface2d only available for TacTip-type sensors.

Papers

The methods and data are described and used in this paper

DigiTac: A DIGIT-TacTip Hybrid Tactile Sensor for Comparing Low-Cost High-Resolution Robot Touch
N Lepora, Y Lin, B Money-Coomes, J Lloyd (2022) IEEE Robotics & Automation Letters
https://arxiv.org/abs/2206.13657.pdf
https://lepora.com/digitac

nlepora/pbts-2d