DIGIT

This codebase allows you:

Collect image frames from DIGIT and annotate circles in each frame.
Save the annotated frame values into a csv file.
Train a baseline MLP model for RGB to Normal mapping.
Generate depth maps in real-time using a fast Poisson Solver.
Estimate 2D object pose using PCA and OpenCV built-in algorithms.

Currently, labeling circles is done manually for each sensor. It can take up to an hour for annotating 30 images.
This codebase has a script that will replace manual labeling and model training process up to 15 mins.(400% faster).
This project is set up in a way that makes it easier to create your own ROS packages later for processing tactile data in your applications.

Visualization

Estimating object pose by fitting an ellipse (PCA and OpenCV):

Depth image point cloud :

Marker movement tracking ( useful for force direction and magnitude estimation):

TODO

Add a Pix2Pix model to generate depth maps from RGB images.
Add an LSTM model for predicting slip from collected video frames.
Add a baseline ResNet based model for estimating total normal force magnitude.

Usage

Change **gel height,gel width, mm_to_pix, base_img_path, sensor :serial_num ** values in rgb_to_normal.yaml file in config folder.

pip install .
cd scripts
- python record.py : Press SPACEBAR to start recording.
- python label_data.py : Press LEFTMOUSE to label center and RIGHTMOUSE to label circumference.
- python create_image_dataset.py : Create a dataset of images and save it to a csv file.
- python train_mlp.py : Train an MLP model for RGB to Normal mapping.

color2normal model will be saved to a separate folder "models" in the same directory as this file.

For ROS, you can use the following command to run the node:

python scripts/ros/depth_value_pub.py
python scripts/ros/digit_image_pub.py

depth_value_pub.py publishes the maximum depth (deformation) value for the entire image when object is pressed. Accuracy depends on your MLP-depth model. digit_image_pub.py publishes compressed RGB images from DIGIT.

lisong-HNU/digit-depth