COMSE 6998 Practical Deep Learning Systems Performance Final Project

Human Activity Recognition Using Tactile Patterns

Using densely deployed flexible pressure sensors to estimate precise human movements.
Built customized data pipeline and multiple lightweight ML/DL models to classify human actions given inputs from our flexible pressure sensors.
Our pressure sensors can be deployed in large scale with relatively low cost.
Our models are computational friendly to edge computing devices.
Human movements that can be recognized:
1. Good Posture
2. Cross-Legged(Right)
3. Cross-Legged(Left)
4. Leaning Right
5. Leaning Left
6. Slouch

./data: folder containing collected data for training, testing and validation
./vis: some visualization result for inputs from pressure sensors
bc2916_6998_project_binary.ipynb:
- deep learning models training, testing and validation pipline for binary classification
bc2916_6998_project_multi_class.ipynb:
- deep learning models training, testing and validation pipline for multiclass classification
data_collection_with_model_finetuning.py:
- script for data collection and XGBoost model finetuning
final_XGB.ipynb:
- XGBoost model training, testing and validation pipeline
pose_prediction.py:
- script running on MCU using exported XGBoost model for final pose prediction
realtime_line_graph.py:
- script for visualizing XGBoost prediction confidence at runtime
sensor_visualization.py:
- script for visualizing pressure map directly from connected pressure sensors

For notebook files, simply following code blocks should work.

Other python scripts can be run directly without input arguments.

However, some of them requires connecting to our custom hardwares including MCU or pressure sensors.

Demo video for human movement classification using pressure sensor
We added up the possibilities of all wrong postures and visualized them out with a line graph in matplotlib
Deep Learning model evaluation result:
XGBoost model evaluation results:
Noisy input data from pressure sensors:
Deep Learning model architecture:
Deep Learning model with Transformer:

Model Quantization, Compression, Distillation
Due to the mechanical principle (Hysteresis effects) of the pressure sensors, LSTM, RNN, Transformers, Vision Transformer future tuning.
Deploying pressure sensors in large scale.
Hosting load balancing and computational heavy deep learning models on cloud.
Teacher-student model between cloud and local edge devices.
Continuously collecting (unlabeled, noisy) data from deployed sensors to address model drifting issues.
More efficient data collection pipeline by leveraging multiple cameras and Open-Pose or similar motion capture techniques.