This project is a demonstration of hand posture, motion recognition using VL53L8 ToF sensors and X-CUBE-AI.
Projects referenced STM32 Model Zoo's handposture and STSW-IMG035_F401 Gesture sample code.
Development targets are stm32F769i disco board Nucleo F401RF board and X-NUCLEO-53L8A1 board.
Nucleo F401RE connected to X-NUCLEO-53L8A1 board, receives VL53L8 ToF sensor data and recognizes hand posture, motions through CNN model processing using X-CUBE-AI and GesturesMZ library in Gestures sample code.
Processed hand motion values are transmitted to STM32F769I-DISCO board via Uart com, and results are displayed GUI.
There are three menus in GUI.
- Recognizes posture of hand(Using X-CUBE-AI)
- Recognizes hand height and heigh(Using GesturesMZ lib)
- Recognizes position of hand(Using GesturesMZ lib)
GUI is built with Touchgfx, and you can move menu with touch of screen.
--Version--
STM32CUBE F7 v1.17.2
STM32CUBE MX 6.11.1
STM32CUBE IDE 1.15.1
X-CUBE-AI 9.0.0
TouchGFX 4.24
STM32_AI_HandPosture_Demo
├─ NucleoF401
│ ├─ .ai
│ ├─ .cproject
│ ├─ .mxproject
│ ├─ .project -------------------------------------------> Project file
│ ├─ .settings
│ ├─ CNN2D_ST_HandPosture_8classes.h5 -------------------> STM32 Model Zoo CNN Model
│ ├─ Core -----------------------------------------------> Main App
│ │ ├─ Inc
│ │ └─ Src
│ ├─ Drivers --------------------------------------------> BSP, HAL Driver
│ │ ├─ BSP
│ │ │ ├─ Components
│ │ │ │ └─ VL53LMZ -----------------------------------> VL53LX ToF BSP
│ │ │ ├─ platform.c
│ │ │ └─ platform.h
│ │ ├─ CMSIS
│ │ └─ STM32F4xx_HAL_Driver
│ ├─ GesturesMZ -----------------------------------------> GestureMZ lib
│ ├─ HandPosture_Demo_F4 Debug.launch
│ ├─ HandPosture_Demo_F4.ioc ----------------------------> CubeMX .ioc file
│ ├─ Middlewares
│ │ └─ ST
│ │ └─ AI -------------------------------------------> X-CUBE-AI middleware
│ ├─ STM32F401RETX_FLASH.ld
│ ├─ STM32F401RETX_RAM.ld
│ ├─ X-CUBE-AI
│ └─ startup
├─ README.md
==================================================================================
├─ STM32F769I_DISCO
├─ .extSettings
├─ .mxproject
├─ Core -------------------------------------------------> Main App
│ ├─ Inc
│ └─ Src
├─ Drivers ----------------------------------------------> BSP, HAL Driver
│ ├─ BSP
│ │ └─ Components
│ │ ├─ Common
│ │ ├─ ft6x06
│ │ ├─ mx25l512
│ │ └─ otm8009a
│ ├─ CMSIS
│ └─ STM32F7xx_HAL_Driver
├─ HandPosture_Demo_F7.ioc ----------------------------> CubeMX .ioc file
├─ LIBJPEG
├─ Middlewares
│ ├─ ST
│ │ └─ touchgfx -------------------------------------> TouchGFX middleware
│ └─ Third_Party
│ ├─ FreeRTOS
│ └─ LibJPEG
├─ STM32CubeIDE
│ ├─ .cproject
│ ├─ .project -----------------------------------------> Project file
│ ├─ .settings
│ ├─ Application -----------------------> Empty dir, Use as a project link
│ ├─ Drivers ---------------------------> Empty dir, Use as a project link
│ ├─ HandPosture_Demo_F7 Debug.launch
│ ├─ Middlewares -----------------------> Empty dir, Use as a project link
│ ├─ STM32F769I_DISCO.launch
│ ├─ STM32F769NIHX_FLASH.ld
│ └─ STM32F769NIHX_RAM.ld
├─ TouchGFX
│ ├─ App
│ ├─ ApplicationTemplate.touchgfx.part
│ ├─ HandPosture_Demo.touchgfx -------------------------> TouchGFX designer file
│ ├─ MATERIAL-ICONS-LICENSE
│ ├─ application.config
│ ├─ assets --------------------------------------------> GUI image and fonts
│ ├─ build
│ ├─ config
│ ├─ generated -----------------------------------------> File generated by TouchGFX designer
│ │ ├─ gui_generated
│ │ ├─ images
│ │ ├─ simulator
│ │ ├─ texts
│ │ └─ videos
│ ├─ gui -----------------------------------------------> GUI App code
│ │ ├─ include
│ │ └─ src
│ ├─ simulator
│ ├─ target.config
│ └─ target
├─ changelog.txt
└─ readme.md
AI model uses a pre-trained CNN 2D model in STM32 Model Zoo.
(ref. https://github.com/STMicroelectronics/stm32ai-modelzoo/tree/main/hand_posture)
- Run .project file in F401RE and F769I dir to add a project to CubeIDE.
(Merging may be necessary due to differences in program versions.) - Build project. (Target: Nucleo-F401RE, STM32F769i disco board)
- Connect target board and RUN it.
- Connect X-NUCLEO-53L8A1 board to Nucleo-F401RE and Connect UART line STM32F769I DISCO board as shown in picture below.
- Press desired menu in GUI to move screen, and then detect hand gesture on sensor, result is output.
1. Hand Posture
Recognize a total of 7 hand motions (Flat, Fist, Like, Dislike, Love, Time, Cross).
Video.
https://github.com/user-attachments/assets/0ec3457b-cf05-4496-bd4a-7ae7c6f7803c
2. Light Dim Control
Change Dimming from 0 to 100 depending on hand height of sensor (up to about 30 cm).
Video.
https://github.com/user-attachments/assets/426a58dd-b0b9-40c4-8a46-94bd820900e8
3. Select Target
Select 0-360 degree menu depending on position of hand of sensor.
Video.
https://github.com/user-attachments/assets/3dfbdc69-15ec-4241-8792-f95f8ef60050
- Due to fast sensor operation speed, communication loss between boards occurs occasionally.
- I don't know why, syscalls.c file disappears when code is regenerated through CubeMX in NucleoF401 project.
- This project is intended for personal study and may be of low quality. 😟😟