TOF SCANNER

EdgeSense: STM32 Edge AI Texture and Gesture Classifier

  ███████╗██████╗  ██████╗ ███████╗███████╗███████╗███╗   ██╗███████╗███████╗
  ██╔════╝██╔══██╗██╔════╝ ██╔════╝██╔════╝██╔════╝████╗  ██║██╔════╝██╔════╝
  █████╗  ██║  ██║██║  ███╗█████╗  ███████╗█████╗  ██╔██╗ ██║███████╗█████╗  
  ██╔══╝  ██║  ██║██║   ██║██╔══╝  ╚════██║██╔══╝  ██║╚██╗██║╚════██║██╔══╝  
  ███████╗██████╔╝╚██████╔╝███████╗███████║███████╗██║ ╚████║███████║███████╗
  ╚══════╝╚═════╝  ╚═════╝ ╚══════╝╚══════╝╚══════╝╚═╝  ╚═══╝╚══════╝╚══════╝

EdgeSense is a compact, high-performance edge artificial intelligence platform that performs on-device 3D gesture recognition, surface/texture classification, static hand posture detection, and optical obscuration sensing.

Powered by the STM32H533CEU6 (ARM Cortex-M33 running at 250 MHz) and the ST VL53L8CH 8x8 direct Time-of-Flight (dToF) sensor, EdgeSense executes quantized neural networks entirely on-chip in real time with sub-7 ms inference latency.

Key Capabilities

EdgeSense features a runtime multi-model execution engine (pipeline_manager.c) allowing dynamic model switching over USB CDC:

1. 3D Gesture Recognition (8x8 ToF, 15 FPS):

   • Recognizes dynamic directional swipes: SWIPE_LEFT, SWIPE_RIGHT, SWIPE_UP, SWIPE_DOWN, and IDLE.
   • Continuous 15-frame sliding window (1.0 s temporal context) with zero gesture-lockout.
   • Physical temporal motion energy guard to eliminate stationary phantom triggers.
   • Inference latency: ~6.8 ms per inference.

2. Distance-Invariant Surface Classifier (4x4 CNH, 48 Bins):

   • Classifies textures and materials: HARD_FLOOR (tile/wood), CARPET (subsurface scattering), SPECULAR (mirrors/polished metal), and VOID (drop-offs/cliffs).
   • Utilizes ST-aligned Cumulative Normalized Histogram (CNH) peak-centered canonical alignment.
   • Inference latency: ~3.2 ms per inference.

3. Static Hand Posture Classifier (8x8 Depth + Signal Intensity):

   • Identifies static human hand postures: FLAT_HAND, LIKE, DISLIKE, BREAK_TIME, FIST, and NONE.
   • Dual-channel 2D-CNN processing distance geometry and photon reflection intensity simultaneously.
   • Robust to ambient lighting variations and skin tone differences.
   • Inference latency: ~4.1 ms per inference.

4. Optical Obscuration & Smoke Detector:

   • Detects microscopic particles, airborne obscuration, and smoke.
   • Computes multi-zone signal attenuation and ambient light scattering indices in real time.

Hardware Specifications

Repository Structure

EdgeSense/
├── Core/                      # STM32H533 HAL drivers, ISRs, and main loop
├── Drivers/                   # CMSIS and STM32H5xx HAL libraries
├── Middlewares/               # ST X-CUBE-AI runtime v10.2.1 & USB Device stack
├── USB_Device/                # USB CDC Virtual COM Port descriptors & handlers
├── VL53LMZ_ULD/               # Ultra-Lite Driver & CNH Histogram Plugin
├── X-CUBE-AI/                 # Edge AI Pipeline Manager & Active Networks
│   └── App/                   # pipeline_manager, model_gesture, model_posture, model_surface, model_smoke
│
├── Models/                    # Pre-trained production ONNX models & datasets
│   ├── gesture_nn_model.onnx  # 5-class 2D-CNN Gesture Classifier
│   ├── posture_nn_model.onnx  # 6-class 2D-CNN Posture Classifier
│   ├── surface_nn_model.onnx  # 4-class 1D-CNN Surface Classifier
│   ├── gesture_dataset.npz    # Default Gesture training dataset
│   ├── posture_dataset.npz    # Default Posture training dataset
│   ├── surface_dataset.npz    # Default Surface training dataset
│   └── Archive/               # Intermediate training iterations
│
├── Tools/                     # Python Studio, GUIs, and deploy automation
│   ├── Studio/                # 4-in-1 EdgeSense Neural Network Studio
│   │   ├── run_nn_studio.bat  # Double-click to launch Studio GUI
│   │   ├── deploy_to_project.bat # Unified model conversion & firmware build
│   │   ├── edgesense_nn_studio.py
│   │   ├── requirements.txt

Getting Started

1. Prerequisites

• STM32CubeIDE (v2.2.0 or newer) with GNU Tools for STM32 (ARM GCC 14.3+)
• STM32Cube.AI Studio (STEDGEAI-CUBEAI) / ST Edge AI Core (stedgeai) v10.2.1+
• Python 3.10+ (64-bit)
• ST-LINK V2 / V3 Programmer

2. Python Environment Setup

Install the required dependencies for the EdgeSense AI Studio:

cd Tools/Studio
pip install -r requirements.txt

3. Launching EdgeSense Studio

Double-click Tools/Studio/run_nn_studio.bat or run:

python Tools/Studio/edgesense_nn_studio.py

From the Studio, you can:

• View live 8x8 depth heatmaps and 4x4 CNH pulse shapes.
• Record new training datasets for Gestures, Postures, or Surfaces.
• Train PyTorch neural networks with 1 click.
• Deploy trained models to the project via deploy_to_project.bat.

4. Building Firmware via Command Line

make -C Debug all

The compiled binary will be located at Debug/EdgeSense.elf and Debug/EdgeSense.bin.

5. Flashing Firmware

Choose your preferred flashing method:

• USB-C (DFU Mode - No external programmer required):
  1. Press and hold SW2 (BOOT0), press and release SW1 (RESET), then release SW2.
  2. Run:

  Tools/Flashing/flash_firmware_usb.bat

• ST-LINK (SWD Mode): Connect your ST-LINK probe to the J3 header and run:

  Tools/Flashing/flash_firmware.bat

• STM32CubeIDE: Open the project in STM32CubeIDE and click Run or Debug.

Serial Communication & AT Commands

Connect to the USB CDC COM port at 921600 baud, 8-N-1.

License

This project is licensed under the GNU General Public License v3.0. See the LICENSE file for details.