Helping hand

Helping Hand - M5 Release Candidate

Course: CEN4908C
Project: Helping Hand
Target milestone: M5 Release Candidate Due: October 28, 2026
Last updated: October 28, 2026

Helping Hand is an American Sign Language (ASL) learning glove we are developing for CEN4908C, Computer Engineering Senior Design. The glove measures finger bend and wrist motion. An ESP32 processes the sensor readings and sends them over Bluetooth Low Energy (BLE) to our Android Flutter app, where users can practice signs, see live feedback, record gestures, and save their progress.

This repository contains our firmware, app, model-training code, hardware documentation, and the KiCad files for our PCB design, which is still in progress.

• Repository: https://github.com/GG1627/helping-hand
• Release candidate submission snapshot: ff6a9ab
• Historical beta application snapshot: d581d19
• Historical alpha validation commit: cce9d38
• Android package: com.example.flutter_app
• Firebase project: helping-hand-83137
• Demo: https://www.youtube.com/shorts/qQmO1KYz998
• Team: Brian, Gael, Kali, and Srinitha

Current development handoff (November 6): Software preparation for the repaired glove is ready for physical calibration and real data collection. Start with the glove/model session handoff for completed changes, current model contracts, verification limits, and the next steps once assembly is done. The milestone dates and historical evidence below remain labeled for their original submissions.

Milestone Overview

The alpha results below are from commit cce9d38. Earlier beta work is documented in docs/BETA_BUILD_STATUS.md. The original assignments are in M4-Beta-Build.md and T2-Beta-Test-Plan.md. Our beta work outline and test plan are available in output/pdf/M4_Beta_Build_Work_Outline.pdf and output/pdf/T2_Beta_Test_Plan.pdf.

Our alpha build connected the following parts of the system:

Glove sensors → ESP32 static classifier → BLE packets → Flutter feedback → saved local progress → Firestore sync

As of October 27, about two of our flex sensors were broken, they were reading as flexed when they were not. We believe this is due to age. We ordered 5 Adafruit Short Flex Sensor #1070 replacements and an ICM-20948 nine-axis IMU #4554. On November 5, Gael confirmed that all required parts have arrived, including the replacement flex sensors, ICM-20948, and piezo elements. Sensor replacement, installation, and calibration remain unconfirmed. Firmware support for the ICM-20948 and AK09916 magnetometer has now been added, while keeping the older sensor path. The November 5 recording update adds optional magnetic data to BLE and CSV/JSON exports as well as serial diagnostics; see the recording notes. The ICM-20948 integration reference covers wiring, calibration, fusion, current driver limitations, and recognition experiments for the received nine-axis board. Developer Mode now includes a guided calibration workflow for measured flex ranges, stationary gyro bias, axis checks, profile export, and profile-linked raw recordings. Applying calibration to inference remains a separate reviewed model/firmware update. The sequence preprocessing guide documents longer-window and 11/14-feature candidate preparation, strict no-trim checks, and Python/Flutter parity. The deployed word model remains 96 × 11 until a replacement is trained and validated. Orientation fusion and new-board testing are still pending. The BLE word-model inputs remain the existing six motion values. We have also received ten piezo buzzers: one for each finger and five in an X on the back of the hand. The firmware now sets up all ten buzzer pins and can play a timed PWM tone on one buzzer at a time without blocking the main loop. We still need to review the circuit, decide how each cue should work, and test it on the glove. The app and classifier do not trigger the buzzers yet.

Since the October 6 demo, we have continued working on the KiCad schematic, planning the board layout, and looking into ways to improve the TCN. The PCB layout is not finished, and we have not measured an improvement in model accuracy from this work. More details are in the October 27 status notes. We have also begun planning on how to sew in the flex sensors and the 9-axis IMU into the glove using conductive thread and have practiced making small circuits with conductive thread to get a feel for it before adding it to our project. We plan to use multiple layers of fabric (like a PCB) to ensure that we do not short any connections.

The app includes Record Signs and a trained TCN for Hello, Please, and Yes. Users start an attempt, perform the sign, and finish the attempt. The phone runs the model locally and saves completion when the prediction meets the matching and confidence requirements. Recorded replay checks passed on the Android emulator. Fresh glove testing is Not run. See the training notes and integration guide.

Release Candidate Implementation

The RC keeps the full beta feature set and adds a custom PCB design:

1. Accounts: Firebase email/password account creation, sign-in, password reset, and sign-out.

• The app opens directly to sign-in/sign-up.
• An existing anonymous session is linked to the new account so its cloud progress is kept.

2. User-scoped progress:

• Local progress files and the Firestore document users/{uid}/progress/current are scoped to the signed-in Firebase user.
• Firebase Authentication stores credentials. Raw glove recordings stay in the CSV/JSON export workflow and are never uploaded to Firebase.

3. Static recognition: the recorded 34-class model is the default. The original 36-class (A–Z, 0–9) model is kept as a regression baseline, along with the stable-prediction completion rule.

4. Dynamic word recognition: TCN inference with saved completion for Hello, Please, and Yes.

5. Custom PCB (new in M5): KiCad schematic work and board-layout planning for the ESP32-S3 and 15 resistors in the flex-sensor and piezo circuits. The board is not yet routed or fabricated.

6. Process documentation:

• BETA_BUILD_PLAN.md

Current Release Candidate Learning Flow

1. The app opens to login/signup. Login, signup, and Home use text-based Helping Hand branding, and the learning screens are styled to match.
2. Alphabet and Numbers are target pickers. 3.Words is an active vocabulary picker.
3. Tapping a letter, number, or word opens a separate live-practice page. Back returns to the picker. The pickers themselves contain no practice. Letters and numbers: a target is completed when a matching prediction at ≥80% confidence stays stable for at least 750 ms and five packets. Progress is saved per account.
4. Hello, Please, Yes:

• Tap Start attempt, perform the whole sign, then tap Finish sign.
• Completion requires the predicted word to match the target at ≥80% confidence.

6. Other words: the pages remain available with recognition marked unavailable.

M5 Work and Rubric Status

Historical Beta Feature Status

Historical Alpha Feature Status

Historical Beta Rubric Traceability

Historical Alpha Rubric Traceability

Architecture

flowchart LR
    FLEX[Five flex sensors] --> ESP[Adafruit Feather ESP32-S3]
    IMU[IMU accelerometer + gyroscope] --> ESP
    MODEL[Recorded 34-class static MLP] --> ESP
    ESP -->|BLE notifications| BLE[Flutter BLE service]
    BLE --> LIVE[Live telemetry and learner feedback]
    LIVE --> TRACKER[Stable-prediction validator]
    TRACKER --> PROGRESS[Progress repository]
    PROGRESS --> LOCAL[App-private local JSON]
    PROGRESS -->|Firebase email/password Auth| FIRESTORE[Cloud Firestore]
    BLE --> RECORD[Record Signs workflow]
    RECORD --> FILES[App-private CSV + JSON export]
    FILES --> BACKEND[Validation and sequence preprocessing]
    BACKEND --> WORDMODEL[Selected real-recording TCN]
    WORDMODEL --> WORDPRACTICE[Flutter complete word attempts]
    BLE --> WORDPRACTICE
    WORDPRACTICE --> PROGRESS

Embedded Runtime

The firmware runs on the Adafruit Feather ESP32-S3 in ESP32/src/main.cpp. It reads five flex sensors on A0–A4 and communicates with the IMU over I2C (SDA=3, SCL=4). imu_sensor.cpp now checks both 0x68 and 0x69 for supported MPU-style parts or the new ICM-20948. The ICM path uses its own banked register map, sets +/-2 g and +/-250 degrees/second ranges, and reads the AK09916 magnetometer through I2C bypass. Compass data is checked for readiness and overflow and reported in uncalibrated microteslas through serial diagnostics. A read failure in the motion sensor marks it offline and enables discovery retries. If the compass is unavailable, acceleration/gyro streaming can still continue.

BLE still sends the existing ax/ay/az in g and gx/gy/gz in degrees/second. The current word model still uses five flex and six motion inputs; the new magnetic readings do not change its input shape. Magnetic calibration, axis alignment, fusion, and recognition on the replacement hardware still need testing. The firmware changes are checked against the TDK datasheet.

The default static model is the recorded 34-class MLP; the original synthetic 36-class model is kept for regression checks. Firmware sends predictions and confidence over BLE as HelpingHand-Glove and resumes advertising after a disconnect. The configured stream rate is 40 Hz (adjustable to 25–50 Hz). Phone-side rate and packet loss still need measurement.

The planned PCB schematic includes the ESP32-S3 and 15 resistors for the flex and piezo circuits. Placement, routing, fabrication, and board testing are not complete. The proposed sewn glove uses five fabric layers total: inner liner, first routing fabric, insulating separator, second routing fabric, and outer cover. This separates two routing levels; final routing and movement tests must confirm whether more local insulation is needed. See M5_Work_Outline.md for the assembly plan and references.

Host-side simulated-I2C regression checks passed; run python3 ESP32/test/imu_host_test.py with Python 3 and g++. The full build attempt timed out during installation of the missing tool-openocd-esp32 package, so board compilation is not confirmed. New IMU bring-up, compass calibration, and sewn-glove testing are Not run.

The buzzers are outputs. piezo.cpp sets all ten pins LOW before waiting for the serial connection. startPiezo(output, frequencyHz, durationMs) plays one tone at 50% duty cycle. updatePiezos() stops it from the main loop when its time is up, and stopPiezos() cancels it early.

The code rejects requests before setup, while another buzzer is active, or with an invalid output, frequency, or duration. It accepts 20–20,000 Hz and 1–5,000 ms; these are code limits, not confirmed ratings for our buzzers. One LEDC channel is used for the selected pin, so only one buzzer can play at a time. We still need to choose a frequency for the actual piezo and driver. The buzzers stay off at startup and are not connected to app/classifier cues yet.

These pins come from the resistor names and board labels in pcb/pcb.kicad_sch, checked against the Feather pin definitions. Left, right, center, up, and down describe the buzzer positions; we have not tested them as directional cues yet. SCK/MISO/MOSI are used as buzzer outputs here, so they cannot also be used for SPI at the same time. Compile-time checks catch repeated pins and conflicts with the flex or I2C pins. Physical buzzer output, how the cues feel/sound, driver suitability, and timing while BLE is running are Not run.

Flutter Application

The Android Flutter app is in flutter_app/.

• Shared BLE service: owns scanning, connection, notification subscription, packet parsing, and reconnect behavior. Learner practice, Record Signs, and BLE Testing all consume the same packet stream.
• Authentication: Firebase email/password accounts gate entry to the app.
• Letter/number practice: tapping a target opens its practice page. Selecting a target does not mark it learned. The app requires a stable classifier match before saving completion, and shows the target, current prediction, confidence, connection state, hold progress, and success/retry messages.
• Word practice: the learner delimits each attempt with Start attempt / Finish sign. The TCN classifies the complete attempt locally on the phone.

Local and Firebase Persistence

Progress uses a local-first repository design:

• The app loads the signed-in user's progress file from app-private application-support storage.
• Progress is written locally before cloud synchronization begins.
• Firebase initializes on demand rather than before runApp().
• Firebase email/password authentication provides the user identity. The alpha used anonymous authentication.
• The progress document syncs at users/{uid}/progress/current.
• Firestore rules allow an authenticated user to read and write only their own fixed progress document. Collection listing and document deletion are denied.
• If Firebase is unavailable, the app continues with its local copy and offers a retry action.

The document contains only:

schema version learned letters learned numbers completed exercise identifiers an update timestamp

Raw glove recordings are never uploaded to Firebase. The UI exposes loading, saved locally, syncing, synced, offline/local-only, and failure states. Malformed local state falls back to a safe empty representation rather than crashing.

Word-Recording and ML Foundation

Record Signs creates labeled trials using the word-sequence-v1 contract.

• Word captures use Stop → review → Save trial / Discard trial.
• Each saved row keeps:
  • the exact BLE packet
  • parsed timestamps and sequence number
  • five flex readings and six IMU readings
  • label, pseudonymous signer ID, orientation, and trial metadata
• Sessions stay app-private until exported as CSV/JSON through the Android share sheet.

The backend can:

• validate columns, values, labels, metadata, sequence order, timestamps, rate, and sensor ranges;
• report malformed packets and gaps without repairing raw evidence;
• resample trials and create fixed-length windows;
• make deterministic splits that keep each trial and signer within one split;
• fit standardization on training data only;
• evaluate top-1 and top-5 accuracy, macro F1, confusion matrices, and signer/orientation groups;
• build and smoke-test TCN, CNN-GRU, and CNN-LSTM candidates with float TFLite conversion.

Synthetic fixtures are software-test data only and are marked non-reportable. The deployed TCN currently covers Hello, Please, and Yes. It was trained on recordings from a single wearer in a neutral orientation and has no rest/unknown class.

Application Evidence

These updated screenshots show our sign-in and account screens, Home, learning pages, and developer tools. Record Signs and BLE Testing show the disconnected state. These images show the interface, not a new glove-recognition test.

Sign in	Create account	Home
Alphabet	Numbers	Words
Record Signs	BLE Testing

Earlier app screenshots

The screenshots below are from the earlier Android emulator build. The glove was disconnected while we prepared the next hardware iteration. The Dashboard shows the earlier app reaching its Synced progress-storage state.

Start	Dashboard and sync	Alphabet practice
Number practice	Record Signs	BLE Testing

Glove Pictures

Our updated glove assembly is shown below. The photo documents the hardware arrangement; the planned sewn assembly, nine-axis IMU integration, and ten-piezo feedback system still need completion and testing.

Historical Physical Integration Evidence

These short videos were recorded at the end of the previous development semester. They establish that real flex-sensor changes reached the physical Android phone over BLE and changed the live hand visualization. The team's end-of-semester integration testing also displayed the static classifier's predicted sign and confidence on the phone.

• Bench flex sensor streaming to the phone (6 seconds)
• Wearable glove connected to the phone (8 seconds)

The videos are historical validation of the glove/BLE baseline. The current Firebase persistence, stable-prediction completion logic, and Record Signs workflow were validated through the current automated tests and emulator build; they were not present in exactly this form in the historical video.

Release Candidate Navigation and Use

1. Start and Dashboard

1. Launch the app and create an account or sign in with email and password.
2. Home loads the signed-in user's local progress copy.
3. The Progress storage card reports whether data is loading, saved locally, syncing, synced, offline, or failed.
4. Retry sync appears when remote synchronization is unavailable.
5. Reset progress requires confirmation before clearing progress locally and synchronizing the empty state.

2. Connect the glove

1. Power the flashed ESP32 glove.
2. Enable Bluetooth and grant the Android permissions requested by the app.
3. Open Developer Mode > BLE Testing and select Start Scan, or select Find glove from Record Signs.
4. The app targets the advertised name HelpingHand-Glove and can auto-connect when it appears.
5. A connected status and incoming values indicate a successful notification subscription.

3. Complete a learner item

1. Open Alphabet or Numbers.
2. Select one target tile to open its practice page. The current default recorded static model supports 0–9 and letters except J/Z.
3. Form the matching static sign with the connected glove.
4. Hold the matching prediction at 80% confidence or higher for at least 750 ms and five packets.
5. The interface displays hold progress, correct, try again, low-confidence, and disconnected states near the exercise.
6. After success, the item is marked learned and saved through the progress repository.

4. Record a word trial

To practice an already trained word, open Words > Hello, Please, or Yes, tap Start attempt, perform the complete sign, then Finish sign. A matching result at 80% confidence or above saves completion. See the word demo guide for timing and rehearsal. The steps below collect additional training recordings.

1. Open Record Signs and connect the glove.
2. Enter a lowercase word label, vocabulary version, pseudonymous signer ID, orientation condition, and unique trial ID.
3. Wait for a complete, fresh packet and select Start recording.
4. Perform one complete gesture, then stop and review duration, packet count, delivered rate, valid packets, and warnings.
5. Save a clean trial or discard it with a reason.
6. Export the selected session's CSV and JSON through the share sheet.

Use pseudonymous IDs such as signer_01. Never enter a participant's name or email address, upload raw recordings to Firebase, or commit participant recordings to Git.

Prerequisites

The documented alpha validation environment on Windows used:

• Git;
• Flutter 3.41.6 stable and Dart 3.11.4;
• Android Studio/Android SDK and an Android emulator or physical Android phone;
• a Java installation supported by the installed Flutter/Android toolchain;
• Python 3.10.11 for the complete TensorFlow backend suite;
• TensorFlow 2.21.0 and pytest 9.1.1; and
• PlatformIO Core 6.1.19.

A physical Android device is required for meaningful BLE validation. An emulator can validate navigation, local persistence, and Firebase behavior but does not represent real glove discovery or sensor streaming.

Clone the repository

git clone https://github.com/GG1627/helping-hand.git
cd helping-hand

If the repository is private, the project owner must grant the instructors access before submission.

Build and run the Flutter app

cd flutter_app
flutter pub get
flutter analyze
flutter test
flutter build apk --debug
flutter devices
flutter run -d <android-device-id>

The debug APK is generated at:

flutter_app/build/app/outputs/flutter-apk/app-debug.apk

The Android Firebase client configuration targets the project listed above. Running the app does not require a service-account key. Use a team-provided test account; do not put credentials in this repository. The current application provides email/password account creation and sign-in. The older alpha used anonymous authentication; that historical behavior is not the current account flow.

For project maintainers only, Firestore rules are defined in flutter_app/firestore.rules. Always verify the selected Firebase project before deployment:

cd flutter_app
npx firebase-tools use helping-hand-83137
npx firebase-tools use
npx firebase-tools deploy --only firestore --project helping-hand-83137

Do not deploy open/test-mode rules, create service-account keys for the mobile app, or select any unrelated Firebase project.

Build, upload, and monitor the ESP32 firmware

Install PlatformIO if it is not already available:

py -3.10 -m pip install platformio

Build the firmware:

cd ESP32
platformio run

Connect the Feather ESP32-S3, identify its serial port, then upload and monitor:

platformio run --target upload --upload-port <COM-port>
platformio device monitor --baud 115200 --port <COM-port>

Expected output includes BLE advertising status and telemetry beginning with seq=..., t_ms=..., and sensor fields. Hard-coded ML test mode is disabled in the submitted source; normal packets use live flex readings.

The target recording rate can be changed at compile time to an integer from 25 through 50 Hz with HH_SENSOR_RATE_HZ. The default is 40 Hz.

Set up and test the backend

Create a clean Python 3.10 environment from the repository root:

py -3.10 -m venv .venv
.\.venv\Scripts\python.exe -m pip install --upgrade pip
.\.venv\Scripts\python.exe -m pip install -r requirements.txt
.\.venv\Scripts\python.exe -m pytest backend/tests -q

Validate a real exported Record Signs session without editing it:

.\.venv\Scripts\python.exe backend/prepare_word_sequences.py path/to/trials.csv

The model-development entry points support further training and evaluation. The three-word demo is already trained; broader recognition requires more real recordings and an evaluation protocol appropriate to the expanded scope:

.\.venv\Scripts\python.exe backend/train_word_models.py --help
.\.venv\Scripts\python.exe backend/evaluate_word_models.py --help

See backend/data/README.md for the schema and privacy contract and backend/models/README.md for the model artifact policy. Optional training-only perturbations for static and word candidates are documented in training augmentation. They default to disabled until the repaired glove supplies measured limits.

BLE packet contract

The firmware and app share Nordic UART-style service and characteristic UUIDs. Each complete notification contains a monotonically increasing sequence ID, device uptime, IMU identity, six motion values, static prediction and confidence, and five raw/normalized flex readings:

seq=...,t_ms=...,who=0xNN,
ax=...,ay=...,az=...,gx=...,gy=...,gz=...,
expected=...,pred=...,pred_conf=...,
flex0_raw=...,flex0_norm=...,...,flex4_raw=...,flex4_norm=...

Malformed notifications are retained by the recording path with their original packet text and are marked invalid rather than silently repaired.

Historical Alpha Validation Results

These commands were run on September 10, 2026 against implementation commit cce9d3887c5e52fafd4e35753f8e0de516051f79:

These results show what passed on the September build. We still need to test the current software on the repaired glove. The older videos show the earlier glove-to-phone connection; that glove was unavailable for another test when the screenshots were taken.

Repository layout

Hardware documentation note

The schematic in design_draft/schematic/ is our original ESP32-C3/LSM9DS1 concept, so it does not show the Feather ESP32-S3 glove wiring. Refer to the firmware pin definitions and hardware photos for the alpha setup. Our current KiCad schematic is pcb/pcb.kicad_sch, with board-design files under pcb/.

Known Bugs and Incomplete Release Candidate Features

1. ESP32 firmware startup - The November 6 update bounds the optional USB Serial wait to approximately 1.5 seconds before continuing to sensor/BLE setup. Firmware compilation and host checks passed; physical battery-only boot is Not run. See startup and demo evaluation tools.

2. Flutter practice lifecycle - A selected letter or number stays active after leaving its practice page. The app shell keeps accepting matching packets and can record completion while the learner is on the picker.

3. Issues with the BLE connection + Power.

4. The flex sensors are bent so the readings are off meaning the ML prediction is incorrect - Purchased new sensors.

AI-Assisted Development Disclosure

We used approved generative AI tools to help with parts of the research, planning, code, testing, and documentation, following the course AI-use policy. AI-assisted commits include GENAI=Yes in the subject and an AI-Assisted trailer. Our team is responsible for reviewing, understanding, testing, and presenting the submitted work.