SignLanguage Glove

Smart Glove for Real-Time Nepali Sign Language to Speech Translation

This repository contains the source code, hardware design files, datasets, and documentation for the Smart Glove for Real-Time Nepali Sign Language (NSL) to Speech Translation system. Developed as a final engineering project at Pashchimanchal Campus, Tribhuvan University, this wearable system translates static and dynamic Nepali Sign Language gestures into spoken audio output in real time to bridge the communication gap between the deaf/hard-of-hearing community and non-signers.

Table of Contents

• Abstract
• System Architecture
• Hardware & Software Requirements
• Methodology
  • Hardware Interfacing & Sensor Calibration
  • Data Collection & Dataset Preparation
  • Model Architecture & Training
  • Real-Time Inference Pipeline
  • Text-to-Speech (TTS) Integration
• Results and Discussion
  • Real-Time Performance
  • Random Forest Analysis
  • LSTM Model Analysis
• Video Demo
• Project Report
• Appendix

Abstract

Communication is essential, yet most non-signers in Nepal do not understand Nepali Sign Language (NSL). This project presents a wearable smart glove system integrating 8 flex sensors (capturing joint flexion across MCP and PIP finger joints) and an MPU6050 6-axis IMU mounted on an ESP32 microcontroller. Sensor data is transmitted via Bluetooth to a host environment, where a dual machine learning model pipeline performs real-time classification:

• Random Forest Classifier: Handles 9 static gestures (4 alphabets: ka, kha, ga, gha, nga and 4 numerals: ek, dui, tin, char) with 99.56% test accuracy.
• 2-Layer LSTM Network: Processes 7 dynamic gestures (namaste, mero, naam, sanjog, ho, malai, chinnu) forming a basic self-introduction sequence, achieving ~94% validation accuracy.

Across 320 real-time trials, the full system achieved an overall accuracy of 93.1%, with average translation latencies of 0.5 seconds for static gestures and ~2.0 seconds for dynamic gestures.

System Architecture

+-------------------------------------------------------------+
|                     WEARABLE SMART GLOVE                    |
|  [8x Flex Sensors (PIP/MCP Joints)]  [MPU6050 6-Axis IMU]   |
+------------------------------+------------------------------+
                               | Analog/I2C
                               v
+-------------------------------------------------------------+
|                     ESP32 MICROCONTROLLER                   |
|   - Signal Acquisition & Filtering                          |
|   - Feature Vector Assembly & Gyroscope Accumulation        |
+------------------------------+------------------------------+
                               | Bluetooth Serial Stream
                               v
+-------------------------------------------------------------+
|                      HOST COMPUTER                          |
|  +-------------------------------------------------------+  |
|  |             Motion Detection Logic                    |  |
|  |     (Rolling Buffer Gyro Variance Thresholding)       |  |
|  +---------------------------+---------------------------+  |
|                              |                              |
|              +---------------+---------------+              |
|              | Static                        | Dynamic      |
|              v                               v              |
|   +---------------------+         +---------------------+   |

Hardware & Software Requirements

Hardware Components

• ESP32 Microcontroller: Dual-core SoC handling ADC acquisition, I2C IMU sampling, and Bluetooth RFCOMM streaming.
• Flex Sensors (x8): Variable bend-resistors positioned across finger joints (2.2" and 4.5" strips).
• MPU6050 IMU: 3-axis accelerometer + 3-axis gyroscope for hand orientation and motion tracking.
• Resistors: $20 \text{ k}\Omega$ precision fixed series resistors for flex sensor voltage dividers.
• Audio Output Device: Speaker/headphones connected to the host system.

Software & Libraries

• Firmware: C++ / Arduino Framework deployed via PlatformIO / VS Code.
• Data Processing & ML/DL: Python 3.x, NumPy, Pandas, Scikit-Learn, TensorFlow / Keras.
• Text-to-Speech: gTTS (Google Text-to-Speech) library.
• Hardware CAD: KiCad EDA for schematic capture and PCB layout.

Methodology

Hardware Interfacing & Sensor Calibration

• Flex Sensor Placement: 8 flex sensors are mapped to capture fine-grained finger articulation:
  • Index, Middle, Ring Fingers: 2 sensors per finger—one across the Proximal Interphalangeal (PIP / "Up") joint and one across the Metacarpophalangeal (MCP / "Low") joint (idxUp, idxLow, midUp, midLow, ringUp, ringLow).
  • Thumb & Pinky: 1 sensor each (thumb, pinky).
• Voltage Divider Optimization: Flex resistance ranges from $\sim 20 \text{ k}\Omega$ (flat) to $\sim 24 \text{ k}\Omega$ (bent). Connecting a $20 \text{ k}\Omega$ series resistor ($R_2$) maximizes the ADC voltage differential to optimize 12-bit ADC sensitivity.
• Sensor Calibration: Physical conditioning (15–20 fist flexes) redistributes sensor conductive ink before use. At startup, the system captures 100 baseline idle samples to compute per-sensor zero offsets.

Data Collection & Dataset Preparation

Sampling is executed at 60 ms intervals. Each feature vector consists of 17 parameters:

x = [idxUp, idxLow, midUp, midLow, ringUp, ringLow, thumb, pinky, a_x, a_y, a_z, g_x, g_y, g_z, acc_g_x, acc_g_y, acc_g_z]

• Static Dataset: 1000 samples per gesture across 9 gestures = 9,000 labeled samples (.csv format). Dynamic gyro parameters were excluded during static training.
• Dynamic Dataset: Motion detection uses a 20-sample rolling buffer tracking accumulated gyroscope variances ($\Delta acc_g$). A threshold $> 20$ triggers active recording, terminating after 7 consecutive static samples. 200 raw sequences per gesture were recorded and expanded to 1,500 samples per gesture via white-noise data augmentation. All sequences were resampled to a fixed length of 70 frames using linear interpolation.

Model Architecture & Training

1. Static Gestures (Random Forest Classifier)

   • Input scaled via StandardScaler. Hyperparameters tuned using GridSearchCV.
   • Configuration: n_estimators=30, max_samples=0.70, Train/Test Split: 70/30.

2. Dynamic Gestures (LSTM Network)

   • $(70 \text{ frames} \times 15 \text{ features})$, normalized via MinMaxScaler.
   • Architecture: 2-layer stacked LSTM (16 hidden units per layer), Dropout rate $= 0.5$, Batch size $= 32$, Adam optimizer, Categorical Cross-Entropy loss.
   • Early Stopping: Monitored validation loss with a patience of 10 epochs.

Real-Time Inference Pipeline

1. ESP32 streams sensor vectors via Bluetooth RFCOMM.
2. Host Python script buffers incoming frames and evaluates gyroscope variance:
   • Low Variance ($\le 20$): Routes current frame to Random Forest (.joblib) $ ightarrow$ Output static gesture.
   • High Variance ($> 20$): Accumulates sequence frame buffer $ ightarrow$ Interpolates to 70 frames $ ightarrow$ Feeds into LSTM (.h5) $ ightarrow$ Output dynamic gesture.

Text-to-Speech (TTS) Integration

Predicted labels map to pre-synthesized Devanagari audio files generated via the gTTS library. The system executes immediate audio playback upon gesture classification completion.

Results and Discussion

Real-Time Performance

The complete hardware-software system was evaluated across 320 real-time trials (20 trials per gesture across 16 gestures).

• Average Static Latency: $\sim 0.5 \text{ seconds}$
• Average Dynamic Latency: $\sim 2.0 \text{ seconds}$

Random Forest Analysis

• Training Accuracy: 99.94%
• Testing Accuracy: 99.56%
• Out-Of-Bag (OOB) Score: 99.62%

The minimal delta between training and test metrics confirms high model generalization with minimal risk of overfitting.

LSTM Model Analysis

• Validation Accuracy: Stabilized at $\sim 94%$.
• Temporal Importance Analysis: Feature importance evaluation revealed that the network heavily weights the final sequence steps (frames 66–69), focusing on "ending signatures." This temporal weighting explains occasional misclassifications between gestures with similar trajectory endings (e.g., mero vs. namaste). Future work will incorporate Attention Mechanisms or Bidirectional LSTMs (BiLSTM) to distribute importance evenly across full trajectories.

Video Demo

An end-to-end video demonstration showing real-time sign recognition, dual-model inference switching, and instant audio vocalization is available below:

Watch Demo Video

Project Report

The complete academic major project report submitted to the Department of Electronics and Computer Engineering, Pashchimanchal Campus, IOE, Tribhuvan University, is included in this repository:

📄 Download Full Major Project Report (PDF)

Appendix

Prototype Smart Glove

Physical wearable prototype featuring 8 integrated flex sensors, custom wiring harness, and glove-mounted ESP32/MPU6050 enclosure.

Custom PCB Schematic

KiCad EDA PCB schematic showing power regulation, ESP32 ADC pin routing for flex sensor voltage dividers, and I2C lines for MPU6050.

Authors & Acknowledgments

Project Team (PAS078BEI Batch):

• Ganesh Rokaya (PAS078BEI013)
• Sanjog Sapkota (PAS078BEI034)
• Santosh Kumar Barai (PAS078BEI035)
• Upendra Raj Joshi (PAS078BEI046)

Supervisor: Asst. Prof. Khem Raj Koirala
Department: Department of Electronics and Computer Engineering, Pashchimanchal Campus, IOE, Tribhuvan University.