NEO Lite

NEO — open-source ESP32-S3 voice assistant with an offline wake word and a local LLM

NEO is a DIY smart speaker built around an ESP32-S3. You say "Hola NEO": a 15 KB TinyML model (TensorFlow Lite Micro, int8) recognizes the wake word on the chip, offline. Then you ask anything, hands-free. A Python server on your PC transcribes the question with Whisper, answers it with a local LLM (Ollama: Gemma, Qwen) or the cloud (Groq), and talks back through the speaker with streaming text-to-speech.

It is a privacy-friendly, hackable Alexa / Google Home alternative you can build on a breadboard for the price of a few modules, and every number in this README was measured on the real board, with the raw data in the repo.

flowchart TB
    subgraph board["ESP32-S3 board"]
        direction LR
        mic["INMP441 mic"] --> ww["Wake word 'Hola NEO'<br/>TFLite Micro + ESP-NN"] --> vad["VAD<br/>300 ms pre-roll"]
        spk["DAC + amp<br/>speaker"]
    end
    subgraph server["Python server: your PC or Hugging Face Spaces"]
        direction LR
        stt["STT<br/>faster-whisper / Groq"] --> intent{"Intent"}
        intent -- command --> api["Weather, crypto,<br/>news, rates"]
        intent -- question --> llm["LLM<br/>Ollama / Groq"]
        api --> tts["TTS by fragments<br/>edge-tts / Piper"]
        llm -- tokens --> tts
    end
    vad -- "PCM 16 kHz, WebSocket" --> stt
    tts -- "PCM, streamed" --> spk

Highlights

• Offline wake word on a microcontroller. A CNN on MFCC features, quantized to int8 and accelerated with Espressif's ESP-NN SIMD kernels: 0.32–0.40 s per inference, ~1.1 s from the end of "Hola NEO" to NEO answering. Test accuracy 95.7 %; the full training pipeline is included.
• Local-first. The board talks to your PC on the LAN; if it is not there, it falls back to a cloud deployment. The server can run fully local (faster-whisper + Ollama) or on free cloud tiers (Groq + edge-tts).
• Streaming responses. The LLM's tokens are cut into phrases, each one is synthesized and sent as soon as it is ready, and the firmware starts playing after 256 ms of buffered audio, without waiting for the full reply.
• Commands without an LLM. Weather, crypto prices, tech news and Cuban informal exchange rates are answered by keyword intent detection and real APIs: faster, cheaper and no hallucinations.
• Robust on bad networks. Prerecorded phrases tell you when WiFi, the server or the internet is down. WiFi sleeps after 10 min idle and the CPU drops to 80 MHz while the wake word keeps listening.
• Measured, not claimed. Word error rate, latency per pipeline stage, wake word confusion matrix and power draw per state, from 31-phrase sessions read to the board.

Quick start

You need the hardware, a PC on the same network (Windows, macOS or Linux) and about 30 minutes.

1. Server

git clone https://github.com/techmiguel/neo-robot.git
cd neo-robot/servidor
python -m venv .venv
.venv\Scripts\activate          # Windows; on macOS/Linux: source .venv/bin/activate
pip install -e ".[dev]"
cp .env.example .env            # Windows: copy .env.example .env

Pick a profile and edit .env:

The command APIs are optional: OPENWEATHER_API_KEY, NEWSAPI_KEY and ELTOQUE_API_KEY (free tiers; CoinGecko needs no key). Every variable is explained in servidor/.env.example.

python -m src.server.ws_server --perfil nube     # or --perfil local

The server listens on ws://0.0.0.0:8765. Note your PC's LAN IP address.

2. Firmware

Install PlatformIO (VS Code extension or pip install platformio).

cd neo-robot/firmware
cp src/config_local.example.h src/config_local.h   # Windows: copy; set NEO_SERVIDOR_HOST to your PC's IP
pio run -e esp32dev --target upload
pio device monitor -e esp32dev

The first build downloads the ESP32 toolchain and TensorFlow Lite Micro and takes a few minutes. Without config_local.h the firmware builds with a placeholder IP (192.168.1.100) and no cloud fallback.

3. First boot

1. NEO has no WiFi yet, so it opens an access point called NEO-Config. Join it from your phone and open http://192.168.4.1.
2. Enter your WiFi name and password. NEO saves them, connects, and finds the server.
3. Say "Hola NEO", wait for the greeting and ask, in Spanish: "¿Cuál es la capital de Japón?", "¿Va a llover mañana en La Habana?", "¿Cuánto vale el bitcoin?".

No wake word? Press the BOOT button briefly to ask without it. The model was trained on two voices; if it does not wake for yours, train it on your voice.

Troubleshooting

Hardware

Wiring, pin tables and the mistakes to avoid are in docs/hardware.md. A compact custom PCB (NEO Lite, 31 × 43 mm, USB-C, ready for JLCPCB assembly) is in hardware/neo-lite/, designed in KiCad 9 but not yet fabricated.

What you can ask

NEO speaks Spanish. After the wake word it detects the intent of the transcription:

Adding a command is a Python class on the server (servidor/src/commands/); the firmware does not change.

Measured results

All numbers come from the board. Method, raw data and analysis are linked.

Wake word "Hola NEO", CNN int8 on the ESP32-S3, two speakers (experiments, metrics)

Conversation, 31 phrases read to the board, same audio sent to every server variant. Word error rate ignores accents. Latency runs from the end of the recording to the first audio of the reply, median (p90).

* The 25/10 run exposed that gemma4:e2b re-processes the system prompt on every question (~37 tokens/s on a laptop i5): the 409-token prompt cost ~8 s. The local profile now uses a 139-token prompt (2.7–3.3 s in Ollama, not yet re-measured end to end). Streaming saves ~1.9 s once answers are long enough to split; edge-tts costs ~1.9 s per call from Cuba. On the board, add 1.5 s of end-of-speech silence and ~0.6 s of WiFi upload.

Power at 5 V (INA219, method, data, preliminary)

Train the wake word on your voice

The model ships in the firmware (firmware/src/models/) and was trained on two speakers. The recordings are not published (they are people's voices), but the pipeline is:

cd firmware
pio run -e dataset_capture --target upload               # firmware that streams clips over serial
pip install -r tools/requirements_dataset.txt
python tools/capture_dataset.py --port COM8              # guided recording per class
pip install -r tools/requirements_train.txt
python tools/train_model.py                              # trains, quantizes to int8, exports TFLite
python tools/gen_vocabulario.py --umbrales models/umbrales.json
python tools/tflite_to_header.py                         # writes src/models/*.h
pio run -e esp32dev --target upload

tools/vocabulario.json (classes, phrases, thresholds) is the single source of truth for capture, training and inference. The experiments behind the current model (cross-validation, leave-one-speaker-out, negative speech) are in firmware/models/experimentos/.

Other firmware environments

Benchmark your own setup

The server includes the validation panel used for every number above: it relays the board's audio to several server variants (models, providers, streaming on/off), scores the transcriptions against the corpus and reads the latency from the board's serial log.

cd servidor
python -m src.benchmark.panel --variante "gemma=local,OLLAMA_MODEL=gemma4:e2b" --variante "nube=nube"

Open http://localhost:8080, read the phrase it shows to NEO, and press Generar informe at the end. Details in servidor/benchmark/README.md.

Repository layout

firmware/            ESP32-S3 firmware (C++, PlatformIO + Arduino)
  src/               audio, display, network, commands (TFLite inference), main.cpp
  tools/             dataset capture, training, header generation, log analysis
  models/            trained models and the experiments behind them
  logs/              serial logs of the measured sessions
servidor/            Python server: STT, intent, LLM, TTS, WebSocket, benchmark panel
hardware/neo-lite/   custom PCB: KiCad project, generator scripts, Gerbers, BOM
docs/                design notes, experiments and measured metrics (in Spanish)
ROADMAP.md           phases and status (in Spanish)

Documentation

The design notes and experiment reports are written in Spanish.

Design decisions

• Only the wake word runs on the chip. A multi-command local classifier would need a dataset per phrase and a reflash per change. Commands live on the server and grow without touching the firmware; the cost is that every question needs the network.
• Keywords before the LLM. Common commands skip the LLM: faster, free and deterministic. The LLM only gets free-form questions.
• Two I2S buses. Microphone and DAC each get their own peripheral; no time multiplexing.
• Inference buffers in internal SRAM. PSRAM access with WiFi active tripped the watchdog.
• One voice. NEO always speaks with edge-tts es-CU-ManuelNeural, including its prerecorded phrases; Piper only answers when edge-tts is unreachable.

Failure modes

Every message below is prerecorded in the firmware, so NEO can say it without a server.

Security and limitations

• The WebSocket has no authentication; it assumes a trusted LAN. Add a token or TLS before exposing the server. API keys live in servidor/.env (gitignored).
• Spanish only: STT, prompts and voice.
• Short memory: the last 5 exchanges per WebSocket session, in RAM.
• USB power: speaking peaks at ~0.8 A at 5 V; a single boosted 18650 cell browns out.
• No OTA updates yet; flash over USB.
• The local profile is slower and less accurate than the cloud one on a laptop CPU.

Contributing

Issues and pull requests are welcome; see CONTRIBUTING.md. Run the tests before sending changes:

cd servidor && pytest
cd firmware/tools && pytest

License and acknowledgements

MIT. Built on TensorFlow Lite Micro, ESP-NN, faster-whisper, Ollama, Groq, edge-tts, Piper, Adafruit SSD1306 and arduinoWebSockets.

¿Hablas español? Lee el README en español.