Robot buddy reflex

Robot Buddy

A kid-safe, expressive robot platform combining real-time motor control, an animated TFT face, and optional networked AI planner.

How It Works

Two ESP32-S3 microcontrollers handle the deterministic, safety-critical work: one drives motors with PID control and enforces safety limits, the other renders an animated face on a 320x240 TFT touch display (esp32-face). A Raspberry Pi 5 orchestrates everything at 50 Hz — reading sensors, running the state machine, applying layered safety policies, and streaming telemetry to a browser UI. An optional AI planner server on a separate machine (3090 Ti) generates expressive behavior plans via a local LLM.

Reflexes are local and deterministic. Planner is remote and optional.

Hardware

Repository Layout

robot-buddy/
├── supervisor/       # Python supervisor (Raspberry Pi 5, process-isolated workers)
│   ├── core/            # 50 Hz tick loop, state machine, safety, behavior engine
│   ├── devices/         # MCU clients (reflex, face), protocol, expressions
│   ├── io/              # Serial transport, COBS framing, CRC
│   ├── workers/         # Process-isolated workers (TTS, vision, AI)
│   ├── messages/        # NDJSON envelope, event/action types
│   ├── api/             # FastAPI HTTP/WebSocket server, param registry
│   ├── mock/            # Mock Reflex MCU for testing (PTY-based)
│   ├── tests/           # pytest test suite
│   └── pyproject.toml   # Package metadata, deps
├── server/              # AI planner server (3090 Ti, FastAPI + backend switch)
│   ├── app/             # FastAPI app, LLM/STT/TTS backends, prompts, schemas
│   ├── tests/           # pytest test suite
│   ├── Modelfile        # Legacy Ollama model config
│   └── pyproject.toml   # Package metadata, deps
├── esp32-face/       # Face MCU firmware (ESP32-S3, C/C++, ESP-IDF)
│   └── main/            # TFT face rendering + touch/buttons + USB protocol
├── esp32-reflex/        # Reflex MCU firmware (ESP32-S3, C/C++, ESP-IDF)
│   └── main/            # Differential drive, PID, IMU, safety, encoders
├── dashboard/           # React dashboard (Vite + TypeScript + Biome)
│   └── src/             # Components, hooks, stores, tabs
├── specs/               # Completed specifications (immutable reference)
├── docs/                # TODO, architecture, protocols, wiring, power, research

Architecture

┌──────────────────────────────────────────────────────┐
│ On Robot                                             │
│                                                      │
│  ┌────────────────────────────────────────────────┐  │
│  │ Raspberry Pi 5 — Supervisor                    │  │
│  │                                                │  │
│  │  50 Hz tick loop:                              │  │
│  │    read telemetry → state machine → safety     │  │
│  │    policies → send commands → broadcast        │  │
│  │                                                │  │
│  │  HTTP API (:8080)  WebSocket (:8080/ws)        │  │
│  │  Vision process (separate OS process, 10-20Hz) │  │
│  └──────┬──────────────────────┬──────────────────┘  │
│         │ USB serial (COBS)    │ USB serial (COBS)   │
│  ┌──────▼──────┐        ┌─────▼───────┐             │
│  │ Reflex MCU  │        │  Face MCU   │             │
│  │ ESP32-S3    │        │  ESP32-S3   │             │
│  │             │        │             │             │
│  │ Motors, PID │        │ 320x240 TFT │             │
│  │ Encoders    │        │ Face +      │             │
│  │ IMU, Range  │        │ Touch UI    │             │
│  │ Safety      │        │             │             │
│  └─────────────┘        └─────────────┘             │
└──────────────────────────────────────────────────────┘

State Machine

BOOT → IDLE → TELEOP / WANDER → ERROR

• BOOT → IDLE: automatic when Reflex MCU connects with no faults
• IDLE → TELEOP/WANDER: via set_mode command
• Any → ERROR: on disconnect, ESTOP, TILT, or BROWNOUT
• ERROR → IDLE: via clear_error() when Reflex is healthy

Safety Policies (Defense in Depth)

1. Mode gate — no motion outside TELEOP/WANDER
2. Fault gate — any fault → zero twist
3. Reflex disconnect → zero twist
4. Ultrasonic range scaling (hard stop at 250 mm, 50% at 500 mm)
5. Stale range fallback (50% cap)
6. Vision confidence scaling
7. Stale vision timeout (500 ms)

Safety-critical enforcement also runs on the Reflex MCU itself (acceleration limits, command TTL, hard stop). The supervisor applies additional caps above this.

Serial Protocol

Binary packets over USB serial with COBS framing:

[type:u8][seq:u8][payload:N][crc16:u16-LE]

For esp32-face, this protocol carries face state/gesture/system/talking commands and touch/button/status telemetry only. Audio transport is supervisor-side USB audio.

Auto-reconnect with exponential backoff (0.5s–5s). See docs/protocols.md for packet definitions.

Tech Stack

Getting Started

Supervisor (Raspberry Pi 5)

cd supervisor
uv sync --group dev

# Run with mock hardware (no physical robot needed)
just run-mock

# Run with real hardware
just run

# Other options
uv run python -m supervisor --no-vision         # Disable vision worker
uv run python -m supervisor --http-port 8080    # Custom HTTP port
uv run python -m supervisor --planner-api http://10.0.0.20:8100 --robot-id robot-1

AI Planner Server (3090 Ti PC)

# Install and run the server
cd server
uv sync --extra dev --extra llm --extra stt --extra tts

# Recommended testing profile (vLLM planner + CPU STT + espeak)
LLM_BACKEND=vllm STT_DEVICE=cpu TTS_BACKEND=espeak \
uv run --extra llm --extra stt --extra tts python -m app.main

The server starts on port 8100. See server/README.md for full API docs and configuration.

ESP32 Firmware

Requires ESP-IDF toolchain.

cd esp32-face   # or esp32-reflex
idf.py build
idf.py flash
idf.py monitor

Dashboard

just run-dashboard         # dev server with hot reload
just build-dashboard       # production build → supervisor/static/

Development

All commands are available via just (see justfile):

just test-all              # run all tests (supervisor, server, dashboard)
just lint                  # check Python + C++ + dashboard
just lint-fix              # auto-fix formatting
just preflight             # full pre-commit check (lint + tests + parity)
just sim                   # run face simulator V3
just check-parity          # verify sim↔MCU constant alignment

Mock Mode

The supervisor includes a PTY-based mock Reflex MCU (supervisor/mock/mock_reflex.py) that simulates serial communication, telemetry, and fault injection. Use just run-mock to run the full supervisor stack without any hardware.

Dashboard

When the supervisor is running, open http://<robot_ip>:8080 in a browser for:

• Live telemetry display with diagnostic tree
• Mode control (IDLE, TELEOP, WANDER)
• E-STOP button
• Face control (moods, gestures, talking, conversation state)
• Parameter tuning sliders (PID gains, speed limits, safety thresholds)
• Monitor tab (device health, comms, power, sensors, faults, workers)
• MJPEG video stream (if vision enabled)

Configuration

Supervisor — YAML config file (schema in supervisor/config.py):

• Sections: serial, control, safety, network, logging, vision
• Default serial paths: /dev/robot_reflex, /dev/robot_face (via udev symlinks)

AI Server — environment variables:

• LLM_BACKEND, VLLM_MODEL_NAME, LLM_MAX_INFLIGHT, PERFORMANCE_MODE
• legacy compatibility: OLLAMA_URL, MODEL_NAME, PLAN_TIMEOUT_S, TEMPERATURE, NUM_CTX
• See server/README.md for the full table

ESP32 — sdkconfig.defaults + config.h constants

Supervisor API

AI Server API

Plan actions: say(text), emote(name, intensity), gesture(name, params), skill(name) — planner proposes intent and supervisor executes deterministic skills.

Supervisor Fallback Policy

Project Status

Working

• Supervisor: 50 Hz control loop, state machine, safety policies, conversation state, mood choreography
• Supervisor: serial transport with COBS framing, CRC, auto-reconnect, protocol v2 (timestamps + seq)
• Supervisor: FastAPI HTTP/WebSocket API with telemetry streaming
• Supervisor: process-isolated workers (TTS, vision, AI, ear)
• Supervisor: mock Reflex MCU for hardware-free development
• AI Server: FastAPI + vLLM planner (Qwen), conversation, TTS (Orpheus + espeak)
• ESP32 Face: TFT face rendering, 13 moods, 13 gestures, conversation border, touch/button telemetry
• ESP32 Reflex: motor control, PID, encoders, IMU, safety enforcement
• Dashboard: React 19, live telemetry, face control, monitor tab, mode control
• Voice pipeline: wake word ("hey buddy") + VAD → STT → LLM → TTS → face animation
• WANDER mode driven by deterministic skills + planner intent
• Deterministic telemetry: timestamps, sequence numbers, clock sync, raw packet logging

In Progress

• Reflex MCU hardware commissioning (breadboard bring-up)
• Personality engine implementation (spec complete, implementation pending)

Future

• Conversation memory / interaction history
• Wake word model improvements (recall 42% → 80%+)
• Additional modes: LINE_FOLLOW, BALL, CRANE, CHARGING
• Home Assistant integration
• Voice ID / speaker identification

See docs/TODO.md for the detailed backlog and specs/ for design specifications.