Intercom

Arthur

Arthur is my personal silicon assistant. Local-first and on the desktop natively in MacOS and through an experimental hardware device that share the same conversational memory.

This repository is the voice layer: capture, speech, devices, and the spoken reply. The spoken-agent definition is config/arthur.agent.json.

STT and TTS run on the machine that hosts the Intercom daemon. whisper.cpp transcribes, Kokoro speaks, and Arbiter handles reasoning.

Components

Intercom daemon — HTTP on :8090 (POST /v1/utterance, health, cancel) and a WebSocket hub on :8093 (/v1/stream). It owns STT, TTS, the turn pipeline, device sessions, optional Home Assistant fast-path, and speak-back from Arbiter’s notification stream.

Mac desk app (macos/Arthur) — a native SwiftUI window plus a menu-bar droplet. Click the droplet, hold it, or use the global shortcut to talk. The app keeps the full conversation, and uses the same WebSocket, device_token, and device id for shared memory.

Hardware device (firmware/, hardware/) — a thin audio endpoint. Firmware for an ESP32-S3 (Arduino Nano ESP32 or a board with the same pin map) plus I2S mic (INMP441) and amp (MAX98357A) lives in firmware/nano-esp32/intercom-endpoint.

Weather (home via Home Assistant, or any city via Open-Meteo), news (Google News RSS / configured feeds), and markets (Yahoo public quotes) each speak a short line and emit a versioned {type:surface} card on the desk WebSocket. Voice-only devices ignore surface. See docs/api.md.

Build

Daemon needs C++20, CMake 3.20+, SQLite3, OpenSSL, and Threads. It fetches cpp-httplib and nlohmann/json.

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

Mac app (Apple Silicon, macOS 26+). Open macos/Arthur/Arthur.xcodeproj, or:

macos/Arthur/build.sh
open macos/Arthur/dist/Arthur.app

Speech tools are not vendored. Install them separately and point intercom.json at the binaries and models:

• whisper.cpp — whisper-cli, whisper-server, and ggml-base.en.bin
• Kokoro — kokoro-tts plus the ONNX model and voices bundle. Intercom starts scripts/kokoro_server.py from that venv unless you already have a server running.

Configure and run

Run Intercom beside arbiter --api on the same host.

cp config/intercom.example.json intercom.json
# also keep config/arthur.agent.json where agent_def can find it
# (same directory as intercom.json, or set agent_def to config/arthur.agent.json)

A local intercom.json is gitignored. Set at least:

• arbiter_token — Arbiter atr_… token (arbiter_base_url defaults to http://127.0.0.1:8080)
• device_token — bearer the wall button and Mac app send (not the Arbiter token)
• whisper.binary / whisper.model and kokoro.binary / kokoro.model / kokoro.voices

Then:

./build/intercom --config intercom.json

Default ports, all on listen_host (example config uses 0.0.0.0):

GET /health is ready when Whisper and Kokoro are up; Arbiter may still be down. Set whisper.use_server / kokoro.use_server to false to force the one-shot CLI path.

Optional home block: when ha_base_url and ha_token are set, hallway phrases for lights, volume, weather at home, timers, and the next alarm skip Arbiter. Optional news and markets blocks configure RSS feeds and the quote host; both work without API keys (see config/intercom.example.json).

Firmware. Copy Wi-Fi and token into firmware/nano-esp32/intercom-endpoint/secrets.h (gitignored) or edit config.h. INTERCOM_HOST must be the LAN address of the Intercom machine — not 127.0.0.1. WebSocket is on INTERCOM_WS_PORT (8093); set it to 0 for HTTP-only. Flash the sketch in firmware/nano-esp32/intercom-endpoint. Contract: docs/device.md.

Mac. Settings load host, ports, and device_token from intercom.json. Choose the hallway device under Shared memory so this Mac and the wall button keep one conversation. Default global talk shortcut is ⌥Space (change it under Push to talk). Microphone permission is required to speak. Quiet text mode (Arthur menu or Settings → Typing) stops Space from starting push-to-talk so the composer is safe to type in; the global shortcut and menu-bar droplet still talk.

Docs

• docs/api.md — HTTP and WebSocket contract, Arbiter mapping, speak-back, Kokoro
• docs/device.md — PCM format, PTT flow, ESP32 endpoint, colocation
• config/intercom.example.json — daemon config
• config/arthur.agent.json — spoken-agent definition

License

Apache-2.0