Workspace

RealityCheck · hardware design sandboxes

An open testing ladder for hardware agents. RealityCheck combines portable C peripheral drivers, independent digital emulators, ngspice analogue simulation and KiCad connectivity/layout checks. Agents can design a board, write firmware, observe failures and iterate before fabrication.

This is an early, unsaturated development benchmark, not a silicon-validated leaderboard. Budgeted pilot runs expose substantial headroom in producing routed, electrically valid designs. Passing reference designs exist within declared model families; restricted, versioned evaluators cover subsets of the full tasks. A passing emulator test does not prove analogue performance or PCB connectivity.

Presentation · Challenge definitions · Reference designs · Architecture

Three challenges

Read the challenge README and its versioned criteria before starting. Cost uses benchmark credits, not market prices. All mandatory gates must pass before a positive restricted reward is earned; unsupported coverage is reported separately from demonstrated failure. These three families and a small pilot do not establish statistical model rankings.

Build the portable library

cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug
cmake --build build
ctest --test-dir build --output-on-failure

C11, CMake and a host compiler are sufficient for the core library. The hw_drivers target has no emulator, allocator or OS dependency. Components: ADS7924, DAC80501Z, TCA9548A, TCA9535, MCP3204 and MCP4921. Models expose bus transactions, pins and virtual time; firmware uses injected callbacks. examples/baremetal-smoke.sh additionally requires arm-none-eabi-gcc and performs compilation, not target execution.

Build the local sandbox

python3 sandbox/build.py
# Offline smoke: native C, peripheral models, SPICE and headless KiCad
docker run --rm --network none hardware-bench:dev

The build wrapper downloads official hash-pinned datasheets locally, then builds the toolchain image. No datasheet PDFs are stored in this repository. Cached documents remain ignored build inputs; evaluation has no vendor-network dependency. --offline skips acquisition when the cache is already populated. Toolchain/package sources are pinned in the Dockerfile and source locks. Documents and package assets retain their own licences; this workflow assembles a local image, not a redistributable vendor-document bundle.

The independent Codex runner currently targets linux/arm64:

python3 sandbox/agent/build.py --acquire --training-base hardware-bench:dev
python3 sandbox/agent/login.py login
python3 sandbox/agent/runner.py --challenge 01 --model MODEL_ID \
  --auth chatgpt --output runs/my-first-attempt

Each invocation creates an independent session/container with a challenge-specific README. Candidate network access is disabled; a bounded broker outside the container provides inference access. Credentials stay outside candidate mounts and exports. ChatGPT login and explicit API-key mode are separate choices; follow the sandbox runbook and official Codex authentication guidance. Model availability depends on the account and pinned CLI. The default attempt budget is 900 seconds; configure it explicitly for comparisons.

Evaluate and iterate

./bench list
./bench init sensor-monitor --output /tmp/my-design
./bench check /tmp/my-design
./bench evaluate /tmp/my-design --seed 42 --output /tmp/my-result

bench is the public preflight interface: templates are incomplete, and missing simulation/firmware adapters fail closed. It is not the authoritative restricted ranker. design.yaml uses the JSON subset of YAML. For scored development use the trusted, versioned evaluator outside the submission:

• Challenge 01 restricted v1 and v2 coverage.
• Challenge 02 restricted training, with published v2 startup criteria.
• Challenge 03 necessary-gate v2: demonstrated violations receive zero; passing these checks does not enable positive ranking.

Historical qualification receipts, waveforms, candidate runs and transcripts are deliberately excluded. A fresh image has a new local identity: regenerate qualification and freeze reviewed source/evidence locks before enabling a release. Missing evidence must fail closed. Some private-fixture comparison tools cannot run from this source-only checkout. Publication and reproduction notes explain these boundaries.

Reproduce a reference with a locally built image:

python3 tools/reproduce_reference.py --challenge 02 \
  --image hardware-bench:dev --output /tmp/reference-02

This regenerates results and records the actual image identity; it does not transfer authority from a historical qualification. References are available for study and therefore must be excluded from held-out agent images, along with private evaluator tools and hidden scenarios. The public agent image excludes these directories.

What is here

include/ and src/: portable firmware · emulator/: digital models/runtime · catalog/: costs, packages and analogue models · challenges/: tasks/criteria · reference/: editable CAD and C solutions · evaluator/: trusted grading source · sandbox/: pinned environment and isolated runner.

See fidelity contracts, evaluation boundary and third-party notices. Simulation scores describe the declared models. Board fabrication, measured hardware conformance, EMC and a broad generalisation study remain future work.

Team RealityCheck, AI for the Physical World hackathon, October 2026.