Attempt-02

PCB Generation Benchmark

Project scaffold for a paired dataset comparing three ways of producing PCB designs from the same prompts:

Status: 10 draft prompts and a completed two-method generation pilot. The prompt catalog includes easy, medium, and hard prompts with topic labels. The pilot results contain 20 generated schematic/PCB outcomes, retained candidates, previews, and raw validation reports. Computer use was excluded at the user’s request. Some generated designs have failing checks or unverified requirements; execution completion is not design acceptance. A deterministic rule scorer is implemented; reusable generation adapters/runner and scoring calibration remain future work.

Layout

prompts/                 Shared prompt catalog and per-prompt assets
configs/                 Experiment and environment templates
methods/                 Workflow contracts for the three methods
schemas/                 Record conventions for future validators
templates/               Copyable prompt, run, and evaluation records
data/runs/               Raw runs, organized by experiment/prompt/method/replicate
data/evaluations/        Separate evaluation records linked to run IDs
data/releases/           Future versioned dataset manifests
src/adapters/            Reserved for method implementations
src/runner/              Reserved for orchestration
src/evaluation/          Reserved for scoring and artifact inspection
scripts/                 Reserved for future commands
tests/                   Reserved for future validation tests
reports/                 Comparison reports and figures
docs/                    Protocol, dataset card, implementation roadmap
work/                    Ignored scratch space

Start here

1. Read docs/protocol.md and docs/roadmap.md.
2. Copy templates/prompt/ into prompts/items/<prompt-id>/, replace placeholders, and add the ID to prompts/catalog.json.
3. Copy configs/experiment.template.json to a named experiment configuration. Freeze its prompt revisions, model, budgets, tools, and repeat count before collecting data.
4. When the runner exists, collect each prompt across all three methods using templates/run/. Store evaluations separately using templates/evaluation.template.json.

Template files are examples of structure, not actual dataset entries. Cataloged prompts are drafts awaiting review. Do not report placeholder metrics as measurements. The recorded pilot uses installed tools and isolated generation tasks; reusable automation and full benchmark collection remain future work.

Deterministic scoring

The scoring implementation evaluates frozen artifacts with versioned rules and fixed 30/30/20/20 weights. Pilot scoring results include per-rule evidence, critical failures, unknowns and possible score bounds. Run bun run score -- 2026-09-05-codegen-pilot-01 to reproduce them. Missing evidence remains null; no model judge is called.

Historical model-judge proposal

See docs/model-judging.md for the model-judge plan, evaluation/rubric.json for scoring rules, and configs/judge.template.json for fixed judge settings. Judgment and baseline-comparison templates record evidence, repeated scores, and improvement/regression decisions. The model judge and baseline-comparison flow remain scaffolds; deterministic scoring is implemented separately as described above.

Pilot results

Start with the results and PCB previews. The frozen configuration and execution notes record tool versions, budgets, the GUI exclusion, and context limitations. Eighteen runs used fresh subagents; two were generated by the parent after a hard agent limit. No comparative quality score is claimed.

Regenerate the file inventory and check prompt hashes/pair completeness with bun run collect -- 2026-09-05-codegen-pilot-01 --require-complete. This inventories existing artifacts and does not rerun generation.

Git and storage

Track prompts, configs, native design files, code, and compact evidence. Do not commit secrets. Large recordings should use an explicitly selected external store or Git LFS when configured; record their URI and SHA-256 in the run manifest. Neither external storage nor LFS is configured yet. Dataset and code licensing remain undecided; see docs/dataset-card.md.

Development with TypeScript and Bun

All active repository code runs with Bun. Install locked dependencies with bun install --frozen-lockfile.

• bun run typecheck — check TypeScript.
• bun test — scoring, geometry, artifact readers, archive integrity, saved-result parity, and website checks. Website checks use the existing generated bundle; run bun run build first after a clean checkout.
• bun run rules — verify/reproduce the ten frozen v1 rule specifications.
• bun run score -- 2026-09-05-codegen-pilot-01 — evaluate retained artifacts into a new immutable TypeScript evaluation archive.
• bun run collect -- 2026-09-05-codegen-pilot-01 --require-complete — inventory existing runs.
• bun run build — render PCB layers and build the website.
• bun run website:publish — build and replace the existing public website using ohmx-box.

The PCB renderer calls the native KiCad CLI (KICAD_CLI or kicad-cli on PATH). Hosting uses the existing ohmx-box CLI. These are external tools, not Python repository scripts. Historical generation scripts and archived Python evaluator sources remain unchanged under data/ for reproducibility. Canonical prompts and frozen run inputs also remain unchanged.