Blog · November 4, 2026

AI PCB design benchmark: 12 of 20 sealed wishes done first try

Sergey Alekseev

This is how we measure AI PCB design in PCBKit: a real coding agent gets one sentence from a person (a wish like “a USB-C CO₂ monitor with a display”) and works alone until it stops. The benchmark counts how often the board it leaves is finished on the first try. On the sealed set of 20 wishes the answer is now 12. Below is why the wishes are sealed, what “finished” has to mean before the number says anything and where the agent still fails.

Why the wishes are sealed

Before this benchmark we measured the agent flow three ways. All three were easy to game without meaning to. The scenario suite builds designs we wrote ourselves, so it measures the engine and says nothing about what an agent writes. QA rounds used one agent’s wishes; the fixes that followed were tuned on exactly those wishes. The eval journeys graded turns against rubrics, with a model’s grade and our regexes, on scenarios the tools had been polished on for days. A number that improves because we tuned on its inputs says nothing about the next person’s wish.

So there are two sets. The dev set is 12 wishes in the repository, wishes we have already tuned on: defects found there are fixed and rechecked there. The held-out set is 20 wishes in our deploy bucket only. The repository holds the file’s SHA-256. The harness refuses a copy with another hash. Results on the held-out set are read as aggregates (shares and medians). Its wishes, designs and failures are not studied for fixes. If a held-out result turns into a fix anyway, the set is spent: we write a new one and never edit the old one.

A wish is written the way a person writes one, with no design hints and no part numbers beyond what a person would name. The sets span sensors on batteries, power, motors, LEDs, audio, buses, USB bridges, HATs and shields, handhelds, keyboards, analog front ends, chargers, high current, mains and wishes in Russian.

The agent under test

The agent is Codex (gpt-6.1-sol, effort medium) on a ChatGPT subscription, with the PCBKit plugin installed the way people install it. No model grades the result: a grade is the judge’s taste and moves with its model. Twenty conversations at three at a time take about two hours of subscription time, so the sealed set runs at milestones and the dev set is for iterating.

What counts as finished on the first try

The first bar was the product’s own verdict: the first turn leaves the board complete (every part placed, every connection routed) with 0 errors, verified through the layout level of the verifier. Under that bar the first sealed run finished 18 of 20.

Then we audited the bar on 36 finished dev boards and found three holes. 8 of the 36 kept no board file anyone could re-check: a tool marked a long file’s later pages as partial and the harness dropped the file. These were the runs of the three hardest wishes. 12 of the 28 re-checkable boards carried an unconfirmed finding about a quantity the wish stated (a track that may not carry the asked current, for one). Unconfirmed findings did not count. Nothing at all checked that the wish’s asks were in the design.

The audit also tried a second judge. A language model read each board’s wish, design, check summary and picture and would order 0 of the 36. Of the 7 defects it named that the design source settles, 6 were wrong (a relay coil “shorted” because a pin named COIL2 sat on a net named COIL1; four charge LEDs “missing” that were there, multiplexed as the chip’s reference circuit has them). That settled it: a model may pick which boards a person looks at; its verdict never counts.

The amended bar, now in the ADR, adds to complete, 0 errors and verified through layout:

• the kept project is the board: every file kept whole and the kept files checked again in a new project give the same verdict;
• no unconfirmed finding doubts what was asked (none at the ratings or intent levels, no current, voltage or inrush finding at the layout level);
• every ask is met: each wish carries machine-checkable asks sealed with it (patterns its design must match, parts by count, parts the wish rules out);
• the board’s area over the area its parts need is at most 1.3, or the wish’s own limit (1.0 for “small”).

The bar moved the sealed result from 19 of 20 to 10 of 20 in one step.

The sealed results

The latest run, on main 37bcfb582, finished 12 of 20 with a median of 47 tool calls and 462 seconds from the wish to the verified board. The re-check of the kept projects by the command line agrees (12 of 20). 18 of the agent’s 20 first designs compiled as written. It was also the first sealed run without our developer guide in the agent’s context: the agent’s work directory is now a git repository of its own.

The times are upper bounds. The server ran on a shared 96-core host (c8g.24xlarge) at a load of 120 to 560 from other teams’ benchmarks. PCBKit’s part maker was down during the whole run, so every request to make a part from a datasheet failed. The same 20 wishes have now been run four times; each time only the aggregates were read.

Where the agent fails

The column that went down matters as much as the one that went up. The strict one-shot, the agent’s first build complete as written, fell to 3 of 20. First builds now mostly end with an error the agent then fixes, at a median of 3 failed calls a conversation. Of the 8 boards that did not finish, the area went over the limit on 5 and findings doubted the asked quantities on 4. None of the 20 conversations finished in under 60 seconds or under 10 calls.

We cannot study the sealed failures, so the causes come from the dev set run on the same commit. There the unfinished boards were the same two kinds. Area: a BLDC driver at 2.36 times the area its parts need, a six-servo board at 1.49. Engine copper the check doubts: the router laid a 0.15 mm track for a motor supply that needs 2.8 mm at 5 A; a 6.2 A LED supply’s pour necked to 0.9 mm with a 103 °C rise. Both are the engine’s to fix: the agent asked for the right current and the engine’s own copper did not carry it.

The dev set’s first builds failed on two repeated patterns. Three agents wrote a bare near(X) without a maximum distance because they copied it from our quickstart (fixed there). Three hit a corner rule on a standard mounting hole the placer could not meet. On 12 dev wishes one run per wish moves the share a lot: 10 of 12 against 6 of 12 on two commits is within what one agent’s choices move, by a 95 % Wilson interval.

What comes next

The goal is 20 of 20, under a minute and at most ten calls a board (roadmap). A board with a processor now also needs its program: since ADR 0114 a build is not decided until each program is built, run in the lab at five corners and meets the expectation the design states. On the first dev run under that rule, one of seven processor boards ended with a program that met it. How the lab runs those programs hot, cold and at both ends of the supply is in running real firmware at the corners.

The benchmark’s current numbers are on the benchmarks page. The language the agent writes is described in the language reference. A first board by hand starts at /docs/start.