Blog ·
PCB design review on 462 real respin faults: KiCad DRC vs PCBKit
Sergey Alekseev
This is a PCB design review benchmark built from real mistakes. We collected 462 defects that people shipped on a KiCad board, found and fixed in the next revision, then ran every checker a hardware team can run today on both revisions: KiCad 10’s own DRC and ERC with the project’s rules, an open-source KiCad review action, an LLM review of the netlist and PCBKit. The question is simple. Of the faults that really cost someone a respin, how many does each checker name before the order? How often does it keep complaining after the fix?
The short answer: PCBKit finds as many as KiCad’s checks or more, with a tenth of its false alarms on the fixed boards. All of them together still miss about four faults in five.
How we collected the faults
The pool is 2,377 fix commits mined from public KiCad repositories: a revision A with a defect and the commit B that fixed it. We split it by repository and sealed it on 28 October 2026: 698 cases for development, 1,679 sealed. The miner’s labels are a guess (70 to 85 % right in earlier hand checks), so every case we use was read and adjudicated by hand.
For each case the reader sees the whole fix commit message, the issues it names and the difference between the two revisions: values, lands, flipped parts, pins whose net changed, parts added or removed. A case is admitted when the evidence says revision A had a defect a built board would carry and B fixes it, with the defect’s class and its site (the parts or nets a check must name). Rejected: design changes and features, preference (“more suitable”), renames, layout tweaks, DRC cleanups, work in progress.
A blind re-adjudication of 30 random cases by a second team agreed on 23 of 30 decisions. Of the seven differences, four were evidence the second reader did not have and three were the rule. That gave the rule three tests, two of them mechanical:
| Test | What it says |
|---|---|
| F1 | Pin-net changes between automatic names (Net-(R1-Pad2)) are not rewiring; the site is what is left that the message names |
| F2 | A footprint defect needs the copper to differ between A and B: same pads under a new name is a rename |
| F3 | A value fault is stated (“wrong”, “should be”, a broken rating), not wished (“more margin”, “probably”) |
F2 matters more than it looks. A board-file checker that judges a land by its name scores on cases where the built board is identical. One of our own prototype rules did exactly that until F2 took those cases out.
Under these rules the development halves hold two sets: set 1 with 140 defects in 138 cases and set 2 with 322 defects in 316 cases. Footprints are the largest class (142 of set 2’s defects), then pinouts (67), connectivity (45), polarity (34) and wrong values (29). The sealed halves (177 and 315 defects) have not been opened.
The rule every checker is scored by
A checker finds a defect when one of its checks names the site on revision A and the same check does not name it on revision B. Then a judge decides whether the finding states the defect: that a person reading it would learn what is wrong and make the fix. A generic violation that happens to sit at the part (a clearance, a mask opening, a courtyard) does not count.
The judge is Claude Haiku 5.5. It sees the defect, the fix’s reason and the tool’s finding texts at the site, without the tool’s name or check codes. We labelled a random 30 by hand first: this strict judge agreed on 29 of 30. A looser first prompt agreed on 25 of 30, every disagreement in the checker’s favour, so the strict one is the number we publish.
The other half of the score is the fixed board. If a checker still flags revision B at the site with a check of the defect’s kind, it cannot tell the person the fix worked.
KiCad DRC versus PCBKit on 462 faults
Each tool ran unmodified on dev2 (96 cores, shared), 32 jobs at a time. KiCad 10.0.7 ran
kicad-cli pcb drc with schematic parity and kicad-cli sch erc with each project’s own settings
and rules. The review action ran as its GitHub Action runs it. The LLM got each revision’s parts and
netlist with a plain reviewer prompt. PCBKit read the board and, in the latest run, the schematic
sheets too.
| Checker | Reads | Found, set 1 (140) | Found, set 2 (322) | Fixed board still flagged |
|---|---|---|---|---|
| PCBKit | board and schematic | 17.9 % | 8.4 % | 4.3 % / 3.7 % |
| KiCad 10 DRC + ERC + schematic parity | project, schematic, rules | 9.3 % | 6.8 % | 55.0 % / 51.9 % |
| An open-source KiCad review action | schematic and board | 0.0 % | 0.6 % | 10.0 % / 9.9 % |
| An LLM netlist review (Claude Sonnet, low effort) | parts and netlist | 4.3 % | 4.7 % | 34.3 % / 35.7 % |
Intervals are Wilson 95 %. On set 1 PCBKit is ahead (25 defects against KiCad’s 13, intervals just touching). On set 2 the intervals overlap: 27 against 22 is close to a tie.
The fixed-board column is where the two differ most. KiCad’s library-link, footprint-field and clearance checks fire on most parts of most boards: 95 to 98 % of fixed sites carry some KiCad violation. Between 52 and 55 % carry one of the defect’s own kind. PCBKit’s class codes still fire on 3 to 4 % of fixed sites. Counting any finding at all, PCBKit is at 45 to 50 %, so it is far from silent either.
What each checker catches
The two main checkers find different faults. KiCad’s catches come from the schematic: parity errors (“pad net doesn’t match net given by schematic”) on swapped SDA/SCL, swapped CAN RX/TX or a reversed diode, footprints that don’t match the symbol’s, a value field that differs and ERC’s unconnected pins. PCBKit’s come from the part data: polarity against the rail, a pinout against the part, values against the circuit (a bootstrap capacitor, a feedback divider, a crystal’s load) and a land against the bought part’s package.
With the schematic sheets PCBKit and KiCad overlap on 8 defects in each set. Together they find 30 of set 1 and 41 of set 2.
The review action’s 65 detectors found 0 and 2. Its hits at the sites are layout and EMC heuristics (courtyards, narrow traces, ground pins on connectors): useful in a review, rarely a fault.
The LLM reads swaps and missing pull-ups well (SDA/SCL, USB D+/D-, a reset tied low, divider ratios). It also reports a median of three “errors” on every board and repeats most of them on the fixed one, which is why its fixed-board rate is over a third. The whole run cost $9.20 for 908 board revisions at batch price; KiCad took a median 10.2 s per revision (p95 21.7 s) on the loaded host.
Most respin faults escape every checker
All four checkers together find 16 to 21 % of these defects. That is the number we would most like to change. It is also the reason we publish it.
We went through every missed defect in set 1 and wrote down the check that would catch it and the facts it needs. The ceiling is sobering:
| With | Set 1 defects a check could reach |
|---|---|
| rules on the board file alone | 27.9 % |
| plus the part’s own facts (pin table, package drawing, datasheet values) | 63.6 % |
| plus the project’s schematic and firmware | 66.4 % |
| plus the bill of materials’ bought parts | 82.1 % |
The remaining 17.9 % are wrong only against something no repository holds: the cable the connector mates with, the screw the hole is for, the filter corner the designer wanted. A JST harness reversed against its board is self-consistent on the board. No checker of the board and its project will reach 95 % on human boards. A design that states its intent can reach further, which is part of why the PCBKit language lets a design say what a connector mates with and what current a part must carry.
Most of the gap between today and that ceiling is part data. On the earlier respin set only 16 % of the parts a fix touched named a manufacturer part number at all. A catalogue’s package text is often just “SMD”. This is the work behind part certificates: reading the land and pin table from the maker’s own drawing.
What PCBKit shows you today
Not every finding above reaches the person yet. A new check’s findings stay unconfirmed until it clears a blind round on boards nobody tuned it on. Of PCBKit’s judged catches with the schematic, a confirmed check fired on 7 in set 1 and 12 in set 2. The rest are true findings still on probation.
The probation is what keeps precision honest. In the latest blind round a separate team judged 702 findings on boards nobody tuned the checks on: 5.7 % were false, down from 35.7 % four rounds earlier. The precision rounds and this recall set pull in opposite directions on purpose: a check that fires everywhere finds more faults and fails the fixed-board test.
How to read these numbers
The defects are what open-hardware authors wrote down and fixed, so they lean to part data and wiring. Signal integrity, thermal and EMC faults rarely appear in a commit message. The picks favour clear messages. Cases cluster by repository, so the Wilson intervals are somewhat narrow. We have not measured commercial checkers: each needs a licence or uploading these repositories to a service.
The development halves are what we tune on. The sealed halves are measured by a separate team and only as totals, so a check that works only on what we read will show up there. The per-tool results and the method are on the verification page; routing has its own benchmark against other tools in Ahead of 12 routers on their own benchmarks. The CI gate runs these same checks on every pull request.