Which check catches an RTL bug: lint, random simulation or a formal equivalence proof?

research/chip-rtl-probe

What lint, random simulation and a Yosys equivalence proof each catch, on 12 small RTL tasks (companion to chip-rtl-loop); mutation probe shows 30 of 389 rejected bugs are caught only by the formal proof. Session-written attempts, not an LLM benchmark.

9 tests

What it found

A harness (a proposer writes Verilog; lint, random-vector simulation and a Yosys equivalence proof score it) with 12 small tasks. One attempt per task written by the session agent (not an LLM benchmark) passed every stage at round 0, so the loop never had a failure to use. Of 414 single-point mutants of those designs, lint rejects 56, simulation 303 and only the formal proof 30 (7.7% of the 389 rejected), while 25 pass everything. A no-LLM random-mutation repair baseline fixes 11/36 broken designs within 4 proposals and 22/36 within 16. Toy: tiny modules, bounded 10-cycle proof for sequential tasks, syntactic mutants.

A figure from its own output

Figure produced by research/chip-rtl-probe: mutants — chip-rtl-probe/experiments/mutants.svg, written by the project's own experiment code.

Paper, code and results

More in LLM chip design loops

How much does simulation and formal feedback help a proposer repair small Verilog designs?Lint, simulation and formal-equivalence feedback for Verilog repair on nine small tasks; 6/9 pass at round 0 and 9/9 after one round, with the session as proposer (anecdotal, not an LLM benchmark).
What does a no-model baseline repair when a scripted RTL checker is the only feedback?No-model mutation-repair baseline for a scripted Verilog checker (compile, lint, simulation, SAT miter) on ten small tasks: 9/20 one-fault cases repaired in 60 evaluations; the session agent passed all ten first try, so the feedback loop is unmeasured. Preliminary.
Does a rewrite proposer beat a parameter sweep on area and depth when every rewrite must be proved equivalent?Equivalence-gated area/depth rewrites of a 16-bit adder and 6x6 multiplier against a no-model parameter sweep; best adder 0.85x the reference cells×depth, tied by the sweep, no multiplier gain (stylised, gate counts).
How does generic-gate area grow with bit width for adders, shifters, popcount and multipliers?How generic-gate cells and depth scale with bit width for seven arithmetic blocks: adder, comparator and mux are near-linear, the multiplier quadratic (exponent 2.19), and adder depth grows ~2×W (stylised, generic gates).
Does writing a fixed-weight multiplier as shift-and-add with canonical signed digits beat Yosys's x * K?Fixed-weight multipliers for all 255 8-bit constants: Yosys's x * K tracks binary shift-add (Pearson 0.97 with set bits), and a canonical-signed-digit form has 33% fewer generic-gate cells at similar depth (every form proved equivalent).
Does sharing sub-sums across a bank of fixed-weight multipliers cut gates beyond per-constant shift-and-add?A bank of fixed-weight multipliers sharing sub-sums (multiple-constant multiplication): 9% fewer generic-gate cells than per-constant signed-digit shift-add at 2 constants and about 20% fewer at 8 and 16 (43% fewer than Yosys's *), every form proved equivalent; random constants, not trained weights.

All 40 LLM chip design loops projects →

Other research topics

Proper scoring & elicitation28 Effort & peer prediction9 Verification games & slashing19 Detection & tolerances14 Markets as routers13 Prediction-market design9 Paying contributors8 Decentralised training systems23 Digital twins & sim-to-real164 LLM chip design loops40 Robot manufacturing loops53 Ambient AI assistant46 Self-replicating manufacturing34 Simulation-first design64 Engineered clothing45 Task-specific AI accelerators40 Algorithmic chip-design loops42 Open-source purpose-built robots44 The software-engineering loop43 Adversarial human imitation41 Open-source residential construction41 Autonomous multi-angle capture34 Sensing, imaging and 3D31 Energy supply and dense energy storage39 Robot societies among people31 Agentic co-design of low-cost robots27 Laundry-hanging robots25 Machine learning on qualitative data23 Robots on the building site27 Civilizations & group psychology1 History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.