How many random vectors does it take to see an RTL bug, and what can 1,280 clean vectors not rule out?

What it found

Detection-probability model of the eda-verify mutation log (91 mutants, 20 seeds). Random vectors catch 41 mutants on the first vector, 24 with a per-vector probability between 0.03 and 0.95, and never catch 25 (16 provably equivalent, 9 formal timeouts). A geometric first-failure model fits the observed miss rate within 0.005 at every budget up to 64 vectors, and predicts 45 vectors for a 1% miss rate and 118 for 0.1% (extrapolated beyond the data). The 1,280 clean vectors bound an undetected bug's per-vector probability below 0.0023 at 95%, so random testing cannot tell a rare bug from an equivalent mutant. Stylised: one design, 91 mutants, a tail set by 2–3 mutants; nothing new was simulated.

A figure from its own output

Paper, code and results

• README and method
• Experiments and results
• Code
• Tests

More in Algorithmic chip-design loops

• Can RTL checks that need no golden reference model catch the bugs a design loop injects?Five golden-free properties (latency, done pulse, output hold, start-glitch and reset-abort invariance) kill 22 of the 66 killable eda-verify mutants, all control-flow bugs, and none of the 15 ROM or other datapath bugs; the 16 directed vectors catch all 22 too.
• How many SKY130 mappings does a generic-gate screen need to find the best area × latency design?A generic-gate cells × cycles screen puts the best SKY130 area × latency design first among 40 MAC engines; over random pools of 20, mapping 3 designs finds the pool's best 97% of the time, against 15% for random choice.
• Does ABC's pre-layout delay and area view of 40 MAC engines survive placement and timing repair on SKY130?The 40 MAC engines placed with the pip OpenROAD wheel: ABC area still ranks placed area (Spearman 0.997+), ABC delay only 0.61 to 0.80, wires add a median 36% to the period, and no design ABC calls feasible at 5 ns meets it after placement.
• Does a cheap rung's rank correlation with the expensive rung predict how well it works as a top-k screen?Across 207 cheap screens of 40 MAC engines, Spearman agreement with SKY130-mapped latency does not predict top-k screening payoff (−0.13 at k = 1): the highest-Spearman screen (0.958) maps the best design first 32% of the time, gate count (0.551) 69%; for area × latency it does (0.60).
• If a design search only sees generic-gate numbers, does it find a worse SKY130 Pareto front?Searching generic-gate objectives instead of SKY130-mapped ones (RMS error 0.036 on area, 0.123 on latency) costs within 0.01 of hypervolume on the 40 MAC-engine designs, and TPE's +0.037 lead over random at budget 20 is unchanged.
• Can a cost model trained on other chip-design families screen a new family's placed area-period Pareto front, even though period is hard to predict?A screen with cost models trained on other families finds the placed (OpenROAD, SKY130) area-period front in 3 to 6 placements where random reaches 0.92 only at 16, but the true fronts hold only 1 to 2 designs, so the test is easy.

All 42 Algorithmic chip-design loops projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.