Do 20,000 random vectors catch any RTL bug that 64 missed, and do they contradict the formal equivalence proofs?

What it found

A deep random draw on the 26 eda-verify mutants that 64-vector random testing never caught or flagged oddly (16 proved equivalent, 10 formal timeouts). 20,000 further uniform vectors catch none of the 25 survivors, so none contradicts a formal equivalence proof, and a random-visible bug among the 9 undecided mutants has per-vector detection probability below 1.4×10⁻⁴ (95%), sixteen times tighter than before. The one apparent failure, del@122, again fails exactly 10 vectors short of the draw, consistent with truncated output rather than a wrong value (not confirmed). Stylised: one design, uniform random inputs, same vector stream for every mutant.

A figure from its own output

Paper, code and results

• README and method
• Experiments and results
• Code
• Tests

More in Algorithmic chip-design loops

• Can a library-free delay estimate rank accelerator designs by SKY130 critical-path delay better than logic depth?Library-free delay proxies on the generic gate netlist of the 40 MAC engines: gate-type weights leave depth's Spearman 0.611 against SKY130-mapped delay unchanged (0.609), adding fan-out lifts it to 0.83 to 0.90, but gate count alone reaches 0.911, and no proxy recovers the true (area, latency) front.
• How many random vectors does it take to see an RTL bug, and what can 1,280 clean vectors not rule out?Per-vector detection probabilities of the eda-verify mutants: a geometric model fits the random-vector miss rate within 0.005, predicts 45 vectors for a 1% miss, and 1,280 clean vectors cannot rule out a bug rarer than 0.0023 per vector (stylised re-analysis).
• Can RTL checks that need no golden reference model catch the bugs a design loop injects?Five golden-free properties (latency, done pulse, output hold, start-glitch and reset-abort invariance) kill 22 of the 66 killable eda-verify mutants, all control-flow bugs, and none of the 15 ROM or other datapath bugs; the 16 directed vectors catch all 22 too.
• How many SKY130 mappings does a generic-gate screen need to find the best area × latency design?A generic-gate cells × cycles screen puts the best SKY130 area × latency design first among 40 MAC engines; over random pools of 20, mapping 3 designs finds the pool's best 97% of the time, against 15% for random choice.
• Does ABC's pre-layout delay and area view of 40 MAC engines survive placement and timing repair on SKY130?The 40 MAC engines placed with the pip OpenROAD wheel: ABC area still ranks placed area (Spearman 0.997+), ABC delay only 0.61 to 0.80, wires add a median 36% to the period, and no design ABC calls feasible at 5 ns meets it after placement.
• Does a cheap rung's rank correlation with the expensive rung predict how well it works as a top-k screen?Across 207 cheap screens of 40 MAC engines, Spearman agreement with SKY130-mapped latency does not predict top-k screening payoff (−0.13 at k = 1): the highest-Spearman screen (0.958) maps the best design first 32% of the time, gate count (0.551) 69%; for area × latency it does (0.60).

All 42 Algorithmic chip-design loops projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.