How many injected RTL bugs do random, coverage-directed and formal checks catch, and at what cost?
research/eda-verify
Mutation study of random, coverage-directed and bounded-formal RTL checks on the MAC accelerator: 64 random vectors kill 99% of killable mutants, formal 71% (timeouts), at ~50x the CPU cost (stylised).
What it found
Mutation study on the MAC accelerator: 91 compilable mutants (operator, literal, deletion and ROM edits), of which 66 are killable and 16 are proved equivalent by induction. 64 random vectors kill 99% of the killable ones, 16 coverage-directed vectors 97% (16 random: 95%), and bounded Yosys formal checking 71% because 19 SAT runs timed out. Cost per kill: about 1 s for simulation against 52 s for a formal kill. Stylised: one design, one bound, one solver.
A figure from its own output
Paper, code and results
More in Algorithmic chip-design loops
- In what order should an automated design loop run random, directed and formal checks?Replays the eda-verify mutation log to cost check cascades: random simulation before formal cuts CPU by 54% with more kills; the order among cheap stages barely matters (stylised re-analysis).
- When can a check cascade stop formal verification early without losing kills?Replays the eda-verify log with a time cap on formal: a 170 s cap loses no kill and saves 41% of cascade CPU but forfeits every equivalence proof; a 230 s cap keeps 15 of 16 proofs and saves 23% (stylised re-analysis).
- Can cheap features say which simulation survivors formal can prove equivalent?Predicts from the mutated source line which simulation survivors formal can prove equivalent: nearest-line prediction cuts formal CPU 55% and finds 14 of 16 proofs on one design (stylised, 27 mutants).
- How sure is the 93% nearest-line equivalence predictor?Puts a line-clustered interval on eda-equivpredict's 93%: 25 of 27 survivors right, 95% CI 76.5–100% over 11 distinct lines, permutation p = 0.012 (stylised, one design).
- Does "the mutant synthesises to the golden netlist" predict formal equivalence on a second design family?Tests whether an identical post-synthesis netlist predicts formal equivalence on a second design (systolic array) as well as the MAC engine: right 3 of 3 times but finds only 3 of 28 proved equivalents (stylised, two designs).
- Do synthesized-netlist features predict which simulation survivors Yosys can prove equivalent?Tests whether Yosys netlist features (cell-count change, miter size) predict which eda-verify survivors are provably equivalent: none beats the 59.3% majority class under leave-one-line-out, against 92.6% for source line (negative, stylised, one design).
All 42 Algorithmic chip-design loops projects →
Other research topics
Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1
Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.