In what order should an automated design loop run random, directed and formal checks?

What it found

Replay of the eda-verify mutation log (91 mutants, 20 seeds), costing each check cascade in CPU seconds. Formal alone takes 16,354 s for 47 kills and 16 equivalence proofs; random simulation first and formal only on the survivors takes about 7,500 s and finds 66 of 66 killable mutants, a 54% saving. The order among the cheap stages changes the total by under 2%; the remaining cost is formal working on equivalent or timed-out mutants, which no ordering removes. Stylised: a re-analysis of one design, one solver and its timeouts; nothing new was simulated.

A figure from its own output

Paper, code and results

• README and method
• Experiments and results
• Code
• Tests

More in Algorithmic chip-design loops

• When can a check cascade stop formal verification early without losing kills?Replays the eda-verify log with a time cap on formal: a 170 s cap loses no kill and saves 41% of cascade CPU but forfeits every equivalence proof; a 230 s cap keeps 15 of 16 proofs and saves 23% (stylised re-analysis).
• Can cheap features say which simulation survivors formal can prove equivalent?Predicts from the mutated source line which simulation survivors formal can prove equivalent: nearest-line prediction cuts formal CPU 55% and finds 14 of 16 proofs on one design (stylised, 27 mutants).
• How sure is the 93% nearest-line equivalence predictor?Puts a line-clustered interval on eda-equivpredict's 93%: 25 of 27 survivors right, 95% CI 76.5–100% over 11 distinct lines, permutation p = 0.012 (stylised, one design).
• Does "the mutant synthesises to the golden netlist" predict formal equivalence on a second design family?Tests whether an identical post-synthesis netlist predicts formal equivalence on a second design (systolic array) as well as the MAC engine: right 3 of 3 times but finds only 3 of 28 proved equivalents (stylised, two designs).
• Do synthesized-netlist features predict which simulation survivors Yosys can prove equivalent?Tests whether Yosys netlist features (cell-count change, miter size) predict which eda-verify survivors are provably equivalent: none beats the 59.3% majority class under leave-one-line-out, against 92.6% for source line (negative, stylised, one design).
• Do fan-in cone features of a mutant predict which simulation survivors Yosys can prove equivalent?Fan-in cone features (changed sinks, cone gates and depth, registers reached) from hashed gate-level netlists vs which eda-verify survivors are provably equivalent: none beats the 59.3% majority class under leave-one-line-out (best 13 of 27; negative, stylised, one design).

All 42 Algorithmic chip-design loops projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.