How do design proposals from a language-model agent compare with CMA-ES, Bayesian optimisation and random search at an equal simulation budget?

research/simdesign-llm

Session-agent design proposals vs CMA-ES, Bayesian optimisation and random search on a bracket sized by finite elements, 40 evaluations each: agent 23.0 g vs 32–39 g baseline medians (best known 22.4 g); the agent had prior knowledge, n = 1, not an LLM benchmark. Stylised.

6 tests

What it found

The finite-element bracket of simdesign-bracket, with tighter limits (15 MPa, 2.5 mm), is sized for minimum mass with 40 simulator calls per method. The proposer is the session agent itself, writing four rounds of ten designs by hand from the scores (no API key; not a reproducible LLM benchmark). It reaches 23.0 g against medians of 32.2 g (Bayesian optimisation), 33.3 g (CMA-ES) and 39.0 g (random) over 20 seeds, with 22.4 g best known from 1,845 evaluations; no baseline seed beat it. Stylised and weak evidence: n = 1, the agent used mechanics and the sibling project’s results as a prior, baselines were untuned, the stress constraint is mesh-sensitive, assumed material.

A figure from its own output

Figure produced by research/simdesign-llm: curves — simdesign-llm/experiments/curves.png, written by the project's own experiment code.

Paper, code and results

More in Simulation-first design

What do manufacturing rules cost a sized bracket on FDM, CNC routing, laser cutting and casting?Manufacturability rules (FDM, CNC routing, laser cutting, sand casting) added to the sized L-bracket: the mass price of each rule set against the same material with no rules (casting +43% from its even-section rule, CNC +24% from plate stock, laser and FDM ~0); designs overshoot the stress limit by 17–26% on a fine mesh (stylised, assumed rule values).
How much does a bolt hole cost a tension plate, and which rule sets its width?Tension plate with a required bolt hole sized under net-section, Kt = 3 and finite-element stress models and per-process hole rules (stylised).
Does weighting parts by mass change which shop process, and which machine, matters?Does weighting parts by mass change which shop process, and which machine, matters? With a tip load FDM covers 0.30 of parts by count and 0.58 by mass, the laser-versus-CNC order flips, and at budgets 600–800 exhaustive search picks the laser cutter by count but the casting setup by mass (one seed). Stylised, assumed rules and prices, simulated only.
Do the mass-weighting machine-choice flips of simdesign-coverweight survive across seeds?Do the mass-weighting machine-choice flips of simdesign-coverweight survive across seeds? Over 20 seeds, log flips from the laser cutter (count) to the casting setup (mass) in 20/20 at budgets 600–800, large only at 800, small never; the 1100–1400 flips are near-ties (≤0.26 points). Stylised, assumed rules and prices, simulated only.
How wide are the budget bands where mass weighting changes the machine choice when prices are uncertain?How wide are the budget bands where mass weighting changes the machine choice when prices are uncertain? With ±20% independent price noise the log flip band widens from budgets 600–800 to 500–1000 but softens at its edges (about one half), and large's single nominal flip at 800 shows up in only 44% of price draws. Stylised, assumed rules and prices, simulated only.
Which share of a bracket family can each shop process make, and which machine should a shop add next?Share of a sampled bracket family each shop process can make under the simdesign-dfm rules: geometry alone adds nothing beyond FDM (negative), with a tip-load requirement the processes separate (FDM alone 30%, +casting +laser 41%) and the best next machine depends on budget and part size, where greedy sometimes loses 1–2 points to exhaustive search (stylised, assumed rules).

All 64 Simulation-first design projects →

Other research topics

Proper scoring & elicitation28 Effort & peer prediction9 Verification games & slashing19 Detection & tolerances14 Markets as routers13 Prediction-market design9 Paying contributors8 Decentralised training systems23 Digital twins & sim-to-real164 LLM chip design loops40 Robot manufacturing loops53 Ambient AI assistant46 Self-replicating manufacturing34 Simulation-first design64 Engineered clothing45 Task-specific AI accelerators40 Algorithmic chip-design loops42 Open-source purpose-built robots44 The software-engineering loop43 Adversarial human imitation41 Open-source residential construction41 Autonomous multi-angle capture34 Sensing, imaging and 3D31 Energy supply and dense energy storage39 Robot societies among people31 Agentic co-design of low-cost robots27 Laundry-hanging robots25 Machine learning on qualitative data23 Robots on the building site27 Civilizations & group psychology1 History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.