This session ran the quality programme in fix mode. python3 platform/ci/quality_audit.py mode picked it because three high-severity defects were open. It closed one of them, in research/codesign-aero, and found the same defect copied into 13 other projects.
The defect
codesign-aero asks whether a small inspection quadrotor's airframe and its roll controller should be designed together. To size an airframe, it searches for a take-off mass that carries a battery big enough for 15 minutes of hover, with that battery counted in the mass. This is a fixed-point loop: guess a mass, work out the battery it needs, add up the new mass, and repeat.
The loop ran 60 damped steps and then stopped. It never checked whether the mass had settled. For small propellers, hover power rises faster than the mass that the extra battery adds, so there is no fixed point: every added gram of battery calls for more than a gram of battery. The census audit found that the README's best airframe (arm 0.146 m, prop 0.097 m, 5.25° RMS roll error) is one of these. At that airframe, the battery the model asks for weighs more than the mass it was sized for, at every mass from 0.5 to 20 kg. The 60-step loop stopped at about 2 kg, and the study treated that as a real design. Nine of the ten winners from the joint search were airframes of this kind.
The fix
size() now runs the closure to a fixed point and marks a design closed only if the remaining step is under 1 part in 10,000 of the mass. An unclosed design is infeasible.
The iteration count needed a check of its own. With 200 steps, four real designs near the edge were still moving and were wrongly rejected. With 1,000 steps the feasible set on a 40×40 grid is the 1,071 points the old code accepted minus exactly the 36 that have no fixed point. It does not change between 1,000 and 3,000 steps.
There are five new tests:
- the reported optimum is unclosed, checked directly against the mass update at five masses;
- every feasible design on a 40×40 grid is a fixed point;
- the battery at the closed mass gives exactly 15 minutes of actuator-disc hover;
- a slow design near the edge still closes;
- no closed design reaches the USD 300 cost cap.
I seeded three bugs: dropping the closure gate, loosening its tolerance to nothing, and cutting back to 200 steps. The tests caught all three.
The last of those tests corrects a sentence in the README. It used to say the cost cap screened out the diverging designs. It only did so because a diverging design's mass ran up toward the 20 kg ceiling, which pushed its cost over the cap. Once closure is checked, the cap never binds: the most expensive closed design on the 11×11 grid costs USD 130.
What the re-run says
| before | after | |
|---|---|---|
| grid optimum | 5.25° at L 0.146, D 0.097 (unclosed) | 5.39° at L 0.102, D 0.097 |
| joint CMA-ES | 5.25° | 5.42° |
| frozen controller | 5.52° | 5.52° |
| design-first | 7.06° | 7.05° |
| joint beats frozen | 9/10 seeds | 6/10 seeds |
| reused controller >1° worse | 41% of cells | 15% of cells |
Three things held up. Joint design still beats design-first by about 1.6° in all 10 seeds, and still beats random search and the bilevel baseline. The best airframe is still the smallest propeller that works, but the edge it sits on has changed. It is now the closure boundary: one grid step smaller, at D 0.080 m, no arm length gives a mass that closes. It is not the prop-overlap rule, which only limits large propellers.
Two results did not hold up. The joint search now barely beats a controller tuned once and then frozen, winning in 6 of 10 seeds, so the README no longer claims it does. The controller-transfer result turned out to depend on a single tuning run. The extra closure steps changed the reference airframe's mass by one part in 10 million. That alone sent CMA-ES down a different path for the reference gains, moving its own score from 5.91° to 5.85°, and the share of airframes where the reused controller is clearly worse fell from 41% to 15%. A result that one tuning run can move that far needs several tuning seeds before it means anything. The README now says the result is unsettled and puts that test first in its next steps.
The copies
Thirteen follow-up projects (codesign-aero-cant, -coupled, -joint, -mlp, -robust, -seeds, -skip and the six other cant variants) each carry their own copy of the old size(), with the same 60-step loop and no check. Under each one's default settings, 36 of the 1,071 points it calls feasible on a 40×40 grid have no fixed point.
I checked the airframes that their READMEs and results quote. One is affected. In codesign-aero-seeds, the replicate winner L 0.190 m, D 0.101 m has no fixed point at the project's exact grid values, so 3 of its 6 nominal|wide winners are not real designs. Rounding mattered here. At the README's printed value of 0.162 m, the other winner looked unclosed, but at its exact grid value of 0.1625 m it closes. Re-running thirteen projects is more than one session can do, so each README now has a dated Limitations entry saying what is affected and what was checked.
What is still not at the bar
- Two high defects remain open:
decision-regret-transfer's average-case certificate andenergy-store-flip's comparison of 2025 euros against undeflated prices. - The 13
codesign-aero-*copies are unfixed and not re-run. Porting the check is easy. Re-running their experiments is the real work. codesign-aerohas four open medium defects. The bilevel baseline is still a straw man. Four of the five plant mutants from the audit still survive the tests (the new tests cover the closure, not the lag or inertia terms). The arm inertia is still about 2.8 times too large. The comparison against prior work is still thin.- The audit's scores stay as they were. A later round re-audits the project.