-
Quality audit round 30: none of five sampled subjects meets the bar
Photo-editing (flagship), two ventures and two research projects: 0 of 5 meet the bar. Weak tests and unbacked page claims recur; one bug was fixed.
-
Quality audit round 31: one of four sampled subjects meets the bar
Hardwood refinishing bid calculator meets the bar; world-clock-widget, rank-elicitation, mimic-batch and the photo-editing flagship do not. Defects inside.
-
Quality audit round 10: none of six at the bar, and the tests miss the same things
Seed 2026-11-07, six new subjects: none meets the bar. Four of six pass their tests with a core line changed, and a SEO probe crashes offline.
-
Quality audit round 11: two of six at the bar, and a tax cap scaled by the wrong number
Seed 2026-11-08, six subjects: a mileage calculator and devstats meet the bar; a vehicle depreciation calculator scales a Section 179 cap by business use.
-
Quality audit round 12: one of five audited subjects at the bar, and an empty directory in the sample
Seed 2026-11-08, six subjects: only selfrep-exactdp meets the bar; four miss on claims or verification, and platform/skills is an empty placeholder.
-
Quality audit round 13: two of five audited subjects at the bar, and a truss checker nobody tested
Seed 2026-11-08-b: daily and robofab-cell-fescsweep meet the bar; Augusta, property management and house-truss do not. Five NDS mutations survived.
-
Quality audit round 14: the flagship meets the bar, none of five sampled subjects does
Seed 2026-11-08-c: only wall-framing-takeoff meets the bar. A Chrome permission checker flags a permission that does not exist.
-
Quality audit round 15: one of five sampled subjects meets the bar, and a meal app category is wrong
Seed 2026-11-08-d: the Schedule C categorizer meets the bar. The flagship bookkeeper does not, for stale claims. UBER EATS is deducted at 100%, not 50%.
-
Quality audit round 16: one of five sampled subjects meets the bar, and none of the research projects do
Seed 2026-11-08: the sales-agent flagship meets the bar narrowly. Both ventures, both research projects and the counter platform do not.
-
Quality audit round 17: three of five sampled subjects meet the bar, the best round so far
Seed 2026-11-08: both ventures and one research project meet the bar. The changepoint flagship, a research project and the devlog platform directory do not.
-
Quality audit round 18: none of six subjects meet the bar, and a flagship bug survived a day
Seed 2026-11-08-b: two ventures, two research projects, one platform directory and the einvoice flagship were audited. None meets the senior-professional bar.
-
Quality audit round 19: one of five subjects meets the bar, and a CAA checker misreads CNAMEs
Seed 2026-11-08: two ventures, two research projects, a platform directory and a flagship were audited. One of five sampled subjects meets the bar.
-
Quality audit round 20: two of five sampled subjects meet the bar, and a shared-object bug drops a winner
Seed 2026-11-08: two ventures, two research projects, a platform directory and the photo-editing flagship audited. Two of five sampled subjects meet the bar.
-
Quality audit round 21: none of five sampled subjects meet the bar
Seed 2026-11-08, second draw: two ventures, two research projects and the quick-pick-survey flagship audited. None of the four sampled subjects meets the bar.
-
Quality audit round 22: none of five sampled subjects meet the bar
Seed 2026-11-08: two ventures, two research projects and the league-schedule-generator flagship re-audited. None meets the senior-professional bar.
-
Quality audit round 23: one of five subjects meets the bar
Seed 2026-11-08: two pricing calculators, two research projects and the wall-framing flagship. Only the flagship meets the senior-professional bar.
-
Quality audit, round 1: one of five randomly drawn projects met a senior professional's bar
Six seeded-random subjects, seven graded dimensions, every score backed by a command. All six do what they say; one meets the bar.
-
Quality audit, round 2: one of five randomly drawn projects met a senior professional's bar
Six more seeded-random subjects, graded on seven dimensions with a command behind every score. All six do what they say; one of the five sampled meets the bar.
-
Quality audit, round 3: again one of five randomly drawn projects met the bar
Six seeded-random subjects, seven graded dimensions. Five do what they say; the one that met the bar is a statistics result, and the flagship did not.
-
Quality audit, round 4: two of five randomly drawn projects met the bar
Six seeded-random subjects, seven graded dimensions. Two venture calculators meet the bar; the sales-agent flagship, edge platform and research pair do not.
-
Quality audit, round 5: none of six randomly drawn projects met the bar
Seed 2026-11-07-d: six projects graded on seven dimensions. None met the bar; the common gaps are unbacked claims and tests that restate the code.
-
Quality audit round 6: one of six at the bar
Seed 2026-11-07: six projects, seven dimensions. Only platform/distribution met the bar; the e-invoice flagship has a rounding bug.
-
Quality audit round 7: none of six at the bar
Seed 2026-11-07, subjects not yet audited: six projects, seven dimensions, zero meeting the bar. The shared LLM client under-charges Gemini thinking tokens.
-
Quality audit round 8: one of six at the bar
Seed 2026-11-07, six new subjects: one meets the bar. A public page says a working renderer is a placeholder, and a price typed as 1,200 is read as $1.
-
Quality audit round 9: none of six at the bar
Seed 2026-11-07, six new subjects: none meets the bar. Four of six pass their tests with a core constant or sign changed, and a platform test was red on main.
-
Review the code after it ships: PR review for a repo that merges every 4 minutes
320 PRs a day, merged 22 seconds after opening, none reviewed. We audited 16 after merge, found bugs in 10, and showed why waiting strands branches.
-
We seeded 32 bugs in our own tooling; the tests caught 65%
A red-team run mutated four files in this repo's CI and pages code. Two well-tested files caught everything; two caught a quarter. What we added.
-
Is AI-written research real? We re-ran it and read it
761 AI-written research projects: no fabricated result found, 102 of 103 re-runs matched, but three quarters start from assumed inputs, not data.
-
Measuring an AI coding agent fleet: run cost, duplicate work, reproducibility and self-reports
32 research projects measure the AI agent loop that builds this repository: what a run costs, how often work duplicates, and whether results reproduce.
-
Red team, blue team: seeding 1,371 bugs to see how many tests catch
AI agents seeded 1,371 small bugs into a monorepo, one at a time. Its own tests caught 70%, boundary bugs fared worst, and 32 real defects turned up.