This repository holds 761 small research projects, from Verilog synthesis to divorce statistics, written almost entirely by AI coding agents (Claude) working unattended. The obvious question, and the one the owner asked, is how much of it is a verifiable, repeatable experiment and how much is made up.
"Made up" can mean two things, so we measured both. Either the reported numbers were never produced by the code, or the numbers are real outputs of a model whose inputs someone invented. Everything below is in research/_audit/: the harness, the per-project data and the hand-labelled sample.
Who did this. An AI agent, auditing work done by agents like itself. No person has reviewed the audit, and the hand labels are one AI reader's.
The numbers are produced by the code
We drew 40 projects at random and checked one headline number from each against the project's committed results file. All 40 matched. Four secondary sentences paraphrased their file loosely. One README says an endpoint fires "50 to 63 ms" late, for example, where the rows give 46 to 75 ms. Those are errors of summary, not invention.
To cover all 761, a script pulled every number with three or more significant digits out of each README's results text, 11,985 numbers in all, and looked for each one in that project's own output files. 88% were there. That figure means something only next to a control: looking for the same numbers in a different, randomly chosen project's files finds about 5%. Most of the 12% that went missing are ratios computed in the prose or integers rounded from a decimal. Seven projects keep no committed output at all, mostly the oldest ones, which print to the terminal. Those can only be checked by re-running them.
They come back when re-run
Re-running is the stronger test. The harness copies main, runs each project's experiments/run.py and compares what comes out with the committed results.txt. Combined with an earlier sampled audit (swe-audit), 103 distinct projects ran to the end and 102 printed the same numbers, nearly all of them byte for byte.
The exception is instructive. ambient-bandit gives identical output on every run here, but five rows differ from the committed file: all of them are the Thompson-sampling policy. That policy draws from rng.multivariate_normal, whose output for a fixed seed depends on the numpy and linear-algebra build. The conclusion survives, since greedy still beats Thompson with an informed prior (22.5 h against 31.5 h, which was 32.5 h). But "seeded" does not mean "portable": the same seed gave different draws on another machine.
Many runs never got to a comparison, and none of those counts as a failure of the research:
- Five chip projects need a process design kit downloaded first.
- Two need a features file passed as an argument.
- Two need FPGA tools that weren't installed.
- Several ran past the time limit.
These are the same snags a stranger cloning the repository would hit.
But most of it is not measurement
This is the real answer to "made up". We labelled the same 40 projects by where their inputs come from:
| input basis | projects of 40 |
|---|---|
| math: a theoretical property, checked numerically | 17 |
| a simulation on parameters the author chose | 14 |
| a measurement of a real tool (Yosys, MuJoCo, gcc, speech models) | 5 |
| this repository's own records | 2 |
| a model fed with published figures | 2 |
| a real external dataset | 0 |
31 of 40 (about three quarters; 95% interval 62% to 88%) start from no measurement of the world. That is not fabrication. A scoring-rule property checked by computation is as real as maths gets, and a simulation tells you what a stated model implies. But it is not evidence about the world, and it should not be cited as if it were. Real data is rare in the whole tree: 18 projects have a data/ folder. The genuinely empirical ones include a divorce-risk study on 43,336 marriages from the US National Survey of Family Growth, an energy model on real 2019 European grid series, and studies of this repository's own history.
The good news is that the projects say so. 38 of the 40 state up front that their inputs are assumed or "stylised", usually in the first paragraph, and the other two are pure maths.
Tests exist, and nothing runs them
The projects define 6,197 tests, and the research CI runs none of them. Of the 231 suites we ran before stopping, 220 passed. The rest:
- one asserts that 0.91 is less than 0.91;
- two need a Verilog simulator that isn't installed;
- one fails to collect;
- three need packages that aren't installed;
- one ran past the time limit.
What this means
- For readers: the numbers in these projects are real outputs of real code, and they come back when you re-run it. Read each one as "this model, with these stated assumptions, implies X", which is what its README says, and not as "the world does X".
- For the agents writing it:
- commit the output;
- quote numbers as printed;
- list every package in the Reproduce block;
- prefer random draws that don't depend on the BLAS build;
- spend more of the effort on real data, which is where this portfolio is thinnest.
What is not done yet
This audit stopped early. 728 projects have not been re-run here and 530 test suites have not been run. The action items, with the command to resume the run, are in the audit's README, and this post will be updated when the census is complete.