How many texts from one source does a weak human-vs-machine text detector need before a batch verdict is reliable?

What it found

The mimic-contexts n-gram detector scores a held-out RAID family; a batch is k independent texts and its statistic is their mean score. One text separates people from a plainly prompted LLM at AUROC 0.935 (mean of 8 families); four all-machine texts reach 0.997 and eight reach 1.00. If only half the batch is machine, k = 8 gives 0.98; a quarter needs about 32 (0.985); at 10% machine 32 texts give only 0.82. The catch is calibration: with a threshold set on 150 human texts, the share of human batches flagged drifts from 7.4% at k = 1 to 15% on average at k = 32 (books 60%), because the batch mean's spread shrinks while the calibration error does not. Preliminary: plain-prompt text, i.i.d. batches, one detector, no LLM run.

A figure from its own output

Paper, code and results

• README and method
• Experiments and results
• Tests

More in Adversarial human imitation

• Does a classifier two-sample test catch digital-twin errors that a paired trajectory error misses?Does a classifier two-sample test catch digital-twin errors that a paired trajectory error misses? Stylised pendulum: boosted-tree AUROC 0.72 for a friction error whose RMSE is a quarter of the sensor noise, 0.92 for Gaussian-vs-Student-t noise at zero paired error. Preliminary, simulated only.
• Does a kernel MMD test catch the same wrong digital twins as a classifier two-sample test?Does a kernel MMD test catch the same wrong digital twins as a classifier two-sample test? Stylised pendulum: raw-trajectory MMD is blind to a Coulomb-friction error (power 0.05 to 0.06) that boosted trees catch from n = 200, but MMD on second differences detects a +25% noise-level error at 0.97 from n = 100.
• Does a classifier two-sample test see the integrator, timestep and friction gaps of a MuJoCo digital twin?Does a classifier two-sample test see the integrator, timestep and friction gaps of a MuJoCo digital twin? Stylised pendulum: AUROC 0.98 for Euler at 0.02 s, 0.53 (null) for a paired error of 0.003 rad, 0.92 for Gaussian-vs-Student-t noise at zero paired error. Preliminary, simulated only.
• Does a kernel MMD test catch the same wrong MuJoCo digital twins as a classifier two-sample test?Does a kernel MMD test catch the same wrong MuJoCo digital twins as a classifier two-sample test? Stylised MuJoCo pendulum: boosted-tree C2ST power 1.00 at n = 200 for Euler at 0.02 s and a noise-shape error where MMD stays near 0.05; MMD on second differences catches a +25% noise-level error; friction errors undetected by all at n <= 200.
• Does a digital twin that a classifier can tell apart from the real system also make a worse controller?Does a digital twin that a classifier can tell apart from the real system also make a worse controller? Stylised pendulum: 6 of 15 wrong twins have zero controller regret at C2ST AUROC 0.58 to 0.999; the worst regret (3.8%) comes from a twin with no sensor noise and zero paired error. Preliminary, simulated only.
• On a swing-up task where friction matters, does a digital twin's detectability predict what tuning on it costs?On a swing-up task where friction matters, does a twin's detectability predict what tuning on it costs? Stylised pendulum: only the two gravity-error twins cost much (3.7% and 8.3% extra real cost); 13 others cost under 0.3% at C2ST AUROC 0.51 to 1.0. Preliminary, simulated only.

All 41 Adversarial human imitation projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.