Does a prompt-free human-vs-machine detector carry over to a generator or decoding it never saw?

What it found

The mimic-contexts n-gram detector fit on all eight RAID families, scored on text from a chat model or decoding left out of the fit (items split in two halves by source_id, halves swapped). AUROC falls from 0.991 to 0.978 (llama-chat), 0.983 to 0.971 (mistral-chat) and 0.981 to 0.959 (mpt-chat); leaving out sampling decoding costs 0.984 → 0.936 and greedy 0.989 → 0.964, with held-out wiki the weak spot (0.83 on sampling). Averaging the score over k same-source texts lifts AUROC from 0.971 (k=1) to 0.999 (k=4). Human batches are more varied than model batches (recipes distinct-2 0.589 against 0.373 for llama and 0.399 for mistral), but mpt-chat matches or exceeds human variety on poetry and reddit. Preliminary: plain-prompt RAID text, three chat models each present in only some families (generator and family partly confounded), one detector type, no LLM run.

A figure from its own output

Paper, code and results

• README and method
• Experiments and results
• Tests

More in Adversarial human imitation

• Does adding a zero-shot perplexity score to the n-gram detector help it catch machine text in families it never saw?Does adding a zero-shot perplexity score to the n-gram detector help on unseen families? The two scores are uncorrelated within class, but averaging them lifts held-out AUROC only from 0.926 to 0.935 (paired gain +0.009, interval −0.010 to +0.026); fitted stacks do no better. Preliminary, 720 held-out texts, small scorer, no LLM run.
• How much text does a prompt-free human-vs-machine detector need, and what does it key on?How much text does a prompt-free detector need, and what does it key on? The mimic-contexts n-gram detector on held-out families: AUROC 0.65 at 50 characters, 0.84 at 400, 0.93 at 1500; stripping punctuation or shuffling words costs at most 0.014. Preliminary, plain-prompt RAID text only.
• If a human-vs-machine detector's threshold is set for a 5% false-positive rate, does that rate hold on a kind of text it never saw?If a detector's threshold is set for a 5% false-positive rate, does it hold on an unseen text type? Leave-one-family-out with the mimic-contexts n-gram detector: realised rate 0.00 to 0.54 (abstracts 0.54, books 0.18), and recipes catch 1.7% of machine text. Preliminary, plain-prompt RAID text only.
• How many human texts does it take to re-set a detector's threshold for a 5% false-positive rate on a new kind of text?How many human texts does it take to re-set a detector's threshold for a 5% false-positive rate on a new kind of text? About 80: with the plain 95th percentile k = 20 gives a mean false-positive rate of 9% (36% chance above 10%), k = 80 gives 6.1%; a conformal rule holds 4.9% but costs recall and needs 19+ texts. Stylised, RAID text, one n-gram detector.
• How many texts from one source does a weak human-vs-machine text detector need before a batch verdict is reliable?How many texts from one source does a weak human-vs-machine detector need before a batch verdict is reliable? Four all-machine texts give AUROC 0.997 (one text: 0.935); a quarter-machine source needs about 32; but a threshold set on 150 human texts drifts from 7% to 15% false flags at k = 32. Preliminary, RAID text, one n-gram detector.
• Does a classifier two-sample test catch digital-twin errors that a paired trajectory error misses?Does a classifier two-sample test catch digital-twin errors that a paired trajectory error misses? Stylised pendulum: boosted-tree AUROC 0.72 for a friction error whose RMSE is a quarter of the sensor noise, 0.92 for Gaussian-vs-Student-t noise at zero paired error. Preliminary, simulated only.

All 41 Adversarial human imitation projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.