How well do prompt-free detectors tell people from a plainly prompted LLM, and do they carry over to unseen kinds of text?

What it found

Eight text families from RAID (300 human and 300 plain-prompt chat-LLM texts each), split by family: train on five, hold out poetry, reviews and wiki. Prompt-free detectors on the held-out families: character/word n-gram logistic regression AUROC 0.93 (95% interval 0.91–0.94), a Binoculars-style zero-shot score from Qwen2.5-0.5B 0.80 (0.77–0.83), perplexity alone 0.76; a length-only detector is at chance (0.52). Zero-shot scores are weak on mpt-chat text (0.65) while n-grams are not (0.95). Limits: the ‘imitator’ is RAID’s own chat generations, not a prompted or optimised one; three chat models; no author-group labels for fairness checks. Preliminary: first step of track 11, no game yet.

Paper, code and results

• README and method
• Tests

More in Adversarial human imitation

• Do the file sizes and line counts of this repository follow Benford’s law?Do this repository's file sizes follow Benford's law? Not as a whole (byte MAD 0.018, nonconforming) because templated files repeat the same size thousands of times; counting each distinct size once gives acceptable conformity (MAD 0.0068).
• Can prompt search alone move a frozen 0.5B imitator past a prompted 0.5B detector, and which proposer does it best at equal budget?Can prompt search alone move a frozen 0.5B imitator past a prompted 0.5B detector? A stylised first run of the imitation game, four proposers at equal budget on two families: the starting detector is below chance, proposers cannot be ranked at this size, and length is a major confound.
• Does the prompt-only imitation game settle or cycle, and does beating the latest detector mean beating the others?Does the prompt-only imitation game settle or cycle? Every imitator prompt from mimic-loop against every detector prompt (0.5B on both sides): pure equilibria below chance (reddit 0.396, reviews 0.292), no cycles at a noise margin, and text length tracks detector accuracy. Stylised, one loop seed; model-size question open.
• Does a prompt-free human-vs-machine detector hold up on context families it never saw?Does a prompt-free n-gram human-vs-machine detector hold up on context families it never saw? Detector side of mimic-anycontext with no LLM (model download was blocked): AUROC 0.934 on unseen families against 0.985 in-family, and 40 target items close most of the gap on recipes. Stylised: plain-prompt RAID text only.
• Which features let a human-vs-machine text detector carry over to a context family it never saw: words, characters, function words, formatting or length?Which features let a human-vs-machine detector carry over to a context family it never saw? Leave-one-family-out AUROC with train-only vectorisers: words 0.931, characters 0.916, both 0.938, function words alone 0.797, surface formatting 0.687 (below chance on abstracts and recipes), length alone 0.155 to 0.885 depending on family. Stylised: one classifier, plain-prompt RAID text, no LLM.
• Can a prompt make a 0.5B model detect machine text, and does the prompt carry to families it never saw?Can a prompt make a 0.5B model detect machine text, and does it carry to unseen families? Negative: the prompted Qwen2.5-0.5B detector is inverted (mean AUROC 0.278 plain, 0.293 universal on held-out families, none above 0.5 on any family) while the n-gram detector gets 0.93. Stylised: one small model, 16 prompts, plain-prompt RAID text.

All 41 Adversarial human imitation projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.