Which features let a human-vs-machine text detector carry over to a context family it never saw: words, characters, function words, formatting or length?
research/mimic-featuretransfer
Which features let a human-vs-machine detector carry over to a context family it never saw? Leave-one-family-out AUROC with train-only vectorisers: words 0.931, characters 0.916, both 0.938, function words alone 0.797, surface formatting 0.687 (below chance on abstracts and recipes), length alone 0.155 to 0.885 depending on family. Stylised: one classifier, plain-prompt RAID text, no LLM.
What it found
Leave-one-family-out AUROC on the RAID-derived mimic-contexts suite, with every vectoriser fitted on training families only and 95% bootstrap intervals over test items. Word 1-2 grams average 0.931, characters 0.916, both 0.938 (in-family 0.987); function words alone still reach 0.797. Fourteen surface-format statistics average 0.687 and fall below chance on abstracts and recipes; length alone ranges from 0.155 (wiki) to 0.885 (poetry), so it flips sign between families. Fitting the vectoriser on training text only does not change the earlier 0.93 headline. Stylised and preliminary: one classifier, plain-prompt machine text, eight families, no LLM run.
Paper, code and results
More in Adversarial human imitation
- Can a prompt make a 0.5B model detect machine text, and does the prompt carry to families it never saw?Can a prompt make a 0.5B model detect machine text, and does it carry to unseen families? Negative: the prompted Qwen2.5-0.5B detector is inverted (mean AUROC 0.278 plain, 0.293 universal on held-out families, none above 0.5 on any family) while the n-gram detector gets 0.93. Stylised: one small model, 16 prompts, plain-prompt RAID text.
- Does MAUVE, a distribution-level score, rank human-vs-machine gaps the way a text detector does?Does MAUVE rank human-vs-machine gaps the way a text detector does? Across 20 RAID family × model cells MAUVE (lexical embedding) and a cross-fit n-gram detector agree loosely (Spearman −0.60) and disagree on recipes and news; every cell is far below the human-vs-human null (0.92 to 0.99). Preliminary, no GPT-2 features.
- Can an imitator fool a prompt-free human-vs-machine text detector with cheap surface edits, such as typos, contractions or dropping the chat preamble?Can cheap surface edits (typos, contractions, no chat preamble) fool the n-gram human-vs-machine detector? Most move held-out AUROC under 0.005; 2% typos cost 0.012 and keeping half the text 0.051 (from 0.927), and a refit detector wins it back.
- How many words must an imitator change to evade a word n-gram human-vs-machine detector that it can read?How many words must an imitator change to evade a word n-gram human-vs-machine detector it can read? Deleting the 20 most machine-ward word types per text takes held-out AUROC from 0.948 to chance (0.487); a refit detector recovers to 0.94-1.00.
- How many detector queries does an imitator need to evade a human-vs-machine text detector it cannot read?How many detector queries does an imitator need to evade a word n-gram human-vs-machine detector it cannot read? About 100 score queries per text, deleting the 20 best word types, take held-out AUROC from 0.948 to 0.520 (white-box 0.487); blind deletion does nothing. Stylised, no fluency check.
- Does refitting a human-vs-machine text detector on attacked text stop a black-box evader?Does refitting a text detector on attacked text stop a black-box evader? It settles after one refit: fresh-attack AUROC stays near 0.81 (from 0.719) while clean AUROC falls from 0.948 to about 0.90. Stylised, no fluency check.
All 41 Adversarial human imitation projects →
Other research topics
Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1
Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.