Can a prompt make a 0.5B model detect machine text, and does the prompt carry to families it never saw?

What it found

Prompted-detector half of mimic-anycontext. Qwen2.5-0.5B-Instruct answers Yes/No to “was this written by a person?” for RAID human and LLM texts in eight families, with 16 candidate prompts (the plain instruction plus random subsets of a ten-fragment bank). Negative result: the detector is inverted, with mean AUROC 0.278 for the plain prompt and 0.293 for a universal prompt picked on six families and scored on the other two (0.5 is chance; no prompt reaches 0.5 on any family, best 0.424). Picking prompts on selection texts adds +0.015 to +0.033, within selection noise, so the seen/held-out gap (about 0.01) says nothing about transfer. The prompt-free n-gram detector in mimic-anycontext-detect gets 0.93 on the same kind of text. Stylised: one small model, shallow random search, plain-prompt machine text, 30+30 test texts per family.

A figure from its own output

Paper, code and results

• README and method
• Experiments and results
• Tests

More in Adversarial human imitation

• Does MAUVE, a distribution-level score, rank human-vs-machine gaps the way a text detector does?Does MAUVE rank human-vs-machine gaps the way a text detector does? Across 20 RAID family × model cells MAUVE (lexical embedding) and a cross-fit n-gram detector agree loosely (Spearman −0.60) and disagree on recipes and news; every cell is far below the human-vs-human null (0.92 to 0.99). Preliminary, no GPT-2 features.
• Can an imitator fool a prompt-free human-vs-machine text detector with cheap surface edits, such as typos, contractions or dropping the chat preamble?Can cheap surface edits (typos, contractions, no chat preamble) fool the n-gram human-vs-machine detector? Most move held-out AUROC under 0.005; 2% typos cost 0.012 and keeping half the text 0.051 (from 0.927), and a refit detector wins it back.
• How many words must an imitator change to evade a word n-gram human-vs-machine detector that it can read?How many words must an imitator change to evade a word n-gram human-vs-machine detector it can read? Deleting the 20 most machine-ward word types per text takes held-out AUROC from 0.948 to chance (0.487); a refit detector recovers to 0.94-1.00.
• How many detector queries does an imitator need to evade a human-vs-machine text detector it cannot read?How many detector queries does an imitator need to evade a word n-gram human-vs-machine detector it cannot read? About 100 score queries per text, deleting the 20 best word types, take held-out AUROC from 0.948 to 0.520 (white-box 0.487); blind deletion does nothing. Stylised, no fluency check.
• Does refitting a human-vs-machine text detector on attacked text stop a black-box evader?Does refitting a text detector on attacked text stop a black-box evader? It settles after one refit: fresh-attack AUROC stays near 0.81 (from 0.719) while clean AUROC falls from 0.948 to about 0.90. Stylised, no fluency check.
• Does a prompt-free human-vs-machine detector carry over to a generator or decoding it never saw?Does a prompt-free detector carry over to a generator or decoding it never saw? The mimic-contexts n-gram detector loses 0.01 to 0.02 AUROC on a chat model left out of the fit (0.96 to 0.98 left out) and 0.025 to 0.05 on a left-out decoding (sampling 0.936); averaging over four same-source texts lifts AUROC to 0.999. Preliminary, plain-prompt RAID text only.

All 41 Adversarial human imitation projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.