Does a human-vs-machine text detector still work on a language model, or a decoding setting, it never saw?
research/mimic-generatorholdout
Does a prompt-free detector transfer to unseen generators? Mostly: char+word n-grams keep AUROC 0.97 to 0.98 on a held-out chat model and 0.94 across a greedy-to-sampling shift; function-word cues alone drop to 0.69 to 0.85.
What it found
Leave-one-generator-out and leave-one-decoding-out on the mimic-contexts suite (RAID text: 2,400 human texts and mistral-chat, llama-chat and mpt-chat machine texts, eight families), five folds split by prompt so paired texts never straddle train and test. Char+word n-gram logistic regression scores AUROC 0.990 with nothing held out and 0.977, 0.984 and 0.970 with mistral, llama or mpt held out; trained on greedy text and tested on sampled text it scores 0.941. Function-word frequencies alone score 0.878 in-distribution and 0.78–0.85 on held-out generators (fold sd 0.065 on mpt). The decoding conditions also train on about half the machine text, so shift and size are confounded. Preliminary: three generators, one classifier, no LLM detector or imitator is run.
Paper, code and results
More in Adversarial human imitation
- Do prompts that fool a prompted LLM detector also fool detectors that use no prompt?Do the mimic-loop imitator prompts fool detectors they never saw? No: matched on length, all 26 testable prompts are still caught (n-gram AUROC 0.83 to 0.96, Binoculars-style 0.84 to 1.00); low raw scores come from text length.
- Can a learned Markov chain over a person's webcam frames fool a detector?Can a Markov chain over a person's frames fool a detector? No: chains with 8, 32 or 128 states, fitted on one person or on everyone, emitting state centres or real frames, are caught at AUROC 1.000 (one cell 0.999), where stitching the same real frames in 50-frame blocks scored 0.57 to 0.65.
- Does fitting a webcam-signal generator to one person fool a detector that has seen that person?Does fitting to one person help? Copying a person's own earlier windows is caught at AUROC 0.59 to 0.65 on their later windows; copying other people is 0.58 to 0.91, and 0.93 to 0.97 once the detector knows the person; a fitted parametric generator is caught at 1.000 either way.
- Does giving a Markov chain dwell times or a second-order memory let it fool a webcam-signal detector?Does giving a chain over a person's frames dwell times or second-order memory help? No: semi-Markov and second-order chains (8 or 32 states, own or pooled fit, centre or real-frame emission) are all caught at AUROC 0.999 to 1.000.
- Why is block-stitched real webcam signal still caught: the joins, or mixing people?Why is block-stitched real webcam signal still caught? With 50-frame blocks random stitching gives AUROC 0.887, one person's windows only 0.687, join-matched 0.796, both 0.588 (= whole-window replay 0.597); at 5 frames all are caught (8 seeds).
- How well do prompt-free detectors tell people from a plainly prompted LLM, and do they carry over to unseen kinds of text?How well do prompt-free detectors tell people from a plainly prompted LLM, and do they carry over to unseen kinds of text? Eight RAID families split by family: n-gram detector AUROC 0.93 and Binoculars-style 0.80 on held-out families, length at chance. Preliminary first step of track 11.
All 41 Adversarial human imitation projects →
Other research topics
Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1
Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.