Research topic
Adversarial human imitation
Can a frozen language model be prompted to write like people, against a detector that is also only prompted? The context suite, the reference detectors and the game loop.
41 projects, each with its own page, code and tests. See them on the full research index.
research/mimic-contexts · 10 tests
How well do prompt-free detectors tell people from a plainly prompted LLM, and do they carry over to unseen kinds of text? Eight RAID families split by family: n-gram detector AUROC 0.93 and Binoculars-style 0.80 on held-out families, length at chance. Preliminary first step of track 11.
Abstract
Eight text families from RAID (300 human and 300 plain-prompt chat-LLM texts each), split by family: train on five, hold out poetry, reviews and wiki. Prompt-free detectors on the held-out families: character/word n-gram logistic regression AUROC 0.93 (95% interval 0.91–0.94), a Binoculars-style zero-shot score from Qwen2.5-0.5B 0.80 (0.77–0.83), perplexity alone 0.76; a length-only detector is at chance (0.52). Zero-shot scores are weak on mpt-chat text (0.65) while n-grams are not (0.95). Limits: the ‘imitator’ is RAID’s own chat generations, not a prompted or optimised one; three chat models; no author-group labels for fairness checks. Preliminary: first step of track 11, no game yet.
research/benford-repo · 5 tests
Do this repository's file sizes follow Benford's law? Not as a whole (byte MAD 0.018, nonconforming) because templated files repeat the same size thousands of times; counting each distinct size once gives acceptable conformity (MAD 0.0068).
Abstract
First-digit frequencies of the byte sizes and line counts of the 35,648 git-tracked files at one commit, scored against Benford’s law by chi-square and Nigrini’s mean absolute deviation (MAD). All files together are nonconforming (byte MAD 0.0183, line MAD 0.0278), because a few sizes repeat thousands of times (1,071 bytes 1,297 times, 47 bytes 993 times, 4 lines 1,822 times). Counting each distinct size once, byte sizes conform acceptably (MAD 0.0068, n = 12,429). By extension, .json and .html stay nonconforming even deduplicated, and distinct Python line counts are acceptable (MAD 0.0103, n = 476). Descriptive: one commit of one generated, templated repository; an illustration of when the law fails, not a test of the law.
research/mimic-loop · 17 tests
Can prompt search alone move a frozen 0.5B imitator past a prompted 0.5B detector? A stylised first run of the imitation game, four proposers at equal budget on two families: the starting detector is below chance, proposers cannot be ranked at this size, and length is a major confound.
Abstract
A first, stylised run of the imitation game: Qwen2.5-0.5B-Instruct as imitator and detector, only prompts change, alternating rounds on two families (reddit, reviews) with four proposers (random, GA, GrIPS-style, local-LLM) at 4 evaluations per side per round. The plain detector starts below chance (paired accuracy 0.12), the final detectors reach 0.12–0.58 on 24 test pairs with intervals of about ±0.2, and the proposers cannot be ranked. Length is a major confound: imitators that wrote longer texts fooled the detector more. One seed; a harness and a first look, not a detection result.
research/mimic-crossplay · 12 tests
Does the prompt-only imitation game settle or cycle? Every imitator prompt from mimic-loop against every detector prompt (0.5B on both sides): pure equilibria below chance (reddit 0.396, reviews 0.292), no cycles at a noise margin, and text length tracks detector accuracy. Stylised, one loop seed; model-size question open.
Abstract
Follow-up to mimic-loop. Every imitator prompt any of its eight runs produced is scored against every detector prompt (Qwen2.5-0.5B-Instruct on both sides, 48 titles per cell, detector paired accuracy, 0.50 = chance), giving a 10×7 matrix on reddit and 9×10 on reviews. Both zero-sum games have a pure equilibrium with value below chance (reddit 0.396, 95% title-bootstrap 0.28 to 0.47; reviews 0.292, 0.17 to 0.34): no detector prompt guarantees chance, and no cell exceeds 0.56. On reviews the unadapted plain imitator prompt is the equilibrium imitator. Mean text length correlates with detector accuracy (Spearman 0.76 on reddit, 0.30 on reviews). No cycles at a 0.15 noise margin; overfitting to the latest opponent is small and not established. Stylised: one loop seed, a 0.5B detector that starts below chance. The model-size question is left open.
research/mimic-anycontext-detect · 5 tests
Does a prompt-free n-gram human-vs-machine detector hold up on context families it never saw? Detector side of mimic-anycontext with no LLM (model download was blocked): AUROC 0.934 on unseen families against 0.985 in-family, and 40 target items close most of the gap on recipes. Stylised: plain-prompt RAID text only.
Abstract
Detector side of mimic-anycontext, run without an LLM because the model download was blocked. A TF-IDF n-gram logistic-regression detector is fitted on seven of eight RAID-derived families and scored on the eighth: mean AUROC 0.934 against 0.985 when fitted inside the family, worst on recipes (0.843), abstracts and wiki. Forty labelled target items close most of the gap on recipes. Held-out generators cost little (AUROC 0.90–1.00) and length does not explain it. Stylised: plain-prompt machine text only, one detector type; the prompt-optimising imitator is not tested.
research/mimic-featuretransfer · 5 tests
Which features let a human-vs-machine detector carry over to a context family it never saw? Leave-one-family-out AUROC with train-only vectorisers: words 0.931, characters 0.916, both 0.938, function words alone 0.797, surface formatting 0.687 (below chance on abstracts and recipes), length alone 0.155 to 0.885 depending on family. Stylised: one classifier, plain-prompt RAID text, no LLM.
Abstract
Leave-one-family-out AUROC on the RAID-derived mimic-contexts suite, with every vectoriser fitted on training families only and 95% bootstrap intervals over test items. Word 1-2 grams average 0.931, characters 0.916, both 0.938 (in-family 0.987); function words alone still reach 0.797. Fourteen surface-format statistics average 0.687 and fall below chance on abstracts and recipes; length alone ranges from 0.155 (wiki) to 0.885 (poetry), so it flips sign between families. Fitting the vectoriser on training text only does not change the earlier 0.93 headline. Stylised and preliminary: one classifier, plain-prompt machine text, eight families, no LLM run.
research/mimic-anycontext-prompt · 7 tests
Can a prompt make a 0.5B model detect machine text, and does it carry to unseen families? Negative: the prompted Qwen2.5-0.5B detector is inverted (mean AUROC 0.278 plain, 0.293 universal on held-out families, none above 0.5 on any family) while the n-gram detector gets 0.93. Stylised: one small model, 16 prompts, plain-prompt RAID text.
Abstract
Prompted-detector half of mimic-anycontext. Qwen2.5-0.5B-Instruct answers Yes/No to “was this written by a person?” for RAID human and LLM texts in eight families, with 16 candidate prompts (the plain instruction plus random subsets of a ten-fragment bank). Negative result: the detector is inverted, with mean AUROC 0.278 for the plain prompt and 0.293 for a universal prompt picked on six families and scored on the other two (0.5 is chance; no prompt reaches 0.5 on any family, best 0.424). Picking prompts on selection texts adds +0.015 to +0.033, within selection noise, so the seen/held-out gap (about 0.01) says nothing about transfer. The prompt-free n-gram detector in mimic-anycontext-detect gets 0.93 on the same kind of text. Stylised: one small model, shallow random search, plain-prompt machine text, 30+30 test texts per family.
research/mimic-mauve · 4 tests
Does MAUVE rank human-vs-machine gaps the way a text detector does? Across 20 RAID family × model cells MAUVE (lexical embedding) and a cross-fit n-gram detector agree loosely (Spearman −0.60) and disagree on recipes and news; every cell is far below the human-vs-human null (0.92 to 0.99). Preliminary, no GPT-2 features.
Abstract
MAUVE between human and RAID chat-model text in 20 family × model cells (TF-IDF/SVD embedding, 5 draws of 48–100 texts), against a cross-fit n-gram detector on the same texts. Human-vs-human MAUVE is 0.92–0.99; every human-vs-model cell is at most 0.83. MAUVE and AUROC rank cells loosely alike (Spearman −0.60, p = 0.005; wiki/mistral-chat is hardest for both), but disagree on recipes (AUROC 0.999, MAUVE 0.32) and news (0.97, 0.75), and MAUVE does not track the distinct-2 ratio (ρ = 0.24, n.s.). Preliminary: the embedding is lexical, not MAUVE’s usual GPT-2 features, so values are not comparable with the literature; plain-prompt text only, no LLM run.
research/mimic-evade · 5 tests
Can cheap surface edits (typos, contractions, no chat preamble) fool the n-gram human-vs-machine detector? Most move held-out AUROC under 0.005; 2% typos cost 0.012 and keeping half the text 0.051 (from 0.927), and a refit detector wins it back.
Abstract
The mimic-contexts n-gram detector (held-out AUROC 0.927) is scored on poetry, reviews and wiki after scripted edits to the machine texts only. Markdown stripping, preamble removal and contractions move it by at most 0.005; 2% typos cost 0.012 (interval −0.014 to −0.010); keeping the first half costs 0.051, the largest drop, and no edit approaches chance. A detector refit on edited training text recovers and exceeds the baseline (0.956 to 0.969), partly by learning text length. Lowercasing and whitespace edits are null by construction. Preliminary: scripted rules, not LLM-chosen edits; one seed.
research/mimic-whitebox · 5 tests
How many words must an imitator change to evade a word n-gram human-vs-machine detector it can read? Deleting the 20 most machine-ward word types per text takes held-out AUROC from 0.948 to chance (0.487); a refit detector recovers to 0.94-1.00.
Abstract
A word 1-2 gram logistic-regression detector (held-out AUROC 0.948 on poetry, reviews and wiki) is attacked by an imitator that knows its weights and deletes or swaps the k most machine-ward word types in each of its own texts. Deleting 10 word types drops AUROC to 0.738, 20 to 0.487 (chance) and 40 to 0.150; swapping 20 gives 0.317. A detector refit on text attacked the same way recovers to 0.94–1.00. Stylised: scripted edits with no fluency check, one seed, one detector family, no LLM run.
research/mimic-blackbox · 5 tests
How many detector queries does an imitator need to evade a word n-gram human-vs-machine detector it cannot read? About 100 score queries per text, deleting the 20 best word types, take held-out AUROC from 0.948 to 0.520 (white-box 0.487); blind deletion does nothing. Stylised, no fluency check.
Abstract
The word 1-2 gram detector of mimic-whitebox (held-out AUROC 0.948) is attacked by an imitator that sees only the score of each text it submits. It deletes each of Q candidate word types in turn, reads the score drop and removes the k largest. With k=20, 25 queries per text give AUROC 0.873, 50 give 0.722 and about 95 give 0.520, against 0.487 for the white-box attack that reads the weights; blind deletion (0 queries) changes nothing. Stylised and preliminary: a linear detector, one seed, no fluency check on the edited text, and no retraining of the detector against the attack.
research/mimic-retrain · 4 tests
Does refitting a text detector on attacked text stop a black-box evader? It settles after one refit: fresh-attack AUROC stays near 0.81 (from 0.719) while clean AUROC falls from 0.948 to about 0.90. Stylised, no fluency check.
Abstract
A black-box attacker with 50 score queries per text takes the mimic-whitebox word 1-2 gram detector from held-out AUROC 0.948 to 0.719. Refitting the detector on attacked training texts and letting the attacker re-attack settles after one round: AUROC on freshly attacked text stays near 0.81 over four rounds, while clean-text AUROC falls to about 0.90. Edits made against the original detector transfer to the refit ones at 0.85. Stylised: linear detector, one seed, no fluency check, no LLM run.
research/mimic-genshift · 5 tests
Does a prompt-free detector carry over to a generator or decoding it never saw? The mimic-contexts n-gram detector loses 0.01 to 0.02 AUROC on a chat model left out of the fit (0.96 to 0.98 left out) and 0.025 to 0.05 on a left-out decoding (sampling 0.936); averaging over four same-source texts lifts AUROC to 0.999. Preliminary, plain-prompt RAID text only.
Abstract
The mimic-contexts n-gram detector fit on all eight RAID families, scored on text from a chat model or decoding left out of the fit (items split in two halves by source_id, halves swapped). AUROC falls from 0.991 to 0.978 (llama-chat), 0.983 to 0.971 (mistral-chat) and 0.981 to 0.959 (mpt-chat); leaving out sampling decoding costs 0.984 → 0.936 and greedy 0.989 → 0.964, with held-out wiki the weak spot (0.83 on sampling). Averaging the score over k same-source texts lifts AUROC from 0.971 (k=1) to 0.999 (k=4). Human batches are more varied than model batches (recipes distinct-2 0.589 against 0.373 for llama and 0.399 for mistral), but mpt-chat matches or exceeds human variety on poetry and reddit. Preliminary: plain-prompt RAID text, three chat models each present in only some families (generator and family partly confounded), one detector type, no LLM run.
research/mimic-stack · 4 tests
Does adding a zero-shot perplexity score to the n-gram detector help on unseen families? The two scores are uncorrelated within class, but averaging them lifts held-out AUROC only from 0.926 to 0.935 (paired gain +0.009, interval −0.010 to +0.026); fitted stacks do no better. Preliminary, 720 held-out texts, small scorer, no LLM run.
Abstract
The mimic-contexts n-gram detector and its Binoculars-style score (Qwen2.5-0.5B) are uncorrelated within each class (−0.02 human, 0.02 machine), yet combining them barely helps on the held-out families poetry, reviews and wiki: n-gram AUROC 0.926, mean of z-scores 0.935 (paired gain +0.009, 95% interval −0.010 to +0.026), fitted logistic stacks −0.003 and −0.019. It helps on wiki (0.892 to 0.935) and hurts on poetry and mpt-chat, where the zero-shot score is weak. Preliminary: 720 held-out texts, three chat models, a small scorer, one split; no LLM was run.
research/mimic-surface · 4 tests
How much text does a prompt-free detector need, and what does it key on? The mimic-contexts n-gram detector on held-out families: AUROC 0.65 at 50 characters, 0.84 at 400, 0.93 at 1500; stripping punctuation or shuffling words costs at most 0.014. Preliminary, plain-prompt RAID text only.
Abstract
The mimic-contexts n-gram detector (TF-IDF character and word n-grams, fitted on five families) is scored on poetry, reviews and wiki after cutting each text to its first N characters: AUROC 0.65 at 50, 0.75 at 100, 0.84 at 400 (the length the imitation-game loop shows its detector) and 0.93 at 1,500; refitting on the short text recovers only 0.02–0.07. Stripping punctuation changes AUROC by +0.003 and shuffling words by −0.014, so the detector reads vocabulary, not syntax or formatting (lowercasing and whitespace views are no-ops for this vectoriser). Preliminary: plain-prompt RAID text only, one detector, one split, no LLM involved.
research/mimic-threshold · 5 tests
If a detector's threshold is set for a 5% false-positive rate, does it hold on an unseen text type? Leave-one-family-out with the mimic-contexts n-gram detector: realised rate 0.00 to 0.54 (abstracts 0.54, books 0.18), and recipes catch 1.7% of machine text. Preliminary, plain-prompt RAID text only.
Abstract
The mimic-contexts n-gram detector is fit on seven RAID families and its threshold set so 5% of human text is flagged (out-of-fold), then applied to the eighth. Realised false-positive rates run from 0.00 to 0.54: abstracts 0.54, books 0.18, while news, poetry, recipes, reddit, reviews and wiki sit at 0.00–0.06. On recipes the cut-off is too strict instead: it catches 1.7% of machine text where a threshold set on recipes would catch 46%. AUROC does not show this (books has the best, 0.993, and the second-worst rate). Preliminary: plain-prompt RAID text, one detector type, one split, no LLM run.
research/mimic-recalib · 5 tests
How many human texts does it take to re-set a detector's threshold for a 5% false-positive rate on a new kind of text? About 80: with the plain 95th percentile k = 20 gives a mean false-positive rate of 9% (36% chance above 10%), k = 80 gives 6.1%; a conformal rule holds 4.9% but costs recall and needs 19+ texts. Stylised, RAID text, one n-gram detector.
Abstract
Follow-up to mimic-threshold. The n-gram detector is fit on seven RAID families; k human texts of the eighth set its cut-off, 500 random draws per cell. Taking the empirical 95th percentile, k = 20 gives a mean false-positive rate of 9% and a 36% chance of exceeding 10%; k = 80 gives 6.1% and 10%; k = 160 gives 5.5%. A conformal rank rule holds the mean at 4.7–4.9% from k = 20 but gives up recall (0.66 against 0.79 at k = 20) and cannot flag anything below 19 calibration texts. Even at k = 80 one draw lands anywhere from 1.7% to 8.8% (10th–90th percentile). Recalibration fixes the false-accusation rate, not the detector: recipes still catches under half of machine text. Preliminary: plain-prompt RAID text, one detector, calibration from the same population as the test, no LLM run.
research/mimic-batch · 7 tests
How many texts from one source does a weak human-vs-machine detector need before a batch verdict is reliable? Four all-machine texts give AUROC 0.997 (one text: 0.935); a quarter-machine source needs about 32; but a threshold set on 150 human texts drifts from 7% to 15% false flags at k = 32. Preliminary, RAID text, one n-gram detector.
Abstract
The mimic-contexts n-gram detector scores a held-out RAID family; a batch is k independent texts and its statistic is their mean score. One text separates people from a plainly prompted LLM at AUROC 0.935 (mean of 8 families); four all-machine texts reach 0.997 and eight reach 1.00. If only half the batch is machine, k = 8 gives 0.98; a quarter needs about 32 (0.985); at 10% machine 32 texts give only 0.82. The catch is calibration: with a threshold set on 150 human texts, the share of human batches flagged drifts from 7.4% at k = 1 to 15% on average at k = 32 (books 60%), because the batch mean's spread shrinks while the calibration error does not. Preliminary: plain-prompt text, i.i.d. batches, one detector, no LLM run.
research/mimic-twinc2st · 8 tests
Does a classifier two-sample test catch digital-twin errors that a paired trajectory error misses? Stylised pendulum: boosted-tree AUROC 0.72 for a friction error whose RMSE is a quarter of the sensor noise, 0.92 for Gaussian-vs-Student-t noise at zero paired error. Preliminary, simulated only.
Abstract
A stylised pendulum with Coulomb friction and heavy-tailed (Student-t) sensor noise is the ‘real’ system; twins differ in friction, noise shape or noise level. A classifier two-sample test (5-fold CV AUROC, 800 vs 800 trajectories, 3 seeds) separates real from twin trajectories with boosted trees: a friction error of 0.03 (10%) that moves the noise-free trajectory by only 0.006 rad RMSE — a quarter of the 0.02 rad noise sd — gives AUROC 0.72, and 0.91 at 20%. A twin with the right dynamics but Gaussian instead of Student-t noise of the same sd has paired error exactly 0 yet AUROC 0.92 once second-difference features are added (0.59 on raw angles). A logistic regression on raw angles stays at chance (0.51) throughout, because the start angles are symmetric. Limits: one toy system, two noise models, three seeds, a handful of classifier settings; simulated only, nothing about real hardware. Preliminary, no LLM involved: a side piece of track 11.
research/mimic-mmd · 8 tests
Does a kernel MMD test catch the same wrong digital twins as a classifier two-sample test? Stylised pendulum: raw-trajectory MMD is blind to a Coulomb-friction error (power 0.05 to 0.06) that boosted trees catch from n = 200, but MMD on second differences detects a +25% noise-level error at 0.97 from n = 100.
Abstract
A kernel two-sample test (unbiased RBF MMD, median-heuristic bandwidth) is compared with a gradient-boosted classifier two-sample test on the stylised pendulum of mimic-twinc2st. Rejection uses the 95th percentile of the exact twin’s own statistic. Raw-trajectory MMD has essentially no power against a Coulomb-friction error at n = 100–400 per class (0.05–0.06 at the median bandwidth, 0.37 at 0.03× for −20% friction), where the classifier reaches 1.00 at −20% from n = 200. MMD on second differences detects a +25% noise-level error at 0.97 from n = 100, where the raw-feature classifier is at 0.08–0.25. Stylised; 24 classifier replicates per cell, so those intervals are wide.
research/mimic-mujocotwin · 6 tests
Does a classifier two-sample test see the integrator, timestep and friction gaps of a MuJoCo digital twin? Stylised pendulum: AUROC 0.98 for Euler at 0.02 s, 0.53 (null) for a paired error of 0.003 rad, 0.92 for Gaussian-vs-Student-t noise at zero paired error. Preliminary, simulated only.
Abstract
A MuJoCo hinge pendulum (RK4, 0.002 s, dry friction, Student-t sensor noise) is the ‘real’ system; twins change the integrator, timestep or friction. A boosted-tree classifier two-sample test (800 trajectories per class, 3 seeds) has AUROC 0.98 for semi-implicit Euler at 0.02 s (paired RMSE 0.028 rad), 0.90 at 0.01 s, 0.67 at 0.005 s (0.007 rad) and 0.53 at 0.002 s (0.003 rad), which is not separable from the null 0.51. A friction error of 10% gives 0.52 and 20% gives 0.61. Gaussian instead of Student-t noise of the same sd has zero paired error but AUROC 0.92. Limits: one toy system, one classifier, 3 seeds; the ‘real’ system is itself simulated. Stylised, preliminary.
research/mimic-mmdmujoco · 8 tests
Does a kernel MMD test catch the same wrong MuJoCo digital twins as a classifier two-sample test? Stylised MuJoCo pendulum: boosted-tree C2ST power 1.00 at n = 200 for Euler at 0.02 s and a noise-shape error where MMD stays near 0.05; MMD on second differences catches a +25% noise-level error; friction errors undetected by all at n <= 200.
Abstract
A MuJoCo hinge pendulum with Student-t sensor noise is the ‘real’ system; twins change the integrator, friction or noise. Rejection uses the 95th percentile of the exact twin’s own statistic. Boosted-tree C2ST power is 1.00 at n = 200 for semi-implicit Euler at 0.02 s and 0.90 at 0.01 s, and 1.00 for a Gaussian-vs-Student-t noise-shape error, where MMD (raw or on second differences) stays at 0.00–0.07. MMD on second differences does catch a +25% noise-level error (0.95 at n = 100). A 10–20% friction error is invisible to all three at n ≤ 200. Limitations: one toy system, 20 C2ST replicates, two sample sizes. Stylised.
research/mimic-twinregret · 9 tests
Does a digital twin that a classifier can tell apart from the real system also make a worse controller? Stylised pendulum: 6 of 15 wrong twins have zero controller regret at C2ST AUROC 0.58 to 0.999; the worst regret (3.8%) comes from a twin with no sensor noise and zero paired error. Preliminary, simulated only.
Abstract
In the stylised pendulum of mimic-twinc2st with a torque input, a PD controller on noisy feedback is tuned on each wrong twin (48-point gain grid, 200 twin start angles) and scored on 2,000 held-out start angles on the ‘real’ system. Regret is its real cost minus the best grid point’s (0.259). Six of 15 wrong twins have regret exactly 0 in all 12 tuning seeds although the classifier two-sample test separates them from the real system with AUROC 0.58–0.999, among them g/L −20%, whose paired error is the largest (0.25 rad). The largest regret, 0.0099 (3.8%), comes from a twin with no sensor noise at all, whose paired error is 0. Across the 15 twins AUROC and regret correlate only loosely (Spearman 0.46, p = 0.08) and paired RMSE not at all (−0.28). Detecting a twin is not the same as the twin mattering for the task. Limits: one system, one task, a coarse grid that quantises regret, 12 tuning seeds. Stylised, simulated, no LLM: a side piece of track 11.
research/mimic-twinswing · 9 tests
On a swing-up task where friction matters, does a twin's detectability predict what tuning on it costs? Stylised pendulum: only the two gravity-error twins cost much (3.7% and 8.3% extra real cost); 13 others cost under 0.3% at C2ST AUROC 0.51 to 1.0. Preliminary, simulated only.
Abstract
Follow-up to mimic-twinregret on the same stylised pendulum, now with a torque limit below gravity so the controller must pump energy against friction, then catch the top with PD. Controllers are tuned on 15 wrong twins (60-point grid, 12 seeds) and scored on held-out real starts. Only the two gravity-error twins cost much: g/L +20% adds 8.3% to the best real cost and g/L −20% 3.7%; every other twin is under 0.3%, including no sensor noise, no Coulomb friction and doubled viscous friction, at C2ST AUROC up to 1.0. Spearman AUROC vs regret ρ = 0.16 (p = 0.57), paired RMSE vs regret 0.43 (p = 0.11). Stylised, one task, coarse grid; best gain on the grid edge for several twins.
research/mimic-twinswinggrid · 9 tests
Does a finer controller grid change which wrong pendulum twins cost something on the swing-up task? No: the same two gravity-error twins cost 3.8% and 6.1%, the rest under 0.3%; detectability vs regret ρ = 0.38 (p = 0.16). Preliminary, simulated only.
Abstract
Robustness check of mimic-twinswing with a 144-point gain grid instead of 60 (same pendulum, 15 wrong twins, 12 seeds). The same two gravity-error twins cost much: g/L +20% adds 6.1% to the best real cost and g/L −20% 3.8%; every other twin is at or below 0.3%. Coarse and fine regret rank the twins alike (Spearman ρ = 0.79, p = 0.001). AUROC vs regret ρ = 0.38 (p = 0.16), paired RMSE vs regret 0.25 (p = 0.38): still no reliable link, but 15 twins give little power. Stylised, one task; AUROC borrowed from the earlier run.
research/mimic-twingrid · 8 tests
Does a wider gain grid, or a controller that uses the twin's gravity, change which wrong pendulum twins are costly? Gravity errors stay the costly ones; giving the controller the twin's g/L made g/L −20% regret 13 times larger (0.013 to 0.177). Preliminary, simulated only.
Abstract
Follow-up to mimic-twinswing on the same stylised swing-up pendulum, removing two confounds: the best pump gain sat at the edge of the grid, and the controller's gravity constant was fixed. With ke up to 32 (126 controllers, 14 twins, 8 seeds) gravity errors remain the costly twins and sensor-noise errors stay under 0.0010. Letting the controller use the twin's own g/L made gravity errors worse: g/L −20% regret rises from 0.013 to 0.177 and g/L +20% falls from 0.036 to 0.017. Spearman AUROC vs regret is ρ = 0.63 (p = 0.021) but is driven by four gravity twins, and is not comparable with the earlier twin set. Preliminary: one toy system, simulated only.
research/mimic-twinrand · 5 tests
Does tuning a controller on a spread of randomized twins cut the regret of tuning on one wrong twin? Stylised pendulum: it brings the no-noise twin's regret from 0.0098 to 0.00006, but a very wide range raises a doubled-noise twin's worst-seed regret 7.5-fold. Preliminary, no LLM.
Abstract
Follow-up to mimic-twinregret on the same stylised pendulum and noisy-feedback PD task. Gains are tuned on 200 start angles spread over 8 randomized twins (noise level, Coulomb and viscous friction, g/L each uniform on nominal ± w× the real value) and scored on 2,000 held-out real start angles, over 24 tuning seeds. For the no-noise twin, whose single-twin regret is 0.0098, randomization with w = 0.5 brings it to 0.00006, the same as the exact twin. No-Coulomb and g/L +20% twins fall from 0.0014 to 0.00012 at w = 1–2. It does not always help: for a doubled-noise twin regret stays about 0.003 for w ≤ 1 and at w = 2 the mean rises to 0.0057 with a worst seed 7.5× the single-twin worst. Limits: one system and task, a 48-point grid that quantises regret, one randomization family with a shared width. Stylised, simulated, no LLM.
research/mimic-twinnoiserand · 4 tests
Does randomizing only the noise level of a wrong twin keep domain randomization's rescue without its blow-up? Stylised pendulum: the no-noise twin's regret goes 0.0098 to 0 and stays small at a wide range (worst seed 0.0015 vs 0.042 for all-four), but friction and gravity errors are not helped. Preliminary, no LLM.
Abstract
Follow-up to mimic-twinrand on the same stylised pendulum PD task. Only the twin's noise sd is spread (uniform on nominal ± w×0.02); friction and g/L stay nominal. A no-noise twin's regret falls from 0.0098 to 0 at w=2 and stays at 0.00035 (worst seed 0.0015) at w=4, where randomizing all four parameters has a worst seed of 0.042. A doubled-noise twin improves monotonically (0.0038 to 0.0012), which all-four did not. Friction and gravity errors are untouched (0.00138 at every width). 24 seeds, 48-point grid; preliminary, one toy system, no LLM.
research/mimic-twinpower · 5 tests
How many real trajectories does a classifier test need to flag a wrong digital twin, and are the twins it flags the ones that cost a controller? Stylised pendulum: all six twins with controller regret of at least 0.001 are flagged by 80 real trajectories, and so are three that cost nothing. Preliminary, simulated only.
Abstract
On the stylised pendulum of mimic-twinregret, a boosted-tree C2ST with n real and n twin trajectories flags a twin when its AUROC beats the 95th percentile of the exact-twin null at the same n (20 replicates per cell, n = 10 to 160). The six twins whose controller regret is at least 0.001 are all flagged in at least 80% of replicates by n = 80 (by 40 for the four noise-level and noise-shape twins), so the test does not miss costly twins at that size. It also flags three twins that cost nothing (g/L −20%, noise ×0.5, noise ×0.75) by n = 40–80, while Coulomb −10% and −20% stay unflagged at 160. Spearman correlation of power at 80 with regret is 0.52 (p = 0.048, 15 twins). A flag is a cheap alarm, not a verdict on usefulness. Limits: one system, one task, 20 replicates, a null threshold that is degenerate at n ≤ 20. Stylised, simulated, no LLM.
research/mimic-dialogue · 7 tests
Does a second turn make a plainly prompted LLM easier to spot in dialogue? A prompt-free n-gram detector gets AUROC 0.93 on one imitated user turn and 0.98 on two, on 400 OpenAssistant threads with a 0.5B imitator. Stylised baseline, no prompt optimised.
Abstract
Qwen2.5-0.5B-Instruct, given only a plain prompt, writes a user's second and third message in 400 real OpenAssistant threads. A prompt-free character and word n-gram detector (5-fold cross-validation grouped by thread) gets AUROC 0.93 on a single turn and 0.98 on the two turns together (paired accuracy 0.98); length alone gets 0.65 and 0.71. A quarter of the imitations read like assistant replies; dropping those threads leaves 0.95 for the thread. Limits: volunteer prompter turns, one 0.5B imitator and prompt, one seed, teacher-forced turns, no group-level false-positive rates. Stylised baseline; no prompt is optimised.
research/mimic-cursor · 9 tests
Can an imitator that changes its own code fool a human-vs-bot cursor detector, and does shared feedback help? On DUX cursor strokes of 46 people, a held-out reference detector catches every final imitator at AUROC 0.99 to 1.00 in all four feedback conditions; two-way sharing gave the strongest detectors in cross-play (0.81 vs 0.70 blind, interval includes 0), and an 8x larger search stops at 0.88, the level of replaying real strokes. Preliminary.
Abstract
Browser use, the first subject of the adversarial-imitation track beyond writing. Cursor strokes of 46 people from DUX (CC-BY-4.0) against an imitator built from code components and a detector that picks 3 of 11 feature functions, 4 feedback conditions × 8 seeds × 30 rounds at equal budget. A reference detector the loop never saw catches every final imitator at AUROC 0.99 to 1.00; detector explanations and shared imitator code did not make a better imitator. Detectors trained with two-way sharing did best against other runs’ imitators (cross-play AUROC 0.81 vs 0.70 blind), but the paired interval includes zero. An 8× larger search ends at 0.88, no better than replaying a real stroke, so the component library is the limit. Preliminary: one web form, 8 seeds, no model-written code.
research/mimic-webcam · 8 tests
Can a generator imitate a person's webcam emotion signals well enough to fool a detector? On DUX Affectiva scores of 46 people, every fixed generator and every loop-tuned imitator is caught at AUROC 0.997 or higher by a detector the loop never saw; only replaying real windows gets close to chance (0.61). Preliminary.
Abstract
Webcam, the third subject of the adversarial-imitation track, signals first: no pixels or faces. Ten-second windows of 12 facial-coding channels (Affectiva scores logged by DUX, CC-BY-4.0) from 46 people, split by person. Six fixed generators from constant to AR(1) with channel correlation, holds and spikes, and an imitator that tunes its own generator code against a 3-feature-group detector for 30 rounds (8 seeds; 4 seeds at 8× the budget), are all separated from held-out people by a boosted-tree reference detector the loop never saw at AUROC 0.997 or higher. Only replaying whole windows of other people gets near chance (0.61, which is the gap between people). Preliminary and stylised: one context, a fixed component library, and windows biased toward moments when the person was using the mouse or keyboard.
research/mimic-webcam-tails · 5 tests
Does giving a webcam-signal generator real heavy tails and bursts fool a detector? No: copula and quiet/burst generators still separate from held-out people at AUROC 1.000 (8 seeds); restoring per-window level and spread brings level and spread to near chance, but extremes (0.995) still give it away; copying a real window's value set (G6) puts those at chance too, yet all-groups AUROC stays 0.998 on temporal structure.
Abstract
Follow-up to mimic-webcam on DUX facial-coding scores of 46 people. Generators that take each channel's marginal from the training people (Gaussian copula through empirical quantiles; quiet/burst regimes) still separate from 14 held-out people at reference AUROC 1.000 (8 seeds). Matching marginals alone made level and spread easier to catch (0.99–1.00 vs 0.91 and 0.94 for the Gaussian baseline); restoring per-window level and spread brings them to 0.54 and 0.57, but extremes (0.995), holds (0.995) and autocorrelation (0.94) still give it away. A sixth generator that copies a real window’s whole value set puts level, spread, extremes and floor at chance (0.44–0.46), yet all-groups AUROC stays 0.998: holds, autocorrelation, roughness and cross-channel structure carry it. Stylised: hand-built generators, one context, post-hoc final variants.
research/mimic-anycontext-size · 4 tests
Does a bigger Qwen make a better human-vs-machine detector? Partly: plain-prompt AUROC rises from 0.278 (0.5B) to 0.329 (1.5B), but 1.5B is still below chance in 84% of family-by-prompt cells.
Abstract
Replays the first four detector prompts of mimic-anycontext-prompt on its 30 human + 30 RAID machine test texts in each of eight families, with Qwen2.5-0.5B-Instruct and Qwen2.5-1.5B-Instruct (Yes/No logit difference, single-text mode). Mean AUROC of the plain prompt is 0.278 at 0.5B and 0.329 at 1.5B; the best of the four prompts reaches 0.296 and 0.389. At 1.5B, 27 of 32 family-by-prompt cells are still below 0.5 (all 32 at 0.5B); only news is near chance (0.504). The sign-flipped score would average 0.72 and 0.67. The 0.5B plain-prompt mean (0.278) matches the earlier project. The gain is uneven: recipes and news improve, books and poetry get worse. Limitations: two sizes of one model family, four prompts, 30+30 texts per family (family AUROC standard error about 0.06), one seed, no selection step. Nothing here is an adapted imitator.
research/mimic-codeloop · 4 tests
A 1.5B code model rewrites the cursor generator against the mimic-cursor detector: only 25 to 29% of its candidates run, and the reference detector stays at AUROC 1.000, tied with the component search at the same 24-candidate budget (one seed, preliminary negative).
Abstract
Replaces mimic-cursor's component search with Qwen2.5-Coder-1.5B-Instruct, run on CPU, which reads the detector's score (and, in one condition, its feature gaps) and rewrites a Python stroke generator: 6 rounds of 4 candidates, the component search's per-round budget. Data, detector, reference detector and copy test are those of mimic-cursor (DUX, 46 people). The reference detector stays at AUROC 1.000 on held-out people in both conditions, as does the component search at 24 candidates (0.999 to 1.000, 8 runs). Only 7 of 24 (blind) and 6 of 24 (feedback) candidates were usable; the rest were rejected as banned tokens (likely imports), failed to load or crashed. Limitations: one seed per condition (three were planned), one small model, 24 candidates, no human templates available to the model, a sandbox whose import ban may account for some rejections. Preliminary and negative; a larger model on a GPU is an owner step.
research/mimic-generatorholdout · 4 tests
Does a prompt-free detector transfer to unseen generators? Mostly: char+word n-grams keep AUROC 0.97 to 0.98 on a held-out chat model and 0.94 across a greedy-to-sampling shift; function-word cues alone drop to 0.69 to 0.85.
Abstract
Leave-one-generator-out and leave-one-decoding-out on the mimic-contexts suite (RAID text: 2,400 human texts and mistral-chat, llama-chat and mpt-chat machine texts, eight families), five folds split by prompt so paired texts never straddle train and test. Char+word n-gram logistic regression scores AUROC 0.990 with nothing held out and 0.977, 0.984 and 0.970 with mistral, llama or mpt held out; trained on greedy text and tested on sampled text it scores 0.941. Function-word frequencies alone score 0.878 in-distribution and 0.78–0.85 on held-out generators (fold sd 0.065 on mpt). The decoding conditions also train on about half the machine text, so shift and size are confounded. Preliminary: three generators, one classifier, no LLM detector or imitator is run.
research/mimic-looptransfer · 5 tests
Do the mimic-loop imitator prompts fool detectors they never saw? No: matched on length, all 26 testable prompts are still caught (n-gram AUROC 0.83 to 0.96, Binoculars-style 0.84 to 1.00); low raw scores come from text length.
Abstract
Replays all 32 imitator prompts from mimic-loop (Qwen2.5-0.5B-Instruct, reddit and film reviews, 48 texts per prompt) and scores them with the mimic-contexts prompt-free detectors, refit without the loop's test items. Raw AUROCs range from 0.13 to 0.99, but they track mean generated length (Spearman +0.86 to +0.97): very short texts get a higher log-perplexity and look more human. Comparing only pairs within 1.5× in length, every prompt with enough pairs (26 of 32) is still caught, n-gram AUROC 0.83–0.96 and Binoculars-style 0.84–1.00. A prompt's paired accuracy against the loop's own prompted detector does not predict its prompt-free AUROC (Spearman −0.19 to +0.16 over all prompts, with length-matched signs negative). Preliminary: one loop seed, one 0.5B model, two families, 24 human texts per family; the detectors are trained on RAID chat-model text, not Qwen text, so some distance is generator shift.
research/mimic-webcam-markov · 5 tests
Can a Markov chain over a person's frames fool a detector? No: chains with 8, 32 or 128 states, fitted on one person or on everyone, emitting state centres or real frames, are caught at AUROC 1.000 (one cell 0.999), where stitching the same real frames in 50-frame blocks scored 0.57 to 0.65.
Abstract
A first-order Markov chain over k-means states (k = 8, 32, 128) of DUX webcam-signal frames, fitted on one person's earlier windows or on the other 45 people's, emitting state centres or real frames, is scored against that person's later windows by a boosted-tree detector (3 seeds). Every cell is AUROC 1.000 except pooled 128-state member emission at 0.999, in the population and the profile-aware detector alike. Nearly every feature group alone separates them at 0.92 or more, so the chain gets none of the window-scale statistics right even when it emits real frames; 50-frame blocks of the same frames reached 0.57 to 0.65 in the predecessor project. A guess that chains lose between-window differences is only partly supported. Stylised: one form-filling context, first-order chains only.
research/mimic-webcam-persongen · 5 tests
Does fitting to one person help? Copying a person's own earlier windows is caught at AUROC 0.59 to 0.65 on their later windows; copying other people is 0.58 to 0.91, and 0.93 to 0.97 once the detector knows the person; a fitted parametric generator is caught at 1.000 either way.
Abstract
Each of 46 DUX webcam-signal people (12 to 54 ten-second windows) is split in time: generators see the first half only, a boosted-tree detector separates the second half from generated windows (6 seeds). Drawing 50-frame blocks from the person's own earlier windows gives AUROC 0.654 (join-matched 0.565, whole-window replay 0.593); drawing them from other people gives 0.913 (0.833, 0.582), rising to 0.971 (0.959, 0.926) when the detector is also given the person's mean level and spread. A fitted parametric generator (window levels, AR(1), channel correlation, holds) is caught at 1.000 whether fitted on one person or on everyone. So person-specific style is most of what separates copies from people; a fitted generator is still far from it. Stylised: one form-filling context; copies are ceilings, not generators.
research/mimic-webcam-semimarkov · 5 tests
Does giving a chain over a person's frames dwell times or second-order memory help? No: semi-Markov and second-order chains (8 or 32 states, own or pooled fit, centre or real-frame emission) are all caught at AUROC 0.999 to 1.000.
Abstract
The predecessor project's first-order Markov chains over DUX webcam-signal frames were caught at AUROC 1.000. Two fixes it named are tried here: a semi-Markov chain that holds each k-means state for a dwell time drawn from that state's empirical run lengths, and a second-order chain whose next state depends on the last two (backing off to first order for unseen pairs). With 8 or 32 states, fitted on one person's earlier windows or on the other 45 people's, emitting state centres or real frames, all 16 cells are scored at AUROC 0.999 to 1.000 by a boosted-tree detector over 3 seeds, in the population and profile-aware detectors alike. Neither fix reaches the 0.57 to 0.65 of stitching 50-frame real blocks. Stylised: one form-filling context, 46 people, k of 8 and 32 only, empirical dwell times without drift.
research/mimic-webcam-stitch · 4 tests
Why is block-stitched real webcam signal still caught? With 50-frame blocks random stitching gives AUROC 0.887, one person's windows only 0.687, join-matched 0.796, both 0.588 (= whole-window replay 0.597); at 5 frames all are caught (8 seeds).
Abstract
Ceiling study on the DUX webcam expression signals: 100-frame windows stitched from real blocks of 5 to 50 frames, scored by a boosted-tree detector on 14 held-out people (8 seeds). With 50-frame blocks random stitching is caught at AUROC 0.887; blocks from one person only give 0.687; join-matched blocks 0.796; both together 0.588, equal to replaying whole windows (0.597). At 5 frames everything is caught (≥0.989). Mixing people is the larger giveaway, join mismatch smaller, and removing both only reaches the replay ceiling. Every frame is real, so this is not a generator. Stylised: one form-filling context, 46 people.
Other research topics
Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1
Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.