Does a hand-coded audit protect a group gap when it only checks the overall count?

What it found

On 3,188 real GoEmotions comments, the short-minus-long gap in negative-emotion prevalence (−0.053) was badly distorted by every naive-Bayes coder (error 0.13–0.25, 5–10 times the naive interval's half-width), even a coder with exactly zero overall prevalence bias. Gating on the overall difference test, as for a single prevalence, gave worst-case 95% coverage of 0.04 at 100 hand-coded items and 0.004 at 1,600: the more it was audited the more surely the coder was trusted. A test on the gap bias itself was 0.23 at 100 items and recovered to 0.97 only from 800. Equivalence gates at tolerance 0.02 and 0.05 never passed the naive-Bayes coders, so they equalled always applying the group-wise prediction-powered correction (0.94–0.99). Coders are a naive-Bayes classifier with a moved cut-off and a word list, not LLMs; one corpus, one split, one seed.

A figure from its own output

Paper, code and results

• README and method
• Experiments and results

More in Machine learning on qualitative data

• Can you estimate a group gap from an audit that hand-codes only a few of the coder's negatives?Can you estimate a group gap from an audit that hand-codes only a few of the coder's negatives? Not usefully: 0.9-wide intervals at 5 items, and a plain random audit (0.23 wide) beats the design. Pooling the groups' negatives covers 3%. 3,188 GoEmotions comments, not an LLM.
• If a hand-coded audit finds no bias in a coder, can you trust its counts?If a hand-coded audit finds no bias in a coder, can you trust its counts? On 3,188 GoEmotions comments, an audit-then-trust rule covered the truth in 8% of draws for a coder 2.6 points off, while always applying the prediction-powered correction covered 94–97%. Not an LLM; one corpus.
• Does a low audit rate for coder-negatives pay when the coder has high recall?Does a low audit rate for coder-negatives pay when the coder has high recall? On 3,188 GoEmotions comments the best share falls with recall (50%, 20%, 2%), but the narrowest interval (0.115–0.119 at 200 items) only matches a random audit with no coder (0.1195). Not an LLM.
• Do hypotheses about what separates two sets of documents survive a held-out check?Do hypotheses about what separates two sets of documents survive a held-out check? On 5,313 GoEmotions comments a word-counting proposer's effects are 7.6× too large at 100 discovery comments (1.5× at 2,000); a BH check on 3,000 held-out comments confirms almost nothing with shuffled labels, but 36–59% of confirmed hypotheses are the label's own vocabulary. Not an LLM proposer.
• How much of a human emotion codebook does a codebook induced from the text alone recover?How much of a human emotion codebook does a codebook induced from the text alone recover? On 5,313 GoEmotions comments, NMF and k-means codebooks agree with the human emotion label above chance but weakly (NMI 0.26 vs 0.12 for 27 codes); the best single code per emotion reaches AUC about 0.6. Topic models, not an LLM.
• Should you hand-code the items the coder called positive, or sample the audit by the coder's label?Should you hand-code only what the coder called positive, or sample the audit by the coder's label? On 3,188 GoEmotions comments, a positives-only audit had 0 of 1,000 intervals covering the truth for every coder (it misses the coder's false negatives); stratifying by label barely helped. Not an LLM; one corpus.

All 23 Machine learning on qualitative data projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.