Does a low audit rate for coder-negatives pay when the coder has high recall?

What it found

On 3,188 real GoEmotions comments, three automated coders (naive-Bayes cut-offs coding 2, 3 and 4 times the human positive share; recall 0.72, 0.90, 1.00; not LLMs) and a hand-coded audit of 100 to 400 items. The best share of the audit among coder-negatives falls as recall rises (about 50%, 20%, 2% or lower), following the oracle Neyman share (0.43, 0.18, 0). But the narrowest 95% interval at 200 items (0.115 to 0.119) is no better than a plain random audit of 200 items with no coder (0.1195), because precision is only 0.25–0.36. Coverage still fails at low shares unless the negative stratum is almost clean (worst coder 0.48 at 2%, 0.77 at 5%, 0.91 at 10%). Bias is removed in every cell by weighting. Recall is bought with low precision by moving a weak classifier's cut-off; an LLM with both high is untested. One code, one corpus, one split; Wald intervals; no finite-population correction.

A figure from its own output

Paper, code and results

• README and method
• Experiments and results

More in Machine learning on qualitative data

• Do hypotheses about what separates two sets of documents survive a held-out check?Do hypotheses about what separates two sets of documents survive a held-out check? On 5,313 GoEmotions comments a word-counting proposer's effects are 7.6× too large at 100 discovery comments (1.5× at 2,000); a BH check on 3,000 held-out comments confirms almost nothing with shuffled labels, but 36–59% of confirmed hypotheses are the label's own vocabulary. Not an LLM proposer.
• How much of a human emotion codebook does a codebook induced from the text alone recover?How much of a human emotion codebook does a codebook induced from the text alone recover? On 5,313 GoEmotions comments, NMF and k-means codebooks agree with the human emotion label above chance but weakly (NMI 0.26 vs 0.12 for 27 codes); the best single code per emotion reaches AUC about 0.6. Topic models, not an LLM.
• Should you hand-code the items the coder called positive, or sample the audit by the coder's label?Should you hand-code only what the coder called positive, or sample the audit by the coder's label? On 3,188 GoEmotions comments, a positives-only audit had 0 of 1,000 intervals covering the truth for every coder (it misses the coder's false negatives); stratifying by label barely helped. Not an LLM; one corpus.
• Can you audit the coder's negatives at a lower rate and weight them back?Can you audit the coder's negatives at a lower rate and weight them back? On 3,188 GoEmotions comments, weighting removes the bias of a positives-only audit but the interval widens as the negative share falls (0.126 at 50%, 0.352 at 5%, 200 items) and under-covers below ~10%. Not an LLM; coders miss half the positives.
• With six groups, how many hand-coded items does it take for every group's prevalence to be right?With six groups, how many hand-coded items does it take for every group's prevalence to be right? On 3,188 GoEmotions comments, six 95% intervals covered all six truths in only 71 to 77% of 120 to 240-item equal audits (Bonferroni: 89 to 94%); a random audit could not estimate a 1.5% group at all. Not an LLM; one corpus.
• How many of the coder's negatives must you hand-code to recover the positives it missed?How many of the coder's negatives must you hand-code to recover the positives it missed? On 3,188 GoEmotions comments, about 50: coverage was 0.29 with 2, 0.87 with 10 and 0.94 with 50, whatever the audit's total size. Not an LLM; one corpus.

All 23 Machine learning on qualitative data projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.