Can you audit the coder's negatives at a lower rate and weight them back?

What it found

On 3,188 real GoEmotions comments, four automated coders (three cut-offs of a naive-Bayes classifier and a word list; not LLMs) and a hand-coded audit of 100 to 400 items, auditing coder-negative items at a lower rate and weighting them back by stratum size removed the positives-only audit's bias (−0.117 to −0.160, coverage 0.000) but did not narrow the interval. At 200 items the mean 95% width was 0.126 with half the audit among coder-negatives, 0.188 at 20%, 0.259 at 10% and 0.352 at 5%, and the Wald interval under-covered below about 10% (worst-coder coverage 0.85 at 5%, 0.51 at 2%) because only 2–20 negatives were audited. The coder-negative stratum is 69–85% of the corpus and holds 12–16% of it as human positives, so most of the variance is there; the oracle Neyman negative share is 0.63–0.82. The estimated recall was biased upward with few audited negatives. These coders miss 47–65% of positives; a high-recall coder (not tested) might change the answer. One code, one corpus, one split; standard errors ignore the finite-population correction.

A figure from its own output

Paper, code and results

• README and method
• Experiments and results

More in Machine learning on qualitative data

• With six groups, how many hand-coded items does it take for every group's prevalence to be right?With six groups, how many hand-coded items does it take for every group's prevalence to be right? On 3,188 GoEmotions comments, six 95% intervals covered all six truths in only 71 to 77% of 120 to 240-item equal audits (Bonferroni: 89 to 94%); a random audit could not estimate a 1.5% group at all. Not an LLM; one corpus.
• How many of the coder's negatives must you hand-code to recover the positives it missed?How many of the coder's negatives must you hand-code to recover the positives it missed? On 3,188 GoEmotions comments, about 50: coverage was 0.29 with 2, 0.87 with 10 and 0.94 with 50, whatever the audit's total size. Not an LLM; one corpus.
• Which confidence interval is right when only 5 to 20 of the coder's negatives are hand-coded?Which confidence interval is right when you hand-code only 5 to 20 of the coder's negatives? Not Wald: 0.63 coverage at 5 items against 0.94 for Wilson or Jeffreys on 3,188 GoEmotions comments. Not an LLM; one corpus.
• Does equal-numbers stratified auditing still match the optimal split when the coder errs more in the rare group?Does equal-numbers stratified auditing still match the optimal split when the coder errs more in the rare group? On 3,188 GoEmotions comments, injecting up to 30% extra label flips in the rarest 11% group moved the optimal audit share only from 0.53 to 0.56, and equal numbers stayed within 0.7% of optimal width. Not an LLM.
• Does a coder's error depend on writing register, and can a hand-coded sample tell?Does a coder's error depend on writing register, and can a hand-coded sample tell? On 3,188 GoEmotions comments, error differs by 1.9 points (naive-Bayes) or none (lexicon) between formal and informal text, yet the naive register gap is three times too large; an audit with up to 800 hand-coded items flags it no more often than chance. Not an LLM (no key); register is a proxy, not a dialect.
• Should the hand-coded audit for a group gap be stratified by group?Should the hand-coded audit for a group gap be stratified by group? On 3,188 GoEmotions comments, it made no difference for a 36% group and a random audit gave intervals 1.7 times as wide for an 11% group at 100 items. Equal numbers per group matched the optimal split. Not an LLM; one corpus.

All 23 Machine learning on qualitative data projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.