Should you hand-code the items the coder called positive, or sample the audit by the coder's label?

research/qual-labelaudit

Should you hand-code only what the coder called positive, or sample the audit by the coder's label? On 3,188 GoEmotions comments, a positives-only audit had 0 of 1,000 intervals covering the truth for every coder (it misses the coder's false negatives); stratifying by label barely helped. Not an LLM; one corpus.

9 tests

What it found

On 3,188 real GoEmotions comments, four automated coders (three cut-offs of a naive-Bayes classifier and a word list; not LLMs) and a hand-coded audit of 50 to 800 items, sampling the audit by the coder's own label barely helped, and auditing only the coder's positives was badly wrong. The prediction-powered interval from a random audit was about 23% wider than the same kind of sample read as stratified by coder label (mean width 0.201 against 0.163 at 100 items), a property of the interval formula rather than of the draw; equal numbers per coder label were 9% wider than that, and oracle Neyman allocation 1–2% narrower. The sensitivity-style audit (hand-code only coder-positive items, assume the rest hold no positives) gave intervals a quarter as wide with coverage 0.000 in all 20 coder-and-size cells, because the coders missed 12–16% of the corpus's positives (bias −0.117 to −0.160 on a true 0.247); it can only estimate the coder's precision. One code, one corpus, one split; standard errors ignore the finite-population correction.

A figure from its own output

Figure produced by research/qual-labelaudit: width_by_design — qual-labelaudit/experiments/width_by_design.svg, written by the project's own experiment code.

Paper, code and results

More in Machine learning on qualitative data

Can you audit the coder's negatives at a lower rate and weight them back?Can you audit the coder's negatives at a lower rate and weight them back? On 3,188 GoEmotions comments, weighting removes the bias of a positives-only audit but the interval widens as the negative share falls (0.126 at 50%, 0.352 at 5%, 200 items) and under-covers below ~10%. Not an LLM; coders miss half the positives.
With six groups, how many hand-coded items does it take for every group's prevalence to be right?With six groups, how many hand-coded items does it take for every group's prevalence to be right? On 3,188 GoEmotions comments, six 95% intervals covered all six truths in only 71 to 77% of 120 to 240-item equal audits (Bonferroni: 89 to 94%); a random audit could not estimate a 1.5% group at all. Not an LLM; one corpus.
How many of the coder's negatives must you hand-code to recover the positives it missed?How many of the coder's negatives must you hand-code to recover the positives it missed? On 3,188 GoEmotions comments, about 50: coverage was 0.29 with 2, 0.87 with 10 and 0.94 with 50, whatever the audit's total size. Not an LLM; one corpus.
Which confidence interval is right when only 5 to 20 of the coder's negatives are hand-coded?Which confidence interval is right when you hand-code only 5 to 20 of the coder's negatives? Not Wald: 0.63 coverage at 5 items against 0.94 for Wilson or Jeffreys on 3,188 GoEmotions comments. Not an LLM; one corpus.
Does equal-numbers stratified auditing still match the optimal split when the coder errs more in the rare group?Does equal-numbers stratified auditing still match the optimal split when the coder errs more in the rare group? On 3,188 GoEmotions comments, injecting up to 30% extra label flips in the rarest 11% group moved the optimal audit share only from 0.53 to 0.56, and equal numbers stayed within 0.7% of optimal width. Not an LLM.
Does a coder's error depend on writing register, and can a hand-coded sample tell?Does a coder's error depend on writing register, and can a hand-coded sample tell? On 3,188 GoEmotions comments, error differs by 1.9 points (naive-Bayes) or none (lexicon) between formal and informal text, yet the naive register gap is three times too large; an audit with up to 800 hand-coded items flags it no more often than chance. Not an LLM (no key); register is a proxy, not a dialect.

All 23 Machine learning on qualitative data projects →

Other research topics

Proper scoring & elicitation28 Effort & peer prediction9 Verification games & slashing19 Detection & tolerances14 Markets as routers13 Prediction-market design9 Paying contributors8 Decentralised training systems23 Digital twins & sim-to-real164 LLM chip design loops40 Robot manufacturing loops53 Ambient AI assistant46 Self-replicating manufacturing34 Simulation-first design64 Engineered clothing45 Task-specific AI accelerators40 Algorithmic chip-design loops42 Open-source purpose-built robots44 The software-engineering loop43 Adversarial human imitation41 Open-source residential construction41 Autonomous multi-angle capture34 Sensing, imaging and 3D31 Energy supply and dense energy storage39 Robot societies among people31 Agentic co-design of low-cost robots27 Laundry-hanging robots25 Machine learning on qualitative data23 Robots on the building site27 Civilizations & group psychology1 History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.