With six groups, how many hand-coded items does it take for every group's prevalence to be right?

What it found

On 3,188 real GoEmotions comments, with group-wise prediction-powered intervals, a 95% interval per group is not a 95% table. With six word-count groups (the smallest 1.5% of the corpus), only 71 to 77% of equal-allocation audits of 120–240 items covered all six true prevalences at once, against 85–88% with three groups; a Bonferroni widening (about a third wider) raised that to 89 to 94%. A random audit failed outright for the rare group: 291 of 300 draws of 120 items had under 5 audited items in it. Equal numbers per group gave intervals 24% narrower than random at 480 items for K=6 and the same for K=3. The coders are naive-Bayes and a word list, not LLMs, with no group-dependent error; one corpus, one seed; audits above 960 items are a large share of the corpus and the intervals ignore the finite-population correction, so those rows are conservative.

A figure from its own output

Paper, code and results

• README and method
• Experiments and results

More in Machine learning on qualitative data

• How many of the coder's negatives must you hand-code to recover the positives it missed?How many of the coder's negatives must you hand-code to recover the positives it missed? On 3,188 GoEmotions comments, about 50: coverage was 0.29 with 2, 0.87 with 10 and 0.94 with 50, whatever the audit's total size. Not an LLM; one corpus.
• Which confidence interval is right when only 5 to 20 of the coder's negatives are hand-coded?Which confidence interval is right when you hand-code only 5 to 20 of the coder's negatives? Not Wald: 0.63 coverage at 5 items against 0.94 for Wilson or Jeffreys on 3,188 GoEmotions comments. Not an LLM; one corpus.
• Does equal-numbers stratified auditing still match the optimal split when the coder errs more in the rare group?Does equal-numbers stratified auditing still match the optimal split when the coder errs more in the rare group? On 3,188 GoEmotions comments, injecting up to 30% extra label flips in the rarest 11% group moved the optimal audit share only from 0.53 to 0.56, and equal numbers stayed within 0.7% of optimal width. Not an LLM.
• Does a coder's error depend on writing register, and can a hand-coded sample tell?Does a coder's error depend on writing register, and can a hand-coded sample tell? On 3,188 GoEmotions comments, error differs by 1.9 points (naive-Bayes) or none (lexicon) between formal and informal text, yet the naive register gap is three times too large; an audit with up to 800 hand-coded items flags it no more often than chance. Not an LLM (no key); register is a proxy, not a dialect.
• Should the hand-coded audit for a group gap be stratified by group?Should the hand-coded audit for a group gap be stratified by group? On 3,188 GoEmotions comments, it made no difference for a 36% group and a random audit gave intervals 1.7 times as wide for an 11% group at 100 items. Equal numbers per group matched the optimal split. Not an LLM; one corpus.
• Can you do machine learning on qualitative data with an LLM, and when can you trust the numbers?Can you do machine learning on qualitative data with an LLM, and when can you trust the numbers? Simulated LLM coders with known error: an 89%-accurate coder (κ = 0.72) understates a 30% code at 26% with 0% interval coverage, and when it reads one group's answers less accurately the naive group gap has the wrong sign (+3.6 against −4 points, "significant" 80% of the time). Prediction-powered inference with 200 hand-coded answers per group removes the bias and narrows intervals 23–26% against hand-coding alone, but is wider below ~80% coder accuracy. Stdlib tool (κ, Krippendorff's α, PPI). Simulated, no real model.

All 23 Machine learning on qualitative data projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.