Does equal-numbers stratified auditing still match the optimal split when the coder errs more in the rare group?

What it found

On 3,188 real GoEmotions comments, the rare group is the longest 10.8% (344 comments, 23 or more words) and the estimand is the gap in negative-emotion prevalence to the rest (+0.072). Two naive-Bayes coders (not LLMs) have extra errors injected in the rare group only, by flipping each of their labels there with probability 0, 0.1, 0.2 or 0.3. The oracle share of the audit for the rare group stayed between 0.53 and 0.56, and an audit of equal numbers per group gave intervals within 0.7% of the optimal split at every injected rate (200 items: 0.333 against 0.331 with no injection, 0.346 against 0.344 at 0.3). A random audit stayed 1.66 to 1.69 times as wide. Coverage of the 95% interval was 0.95 to 1.00 for stratified designs (0.90 to 1.00 for random) (400 draws per cell, so about ±0.02). Limits: the coder already errs on 42–43% of rare-group items before injection, so random flips cannot raise the error spread much (the error rate reaches 0.50 and then falls back); a coder whose errors are more structured than coin flips is untested. One corpus, one grouping, one seed; no LLM was called.

A figure from its own output

Paper, code and results

• README and method
• Experiments and results

More in Machine learning on qualitative data

• Does a coder's error depend on writing register, and can a hand-coded sample tell?Does a coder's error depend on writing register, and can a hand-coded sample tell? On 3,188 GoEmotions comments, error differs by 1.9 points (naive-Bayes) or none (lexicon) between formal and informal text, yet the naive register gap is three times too large; an audit with up to 800 hand-coded items flags it no more often than chance. Not an LLM (no key); register is a proxy, not a dialect.
• Should the hand-coded audit for a group gap be stratified by group?Should the hand-coded audit for a group gap be stratified by group? On 3,188 GoEmotions comments, it made no difference for a 36% group and a random audit gave intervals 1.7 times as wide for an 11% group at 100 items. Equal numbers per group matched the optimal split. Not an LLM; one corpus.
• Can you do machine learning on qualitative data with an LLM, and when can you trust the numbers?Can you do machine learning on qualitative data with an LLM, and when can you trust the numbers? Simulated LLM coders with known error: an 89%-accurate coder (κ = 0.72) understates a 30% code at 26% with 0% interval coverage, and when it reads one group's answers less accurately the naive group gap has the wrong sign (+3.6 against −4 points, "significant" 80% of the time). Prediction-powered inference with 200 hand-coded answers per group removes the bias and narrows intervals 23–26% against hand-coding alone, but is wider below ~80% coder accuracy. Stdlib tool (κ, Krippendorff's α, PPI). Simulated, no real model.
• What happens to a regression coefficient when the regressor is an LLM's code?What happens to a regression coefficient when the regressor is an LLM's code? Naive OLS gives 0.74 for a true 1.00 (0% coverage) and biases a perfectly measured covariate when coder error depends on it; DSL with ~210 hand-coded items fixes both (coverage 94–97%) but narrows the coded coefficient's interval only 14%; per-class PPI fixes 3-class shares. Simulated coders.
• Does the prediction-powered correction work on real hand-coded text?Does the prediction-powered correction work on real hand-coded text? On 3,188 GoEmotions comments a naive-Bayes coder (kappa 0.14) is wrong on prevalence (13% against 25%, 0% coverage) and overstates a length gap six-fold; PPI repairs both but is wider than hand-coding alone. Not an LLM (no key).
• How much hand-coding can an LLM coder replace, with a valid interval, when its error differs by group and the category is rare?Research programme: valid inference from LLM-coded qualitative data under group shift and rare categories; on 44,485 real tweets a pooled PPI interval covers as little as 67% when the audit's language mix differs from the corpus, a language-stratified one 94 to 96%

All 23 Machine learning on qualitative data projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.