Machine learning on qualitative data

An LLM can code free text (“does this answer mention cost?”) so that it can be counted and modelled like numbers, but its codes need a correction first. In a seeded simulation, an 89%-accurate LLM coder (Cohen’s κ = 0.72) understates a 30% code at 26%, and its naive 95% interval covers the truth in 0 of 1,000 studies. When the LLM reads one group’s answers less accurately, the naive group gap comes out at +3.6 points against a true −4, the wrong sign, and is “significant” 80% of the time. Prediction-powered inference with 200 random hand-coded answers per group removes the bias, covers the truth 95–98% of the time and gives intervals 23–26% narrower than hand-coding alone. Below about 80% coder accuracy, though, its interval is wider. Includes a stdlib tool (κ, Krippendorff’s α, PPI) and a survey of five ways LLMs bring qualitative data into ML. Simulated coders, not a real model.

Regressing an outcome on an LLM's codes attenuates the coded variable's coefficient (0.74 for a true 1.00, interval coverage 0% in 300 simulated studies) and, when the coder's error depends on another covariate, biases that perfectly measured covariate too (0.467 against 0.50, coverage 60%). Design-based supervised learning (DSL) with about 210 hand-coded items drawn at a known 7% rate recovers both (1.01 and 0.50, coverage 94–97%) but narrows the coded coefficient’s interval by only 14%. For a 3-class code, naive class shares are off by up to 8.5 points with 0% coverage; per-class PPI covers 95.5–97%. Simulated coders only; no real LLM was run.

On 3,188 real Reddit comments with human emotion labels (GoEmotions), a naive-Bayes coder (kappa 0.14) reports 13.3% negative emotion against a human 25.3% (coverage 0 of 500 draws) and a short-versus-long gap of −0.170 against a true −0.028, because it misses 94% of negative emotion in short comments but 70% in long ones. PPI with 200 hand-coded items recovers both (coverage 0.94–0.97) but is 6–19% wider than using the hand-coded items alone, since the coders are weak. The coders are a trained classifier and a word list, not an LLM (no API key).

An open research programme. First real-data result (H1): on 44,485 NaijaSenti tweets in four languages, a 400-tweet audit whose language mix differs from the corpus makes the pooled prediction-powered 95% interval cover as little as 67% (English-trained coder) or 69% (in-language coder); PPI++ reaches 89%, and pooled hand-coding alone 87% under equal quotas. A language-stratified estimator with corpus-known sizes covers 93.8 to 95.9% in every design (planned bar 94%, one cell 0.2 points under) but is not narrower than stratified hand-coding, because the three stand-in coders (no LLM yet) are weak. One corpus, three audit mixes. A stdlib estimator library is built and tested. First preliminary result, on simulated corpora with a stand-in coder: the normal interval of a coder-label-stratified audit covers 57% at 1% prevalence, 83% at 3% and 94% at 20% (nominal 95%), so rare-category audit designs need small-count-safe intervals. An LLM's codes for qualitative text carry error that can differ between groups, and prediction-powered inference corrects counts, gaps and coefficients from a random hand-coded audit. The earlier qual-* projects showed on GoEmotions, with stand-in coders and no LLM, that naive counts are confidently wrong, that the correction is valid but wider than hand-coding alone for weak coders, and that a random audit cannot estimate a rare group. The programme asks when the audit can be small and how to sample it when the coder's error differs by group and the target category has prevalence under 5%. Hypotheses H1 to H5 cover pooled-correction failure under group shift, the end of the gain for rare categories, a real LLM coder, audit design, and a closed-form audit-size rule. Limits at this stage: the proposal's references were written from memory because the arXiv and Crossref hosts were blocked, and are to be machine-checked in milestone 1. Milestone 4 (H2, H4; both refuted as stated): on GoEmotions (38,234 comments, nine emotions at 0.10 to 1.96%, audit of 200, weak stand-in coder) normal-approximation intervals under uniform, PPI++ and label-stratified audits cover as little as 10 to 61% below 0.5% prevalence, so apparent width gains are not valid; Wilson on a uniform audit covers 91 to 99%, and an uncertainty-weighted audit covers 93 to 96% for seven of nine categories (86% for disgust, 89% for sadness) and is 22 to 52% narrower than Wilson for six of them.

The Claude session agent coded 300 real GoEmotions comments (blind to the human labels, one pass; not an API-called model) and over-coded: 132 flagged against 70 by raters, sensitivity 0.81, specificity 0.67, kappa 0.37, so the naive prevalence is 0.44 against a true 0.23 (coverage 0). That is the opposite error from the naive-Bayes and word-list stand-ins of qual-real, which undercount by 10 points. Its accuracy (0.71) is indistinguishable from naive Bayes (exact McNemar p = 0.4) and worse than a 38-word list (0.84, p = 0.0001). Its error differs by comment length (sensitivity 0.95 short, 0.76 long), but prediction-powered inference stays valid (bias under 0.006, coverage 0.96–0.996) and was 29% wider than hand-coding alone at 60 hand-coded items. Limits: one reader and pass, codebook mismatch with the raters' 11-emotion majority, possible dataset exposure, and a 300-item corpus that makes intervals conservative.

NaijaSenti tweets (Muhammad et al. 2022) in Hausa, Igbo, Nigerian Pidgin and Yoruba carry native-speaker sentiment labels, so the true negative share of each language (0.22 to 0.60) and every between-language gap is known. Three automated coders stand in for an LLM, which the session could not run: an English word list, a naive-Bayes classifier trained on English GoEmotions comments, and one trained on 19,065 of the Nigerian tweets with a single global threshold. The word list and the English classifier have sensitivity 0.004–0.108 on Hausa, Igbo and Yoruba and 0.09 and 0.39 on Pidgin, whose vocabulary is largely English-derived. Naive coded-share gaps are therefore wrong: Hausa–Yoruba is +0.106 by human labels and −0.065 by the English classifier, Igbo–Yoruba +0.035 against −0.052, and the in-language classifier overstates Hausa–Igbo at +0.293 against +0.071. Naive intervals cover the true gap 0 times in 500 draws for every pair and coder. Prediction-powered inference with 400 hand-coded tweets covers 93–96% but is 0.4–26% wider than hand-coding alone, since the coders are nearly uninformative. Limits: no LLM was run, so nothing here says how a frontier model reads these languages; language is confounded with topic and period; the hand-coded sample is random over the pooled corpus, not stratified by language.

On 3,188 real GoEmotions comments, an equivalence (TOST) audit gate either never passed the coder (tolerance 0.02 or tighter, so it equalled always applying the prediction-powered correction, coverage 0.94–1.00) or passed it too often (tolerance 0.05, worst coverage 0.95 at 100 hand-coded items falling to 0.35 at 1,600). Showing a bias under 0.01, the naive interval's scale, needs about 7,700 audited items, more than the corpus. The earlier difference-test gate was at 0.39 worst case with 1,600 items, and splitting the sample did not help. The coders are a naive-Bayes classifier with a moved cut-off and a word list, not LLMs; one corpus, one seed.

On GoEmotions comments (predict a negative-emotion label, AUC on 1,500 held-out items), a logistic model on 12 word-list codebook codes plus log length beats bag-of-words at every labelling budget from 25 to 1,600 items: 0.65 against 0.56 AUC at 100 labels and 0.75 against 0.67 at 1,600, with bag-of-words still gaining. Adding the codes to the bag of words lands between the two, and a 64-dimension random projection of the words is near chance (0.55 at 1,600). The codes are word lists, not an LLM, and a learned sentence embedding was not tested; one corpus, one label, 12 subsets per size.

On 5,313 real GoEmotions comments, 28 human-coded emotion codes (an upper bound for an LLM coder) predicted whether a comment replies to another comment with test AUC 0.527 at 1,600 labelled outcomes, below bag-of-words (0.557) and a 100-d LSA embedding (0.595 with numeric features); 5% simulated coder error erased the codes’ small gain over numeric features alone. Every feature set was weak (AUC at most 0.60) because the outcomes are not emotional, a coin-flip control stayed within 0.494–0.511, and a nine-subreddit outcome was too thin to rank anything. The codes are human votes, not an LLM’s.

On 3,188 real GoEmotions comments, the short-minus-long gap in negative-emotion prevalence (−0.053) was badly distorted by every naive-Bayes coder (error 0.13–0.25, 5–10 times the naive interval's half-width), even a coder with exactly zero overall prevalence bias. Gating on the overall difference test, as for a single prevalence, gave worst-case 95% coverage of 0.04 at 100 hand-coded items and 0.004 at 1,600: the more it was audited the more surely the coder was trusted. A test on the gap bias itself was 0.23 at 100 items and recovered to 0.97 only from 800. Equivalence gates at tolerance 0.02 and 0.05 never passed the naive-Bayes coders, so they equalled always applying the group-wise prediction-powered correction (0.94–0.99). Coders are a naive-Bayes classifier with a moved cut-off and a word list, not LLMs; one corpus, one split, one seed.

Re-running the small-k coder-negative audit of qual-negint for a short-minus-long gap in negative-emotion prevalence (3,188 real GoEmotions comments, four automated coders that are not LLMs, 200 audited items, 500 draws). With 5 coder-negative items a Wald interval covers the true gap (−0.059) in 58% of draws at worst; Jeffreys and plus-two intervals cover 94–99% but are 0.9–1.0 wide. A plain random 200-item audit is 0.23 wide at 95% coverage, narrower than any coder-label design up to 50 negatives (0.35). An interval that pools both groups' negatives is narrow and covered 3% of draws for the word-list coder, whose miss rate differs by group (0.155 vs 0.200). No LLM; one corpus and one grouping.

On 3,188 real GoEmotions comments, an audit-gated estimator (use the coder's labels if a 200-item hand-coded paired z test finds no bias, else apply a prediction-powered correction) covered the true prevalence in only 8% of draws when the coder over-counted by 2.6 points, and 27% when it under-counted by 5.1; it reached 88% only with a 12.8-point bias. At zero bias it behaved (corrects 4% of draws, covers 96%). Always applying the prediction-powered correction covered 94–97% at every bias. The coders are a naive-Bayes classifier with a moved cut-off and a word list, not LLMs; one corpus, one seed.

On 3,188 real GoEmotions comments, three automated coders (naive-Bayes cut-offs coding 2, 3 and 4 times the human positive share; recall 0.72, 0.90, 1.00; not LLMs) and a hand-coded audit of 100 to 400 items. The best share of the audit among coder-negatives falls as recall rises (about 50%, 20%, 2% or lower), following the oracle Neyman share (0.43, 0.18, 0). But the narrowest 95% interval at 200 items (0.115 to 0.119) is no better than a plain random audit of 200 items with no coder (0.1195), because precision is only 0.25–0.36. Coverage still fails at low shares unless the negative stratum is almost clean (worst coder 0.48 at 2%, 0.77 at 5%, 0.91 at 10%). Bias is removed in every cell by weighting. Recall is bought with low precision by moving a weak classifier's cut-off; an LLM with both high is untested. One code, one corpus, one split; Wald intervals; no finite-population correction.

On 5,313 real GoEmotions comments, a word-counting proposer that picks the 20 words most associated with negative emotion in a small discovery sample overstates their effects 7.6× at 100 discovery comments (0.095 against 0.013 held-out) and 1.5× at 2,000. A Benjamini–Hochberg check on 3,000 held-out comments confirms 0.8 of the 20 at 100 comments and 14.7 at 2,000, and with shuffled labels confirms almost none (at most 5.5% of splits any), but 36–59% of confirmed hypotheses are the label's own vocabulary (“sorry”, “hate”), so replication is not explanation. The proposer is not an LLM, and the restatement flag is a crude word list.

On 5,313 real GoEmotions comments, codebooks induced from half the text with no labels (NMF topics and k-means on LSA, 5 to 60 codes) were applied to the other half and compared with the 27-emotion human codebook. Agreement with the human primary emotion was above a size-matched random assignment but small: NMI 0.26 for 27 NMF topics against 0.12 at chance, and 0.34 for 60 against 0.20. The best single induced code per emotion reached held-out AUC 0.60–0.62 on average, at least 0.7 for 14–25% of the 15 emotions with enough positives. NMF topics were almost uncorrelated (mean |r| 0.02); k-means with few codes was redundant (11% of pairs above 0.3 at k=10). The induced codes are topic models, not an LLM’s TopicGPT or LLooM codebook, and the agreement score rests on 1,906 comments with a unique majority emotion.

On 3,188 real GoEmotions comments, four automated coders (three cut-offs of a naive-Bayes classifier and a word list; not LLMs) and a hand-coded audit of 50 to 800 items, sampling the audit by the coder's own label barely helped, and auditing only the coder's positives was badly wrong. The prediction-powered interval from a random audit was about 23% wider than the same kind of sample read as stratified by coder label (mean width 0.201 against 0.163 at 100 items), a property of the interval formula rather than of the draw; equal numbers per coder label were 9% wider than that, and oracle Neyman allocation 1–2% narrower. The sensitivity-style audit (hand-code only coder-positive items, assume the rest hold no positives) gave intervals a quarter as wide with coverage 0.000 in all 20 coder-and-size cells, because the coders missed 12–16% of the corpus's positives (bias −0.117 to −0.160 on a true 0.247); it can only estimate the coder's precision. One code, one corpus, one split; standard errors ignore the finite-population correction.

On 3,188 real GoEmotions comments, four automated coders (three cut-offs of a naive-Bayes classifier and a word list; not LLMs) and a hand-coded audit of 100 to 400 items, auditing coder-negative items at a lower rate and weighting them back by stratum size removed the positives-only audit's bias (−0.117 to −0.160, coverage 0.000) but did not narrow the interval. At 200 items the mean 95% width was 0.126 with half the audit among coder-negatives, 0.188 at 20%, 0.259 at 10% and 0.352 at 5%, and the Wald interval under-covered below about 10% (worst-coder coverage 0.85 at 5%, 0.51 at 2%) because only 2–20 negatives were audited. The coder-negative stratum is 69–85% of the corpus and holds 12–16% of it as human positives, so most of the variance is there; the oracle Neyman negative share is 0.63–0.82. The estimated recall was biased upward with few audited negatives. These coders miss 47–65% of positives; a high-recall coder (not tested) might change the answer. One code, one corpus, one split; standard errors ignore the finite-population correction.

On 3,188 real GoEmotions comments, with group-wise prediction-powered intervals, a 95% interval per group is not a 95% table. With six word-count groups (the smallest 1.5% of the corpus), only 71 to 77% of equal-allocation audits of 120–240 items covered all six true prevalences at once, against 85–88% with three groups; a Bonferroni widening (about a third wider) raised that to 89 to 94%. A random audit failed outright for the rare group: 291 of 300 draws of 120 items had under 5 audited items in it. Equal numbers per group gave intervals 24% narrower than random at 480 items for K=6 and the same for K=3. The coders are naive-Bayes and a word list, not LLMs, with no group-dependent error; one corpus, one seed; audits above 960 items are a large share of the corpus and the intervals ignore the finite-population correction, so those rows are conservative.

On 3,188 real GoEmotions comments, four automated coders (three cut-offs of a naive-Bayes classifier and a word list; not LLMs) and a hand-coded audit that puts k items on coder-negatives and the rest on coder-positives, weighted back by stratum share: the positives-only audit (k = 0) is biased by the coder's miss rate (−0.128 to −0.169 on a true 0.252) and covers the truth in 0 of 1,000 draws. Auditing any coder-negatives removes the bias, but the 95% interval is short of nominal until about 50 coder-negative items (worst coverage over coders 0.29–0.31 at k = 2, 0.85–0.87 at k = 10, 0.94 at k = 50), and the width at k = 50 is 0.15–0.19 whether the audit has 100 or 400 items in total. Two negatives is worse than none by RMSE (0.216 against 0.152). One code, one corpus, one split; Wald intervals ignoring the finite-population correction.

Re-running qual-negaudit's audit (3,188 real GoEmotions comments, four automated coders that are not LLMs, 200 audited items, 1,000 draws) with four 95% intervals computed from the same draws. With 5 coder-negative items the Wald interval covers the truth in 63% of draws at worst; Wilson, Jeffreys and add-one (plus-two) intervals cover 93–95%, so most of the earlier 'about 50 negatives' requirement was the interval. At 10 items: Wald 0.85, others 0.94–0.97. The cost is width: 0.43 at 5 items (Wilson), 0.17 at 50. One code, one corpus, no LLM; the Wilson interval is a per-stratum combination, not an exact stratified Wilson; worst-of-four coverage is biased low by selection.

On 3,188 real GoEmotions comments, the rare group is the longest 10.8% (344 comments, 23 or more words) and the estimand is the gap in negative-emotion prevalence to the rest (+0.072). Two naive-Bayes coders (not LLMs) have extra errors injected in the rare group only, by flipping each of their labels there with probability 0, 0.1, 0.2 or 0.3. The oracle share of the audit for the rare group stayed between 0.53 and 0.56, and an audit of equal numbers per group gave intervals within 0.7% of the optimal split at every injected rate (200 items: 0.333 against 0.331 with no injection, 0.346 against 0.344 at 0.3). A random audit stayed 1.66 to 1.69 times as wide. Coverage of the 95% interval was 0.95 to 1.00 for stratified designs (0.90 to 1.00 for random) (400 draws per cell, so about ±0.02). Limits: the coder already errs on 42–43% of rare-group items before injection, so random flips cannot raise the error spread much (the error rate reaches 0.50 and then falls back); a coder whose errors are more structured than coin flips is untested. One corpus, one grouping, one seed; no LLM was called.

On 3,188 real GoEmotions comments split by writing register (formal: capitalised and ends in punctuation), a naive-Bayes coder's error rate differs by only 1.9 points (0.282 formal, 0.263 informal) and a lexicon coder's by none, yet the naive register gap is three times too large (−0.043 against −0.0145, coverage 0%). A z test for differential error on a random hand-coded sample of 50 to 800 items flagged 3–6.5% of samples, the false-positive rate, so an ordinary audit cannot see a difference of this size (a rough calculation puts it near 8,000 items per group); prediction-powered inference covers the gap (95.6–96.8%). Register is a typography proxy, not a dialect label, and the coders are not LLMs.

On 3,188 real GoEmotions comments, with the group-wise prediction-powered gap estimator, the design of the hand-coded audit mattered a great deal for a rare group and not at all for a common one. For a short-vs-long split (36% of the corpus in the smaller group) a random audit, equal numbers per group and the variance-minimising split gave interval widths within 1–2% of each other at every size from 100 to 1,600 items. For a rare group (the longest 11% of comments, 352 items) a random audit gave intervals 1.7 times as wide at 100 items (0.75 against 0.44) and 1.2 times at 1,600, left 1–2% of 100-item draws with under 5 audited items in the group, and had worst-case 95% coverage of 0.91 at 100 items against 0.96 for stratified audits. The oracle optimal split (about 55% to the rare group) was indistinguishable from an even split. A z test on the gap bias had less power under random sampling for the rare group (a coder with a gap error was trusted in 25% of draws at 400 items, against 0% stratified). Coders are naive-Bayes classifiers with a moved cut-off and a word list, not LLMs; one corpus, one split, one seed.