How well does an LLM-style coder count negative emotion in real comments, and does the correction still pay?

What it found

The Claude session agent coded 300 real GoEmotions comments (blind to the human labels, one pass; not an API-called model) and over-coded: 132 flagged against 70 by raters, sensitivity 0.81, specificity 0.67, kappa 0.37, so the naive prevalence is 0.44 against a true 0.23 (coverage 0). That is the opposite error from the naive-Bayes and word-list stand-ins of qual-real, which undercount by 10 points. Its accuracy (0.71) is indistinguishable from naive Bayes (exact McNemar p = 0.4) and worse than a 38-word list (0.84, p = 0.0001). Its error differs by comment length (sensitivity 0.95 short, 0.76 long), but prediction-powered inference stays valid (bias under 0.006, coverage 0.96–0.996) and was 29% wider than hand-coding alone at 60 hand-coded items. Limits: one reader and pass, codebook mismatch with the raters' 11-emotion majority, possible dataset exposure, and a 300-item corpus that makes intervals conservative.

A figure from its own output

Paper, code and results

• README and method
• Experiments and results

More in Machine learning on qualitative data

• Does an automated coder's error differ across languages, and how much does that move group comparisons?Does an automated coder's error differ across languages? On 44,485 real tweets in Hausa, Igbo, Nigerian Pidgin and Yoruba, an English word-list coder and an English-trained classifier read almost none of the negative sentiment in Hausa, Igbo and Yoruba (sensitivity 0.4–11%), flipping some language gaps' sign (Hausa–Yoruba +10.6 points true, −6.5 reported); PPI keeps coverage at 93–96% but is never narrower than hand-coding. Not an LLM (no key)
• How big must a hand-coded audit be before you can trust a coder's counts?How big must a hand-coded audit be before you can trust a coder's counts? On 3,188 GoEmotions comments, a tight equivalence test never passed the coder and a loose one passed it too often (coverage 0.35 at 1,600 audited items). Not an LLM; one corpus.
• Do a dozen codebook codes predict better than bag-of-words when labels are scarce?Do a dozen codebook codes predict better than bag-of-words when labels are scarce? On GoEmotions, 12 word-list codes reached 0.65 AUC at 100 labels against 0.56 for bag-of-words, and still led at 1,600 (0.75 against 0.67). Word lists, not an LLM; no learned embedding tested.
• Do LLM-style qualitative codes predict outcomes better than bag-of-words at the same labelling budget?Do LLM-style qualitative codes predict outcomes better than bag-of-words at the same labelling budget? On 5,313 GoEmotions comments, not for a reply-vs-top-level outcome: emotion codes (human, an upper bound) reach AUC 0.527 against 0.557 for words and 0.595 for LSA, and 5% coder error erases their gain. All sets are weak; not an LLM.
• Does a hand-coded audit protect a group gap when it only checks the overall count?Does a hand-coded audit protect a group gap when it only checks the overall count? On 3,188 GoEmotions comments, a coder with zero overall bias still distorted a short-vs-long gap by 0.20, and the overall-bias gate covered the gap 0.4% of the time at 1,600 audited items. Not an LLM; one corpus.
• Can you estimate a group gap from an audit that hand-codes only a few of the coder's negatives?Can you estimate a group gap from an audit that hand-codes only a few of the coder's negatives? Not usefully: 0.9-wide intervals at 5 items, and a plain random audit (0.23 wide) beats the design. Pooling the groups' negatives covers 3%. 3,188 GoEmotions comments, not an LLM.

All 23 Machine learning on qualitative data projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.