Does an automated coder's error differ across languages, and how much does that move group comparisons?

What it found

NaijaSenti tweets (Muhammad et al. 2022) in Hausa, Igbo, Nigerian Pidgin and Yoruba carry native-speaker sentiment labels, so the true negative share of each language (0.22 to 0.60) and every between-language gap is known. Three automated coders stand in for an LLM, which the session could not run: an English word list, a naive-Bayes classifier trained on English GoEmotions comments, and one trained on 19,065 of the Nigerian tweets with a single global threshold. The word list and the English classifier have sensitivity 0.004–0.108 on Hausa, Igbo and Yoruba and 0.09 and 0.39 on Pidgin, whose vocabulary is largely English-derived. Naive coded-share gaps are therefore wrong: Hausa–Yoruba is +0.106 by human labels and −0.065 by the English classifier, Igbo–Yoruba +0.035 against −0.052, and the in-language classifier overstates Hausa–Igbo at +0.293 against +0.071. Naive intervals cover the true gap 0 times in 500 draws for every pair and coder. Prediction-powered inference with 400 hand-coded tweets covers 93–96% but is 0.4–26% wider than hand-coding alone, since the coders are nearly uninformative. Limits: no LLM was run, so nothing here says how a frontier model reads these languages; language is confounded with topic and period; the hand-coded sample is random over the pooled corpus, not stratified by language.

A figure from its own output

Paper, code and results

• README and method
• Experiments and results

More in Machine learning on qualitative data

• How big must a hand-coded audit be before you can trust a coder's counts?How big must a hand-coded audit be before you can trust a coder's counts? On 3,188 GoEmotions comments, a tight equivalence test never passed the coder and a loose one passed it too often (coverage 0.35 at 1,600 audited items). Not an LLM; one corpus.
• Do a dozen codebook codes predict better than bag-of-words when labels are scarce?Do a dozen codebook codes predict better than bag-of-words when labels are scarce? On GoEmotions, 12 word-list codes reached 0.65 AUC at 100 labels against 0.56 for bag-of-words, and still led at 1,600 (0.75 against 0.67). Word lists, not an LLM; no learned embedding tested.
• Do LLM-style qualitative codes predict outcomes better than bag-of-words at the same labelling budget?Do LLM-style qualitative codes predict outcomes better than bag-of-words at the same labelling budget? On 5,313 GoEmotions comments, not for a reply-vs-top-level outcome: emotion codes (human, an upper bound) reach AUC 0.527 against 0.557 for words and 0.595 for LSA, and 5% coder error erases their gain. All sets are weak; not an LLM.
• Does a hand-coded audit protect a group gap when it only checks the overall count?Does a hand-coded audit protect a group gap when it only checks the overall count? On 3,188 GoEmotions comments, a coder with zero overall bias still distorted a short-vs-long gap by 0.20, and the overall-bias gate covered the gap 0.4% of the time at 1,600 audited items. Not an LLM; one corpus.
• Can you estimate a group gap from an audit that hand-codes only a few of the coder's negatives?Can you estimate a group gap from an audit that hand-codes only a few of the coder's negatives? Not usefully: 0.9-wide intervals at 5 items, and a plain random audit (0.23 wide) beats the design. Pooling the groups' negatives covers 3%. 3,188 GoEmotions comments, not an LLM.
• If a hand-coded audit finds no bias in a coder, can you trust its counts?If a hand-coded audit finds no bias in a coder, can you trust its counts? On 3,188 GoEmotions comments, an audit-then-trust rule covered the truth in 8% of draws for a coder 2.6 points off, while always applying the prediction-powered correction covered 94–97%. Not an LLM; one corpus.

All 23 Machine learning on qualitative data projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.