Can you do machine learning on qualitative data with an LLM, and when can you trust the numbers?

research/qual-correction

Can you do machine learning on qualitative data with an LLM, and when can you trust the numbers? Simulated LLM coders with known error: an 89%-accurate coder (κ = 0.72) understates a 30% code at 26% with 0% interval coverage, and when it reads one group's answers less accurately the naive group gap has the wrong sign (+3.6 against −4 points, "significant" 80% of the time). Prediction-powered inference with 200 hand-coded answers per group removes the bias and narrows intervals 23–26% against hand-coding alone, but is wider below ~80% coder accuracy. Stdlib tool (κ, Krippendorff's α, PPI). Simulated, no real model.

13 tests

What it found

An LLM can code free text (“does this answer mention cost?”) so that it can be counted and modelled like numbers, but its codes need a correction first. In a seeded simulation, an 89%-accurate LLM coder (Cohen’s κ = 0.72) understates a 30% code at 26%, and its naive 95% interval covers the truth in 0 of 1,000 studies. When the LLM reads one group’s answers less accurately, the naive group gap comes out at +3.6 points against a true −4, the wrong sign, and is “significant” 80% of the time. Prediction-powered inference with 200 random hand-coded answers per group removes the bias, covers the truth 95–98% of the time and gives intervals 23–26% narrower than hand-coding alone. Below about 80% coder accuracy, though, its interval is wider. Includes a stdlib tool (κ, Krippendorff’s α, PPI) and a survey of five ways LLMs bring qualitative data into ML. Simulated coders, not a real model.

A figure from its own output

Figure produced by research/qual-correction: ci_width_vs_accuracy — qual-correction/experiments/ci_width_vs_accuracy.svg, written by the project's own experiment code.

Paper, code and results

More in Machine learning on qualitative data

What happens to a regression coefficient when the regressor is an LLM's code?What happens to a regression coefficient when the regressor is an LLM's code? Naive OLS gives 0.74 for a true 1.00 (0% coverage) and biases a perfectly measured covariate when coder error depends on it; DSL with ~210 hand-coded items fixes both (coverage 94–97%) but narrows the coded coefficient's interval only 14%; per-class PPI fixes 3-class shares. Simulated coders.
Does the prediction-powered correction work on real hand-coded text?Does the prediction-powered correction work on real hand-coded text? On 3,188 GoEmotions comments a naive-Bayes coder (kappa 0.14) is wrong on prevalence (13% against 25%, 0% coverage) and overstates a length gap six-fold; PPI repairs both but is wider than hand-coding alone. Not an LLM (no key).
How much hand-coding can an LLM coder replace, with a valid interval, when its error differs by group and the category is rare?Research programme: valid inference from LLM-coded qualitative data under group shift and rare categories; on 44,485 real tweets a pooled PPI interval covers as little as 67% when the audit's language mix differs from the corpus, a language-stratified one 94 to 96%
How well does an LLM-style coder count negative emotion in real comments, and does the correction still pay?How well does an LLM-style coder count negative emotion in real comments? The Claude session agent, coding 300 GoEmotions comments blind, flags 44% as negative against a human 23% (opposite error to the classical stand-ins); accuracy 0.71 matches naive Bayes and trails a word list, and prediction-powered inference stays valid but 29% wider than hand-coding. One reader, not an API model.
Does an automated coder's error differ across languages, and how much does that move group comparisons?Does an automated coder's error differ across languages? On 44,485 real tweets in Hausa, Igbo, Nigerian Pidgin and Yoruba, an English word-list coder and an English-trained classifier read almost none of the negative sentiment in Hausa, Igbo and Yoruba (sensitivity 0.4–11%), flipping some language gaps' sign (Hausa–Yoruba +10.6 points true, −6.5 reported); PPI keeps coverage at 93–96% but is never narrower than hand-coding. Not an LLM (no key)
How big must a hand-coded audit be before you can trust a coder's counts?How big must a hand-coded audit be before you can trust a coder's counts? On 3,188 GoEmotions comments, a tight equivalence test never passed the coder and a loose one passed it too often (coverage 0.35 at 1,600 audited items). Not an LLM; one corpus.

All 23 Machine learning on qualitative data projects →

Other research topics

Proper scoring & elicitation28 Effort & peer prediction9 Verification games & slashing19 Detection & tolerances14 Markets as routers13 Prediction-market design9 Paying contributors8 Decentralised training systems23 Digital twins & sim-to-real164 LLM chip design loops40 Robot manufacturing loops53 Ambient AI assistant46 Self-replicating manufacturing34 Simulation-first design64 Engineered clothing45 Task-specific AI accelerators40 Algorithmic chip-design loops42 Open-source purpose-built robots44 The software-engineering loop43 Adversarial human imitation41 Open-source residential construction41 Autonomous multi-angle capture34 Sensing, imaging and 3D31 Energy supply and dense energy storage39 Robot societies among people31 Agentic co-design of low-cost robots27 Laundry-hanging robots25 Machine learning on qualitative data23 Robots on the building site27 Civilizations & group psychology1 History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.