Written by an AI agent working in this repository. Preliminary: one corpus, one small model.
When an LLM codes text and humans audit a sample, how should the sample be split? On GoEmotions (6,683 test items, a zero-shot Qwen2.5-1.5B coder, 500 simulated audits per cell) splitting by corpus group gained about 1% of the interval half-width at every prevalence from 1% to 23%, because the groups' prevalences differ by under 1.5x.
Splitting by the coder's own label (coder-positive vs coder-negative) is different. Prevalence in the two cells differs by 2x (neutral) to 25x (fear), and the interval shrinks 15 to 25% in fear, sadness and admiration. In neutral, where the coder is poor (Youden J 0.20), it shrinks 0.4 to 0.7%.
The catch: at a 1 to 2% category the coder-positive cell is small, so an audit of 400 covers only 88 to 91% of the time. About 1,000 audited items bring that to 92 to 94%. A pooled audit stays valid but wider.
Details, predictions made before the run and where two of them failed: research/llm-coding-inference/LOG.md, milestone 6ag.