A weak coder makes a stratified audit worse

At 2% prevalence on NaijaSenti splitting the audit by the coder's labels narrowed intervals 8% for a good coder, widened them 10% for a poor one.

When a model codes a rare category in a text corpus and people check a sample, you can split that sample by the model's own labels (positive cell, negative cell). Does that pay? We tested it on NaijaSenti tweets, with the "negative" category thinned to 2% prevalence (6,000 items, audit of 1,000, 500 of them a pilot, 1,000 repeated draws).

The answer depends on how informative the coder is (Youden's J, sensitivity plus specificity minus one):

coderJcells vs pooled half-widthcoverage of cells
in-language naive Bayes0.348% narrower93.3%
English-trained naive Bayes0.118 to 11% wider95.1 to 95.8%
English lexicon0.02same93.9 to 95.0%

Pooled intervals sit at 92.8 to 93.5% coverage in all three. The wider intervals at low J are conservative, not wrong. Forcing the split costs width when the coder carries little signal and the allocation is estimated from about ten pilot positives, which is why the library screens on an estimated J first. Coverage gaps near 1 point are inside binomial noise (about 1.4 points) and are not claimed.

Limits: one corpus, one prevalence, three coders, no coder above J 0.34, and the prediction for this run was held in mind rather than committed before it ran. Code, outputs and the log are in research/llm-coding-inference/ (milestone 7p).

All writing · More in Research findings · All tools and research