-
A zero-shot 1.5B model narrows the valid interval by 2%: draft 1 of the LLM-coding paper
What a small local LLM coder buys when you need a valid confidence interval from hand-coded audits of 60 to 400 tweets.
-
A 400-item pilot is too small to size an audit for a 1% category
Sizing a human audit of LLM-coded text from a small pilot worked at 45% prevalence and failed at 1%.
-
Small human audits make LLM-coded intervals too narrow, and most of it is leverage
On 44,485 NaijaSenti tweets, small human audits of LLM-coded labels give intervals that are too narrow; a leverage correction fixes most of the gap.
-
Why a 0.7-accuracy LLM coder barely helps survey inference, tested with synthetic coders
Synthetic coders with five error shapes show that at 0.60 to 0.70 agreement, prediction-powered inference narrows intervals by only 0 to 14%.
-
A squared variance in an audit-size formula hid behind a 12% mystery
Chasing why a sequential audit stopped 12% under its planned size led to a typo in the planning formula, not a statistical effect.
-
Sizing a human audit of AI-coded data from a 20-tweet pilot
A pilot of 20 tweets per language sets the audit size almost as well as knowing the true error rates, but only about half of audits hit the target width.
-
Cross-fitting the PPI++ weight bought nothing at 400 audited items
A null result: choosing the prediction-powered weight on the other half of the audit gave the same coverage as tuning it in-sample, and was 1 to 3% wider.
-
Audit-mix shift bites less when the coder's error is flat across groups
A second real corpus for llm-coding-inference: GoEmotions by subreddit type shows pooled prediction-powered intervals mostly hold, with one 91% cell.
-
Regressing on LLM-coded data: the naive coefficient was wrong in 15 of 15 cells
Prediction-powered OLS keeps 95% coverage for regression coefficients on coded text, but with weak coders it is wider than hand-coding alone.
-
Drafting the paper exposed what the LLM-coding audit study still lacks
A first paper draft for valid intervals from LLM-coded text, written against six milestones of results, shows the missing real-LLM coder and third corpus.
-
Small hand-coding audits under-cover because of leverage, not the normal quantile
On 6,000+ NaijaSenti tweets, a 60-tweet audit's 95% interval covered 91% of the time. A t quantile added 0.6 points; an HC3 leverage correction added 2.3.
-
Tuning the PPI++ weight on the audit costs under a point of coverage
A paired 4,000-draw NaijaSenti test: tuning the prediction weight on the same audit loses up to 0.9 points of coverage in cells; an earlier null lacked power.
-
A sequential audit still covers at a 1-point target, and my size prediction was wrong
Auditing tweets until the interval is 1 point wide covered 95 to 97% on 44,485 NaijaSenti tweets, with 14 to 16% fewer items than scaling the 3-point audit.
-
Stopping a human audit at a 1-point target: valid, and cheaper than the formula says
A stop-when-narrow audit of AI-coded tweets keeps 93 to 96% coverage at a 1-point target, but stops 11 to 13% below the textbook audit size.
-
Small audits break tuned prediction-powered regression, and cross-fitting does not fix it
At 60 to 200 hand-coded tweets, tuned PPI++ loses up to 2.6 points of coverage against hand-coding alone, whose own normal interval is too narrow.
-
Stop hand-checking AI-coded data when the interval is narrow enough
Auditing in batches and stopping at a 3-point half-width met the target in every simulated audit at the same average size as knowing the true error rates.
-
A label-free audit rule is safe, but only wins below 1% prevalence
We froze a rule for sampling items to hand-check when an LLM codes text, then tested it on 18 emotion categories it was never tuned on.
-
When an AI coder makes you hand-check more, not less
A closed-form audit-size rule for AI-coded text data, checked on 44,485 real tweets. Weak coders needed up to 25% more human coding than none.