-
Six ways to repair a reused audit pilot for LLM-coded data, and why none pays
27 pre-registered repairs for conditional undercoverage when an audit pilot picks the design and is then reused: none keeps 90% of the gain and fixes half.
-
Can you tell before an audit how often the rare-category guard will switch on?
A closed-form chance that a pilot passes the rare-category guard matches 36 audit simulations within 3.2 points; my pre-registered forecast missed five.
-
How to audit an LLM coder: the one-paragraph rule, and where it stops being tested
After dozens of small experiments, the rule our programme can defend for auditing LLM-coded text, with the cells where it gains nothing.
-
Does the small-audit rule for LLM-coded data hold for a rare category on a second corpus?
A negative-cell guard keeps confidence intervals valid for rare categories in LLM-coded text; a second corpus at 1% and 3% prevalence agrees.
-
Does a safeguard for LLM-coded audits hold on a second corpus?
The negative-cell guard for coder-cell audits covers 94.0 and 95.8% on NaijaSenti and is within 1% of the pooled width.
-
Does an audit rule tuned on a 1.5B LLM coder still work with a weaker 0.5B one?
With a Qwen2.5-0.5B coder (Youden J 0.26) on GoEmotions, the negative-cell guard covers 95.4 to 96.6% and is 6 to 10% narrower than pooled at n = 2,000.
-
Do you need a screen on a small LLM-coded audit? A guard on three positives does most of the work
With a 1.5B LLM coder on GoEmotions, a guard of 3 pilot positives kept audit intervals valid from 1 to 23% prevalence; the extra screen mostly cost width.
-
Does an LLM-coding audit need its screen? A held-out check on GoEmotions neutral
With a weak zero-shot coder at 23% prevalence, the cell-based audit design covers about 94 to 96% and gains under 1% width over pooled. Harmless, not a win.
-
Why a decent LLM coder barely shrinks the audit, and the guard that keeps it honest
A closed form caps what an LLM coder can save in a hand-coding audit, a fitted control variate captures it, and a three-positive guard keeps rare cases valid.
-
Does a pilot screen cost width when an LLM codes the text? Only when the audit is big
On 6,683 GoEmotions comments coded by a 1.5B zero-shot LLM, a pilot screen cost 17 to 29% interval width at 2,000 audited items and nothing at 500.
-
Does a small LLM coder let you audit a rare category with fewer hand-labels?
Audit planner with a real LLM coder on three GoEmotions categories: valid everywhere, narrower only where the category is not rare.
-
Does a guard that skips useless audit strata still cover at 1% prevalence?
On GoEmotions fear (1.2%) and sadness (2.2%), a pool-unless-3-positives guard covers 94.8 to 95.2% with 8 to 9% narrower intervals.
-
The better the coder, the less you should trust its narrow interval
At 2% prevalence and 10 pilot positives, auditing within the coder's own cells gives 43% narrower intervals for a strong coder, and covers 85% instead of 95%.
-
How much can a coder's labels narrow an audit? A prevalence-free ceiling
Auditing LLM-coded text by the coder's own labels can narrow the interval by at most sqrt(s(1-c))+sqrt((1-s)c), whatever the category's prevalence.
-
Does auditing by the coder's own labels pay on African-language tweets?
An informative coder (J 0.39) on NaijaSenti makes the audit rule pick coder cells, yet the interval is only 0.6% narrower; the GoEmotions gain did not transfer.
-
Tuning the threshold did not rescue a weak LLM coder
A negative result from the llm-coding-inference programme - tuning a small zero-shot LLM's cutoff on 2,000 labels left it too weak for a better audit design.
-
A 92% coverage cell that turned out to be noise
A 500-draw simulation put an interval's coverage at 92.0% against a 95% target. A 2,000-draw re-run gives 94.1%, so the dip was Monte Carlo noise.
-
500 simulation draws can mislead by two points: rechecking an audit's coverage
An audit interval showed 92.0% coverage in 500 draws and 94.1% in 2,000. A rerun of the same cell shows how far a small simulation can wander.
-
A pilot screen that decides whether to stratify an LLM-label audit
A 30% pilot estimates the LLM coder's informedness; pool when it is empty, use coder cells when informative. Tested on GoEmotions and NaijaSenti.
-
A two-line rule for when to stratify a hand-coding audit
Pilot 30% of the audit, then use the coder's own labels as strata only if it looks informative and the pilot holds 20 positives. The rule is now library code.
-
Does the audit-planning code give the same answer as the experiments?
The audit-design rule for LLM-coded text is now library code. Run through the library, it matches the earlier experiment: half-width within 2% in 14 cells.
-
Does the 0.3 informedness cutoff matter? A sweep on LLM-coded audits
Sweeping the J_hat cutoff of a pooled-vs-coder-cell audit screen from 0.1 to 0.5 moved half-widths by under 1% once a prevalence guard was in place.
-
Is the 0.3 informedness cutoff a knife edge? Not between 0.2 and 0.5
Sweeping the cutoff that splits an audit by an LLM coder's labels: width moves under 1% from 0.2 to 0.5; only 0.1 is too eager.
-
One call from a hand-coded pilot to the rest of the audit
llmci.plan_audit takes the coder's codes and a 30% hand-coded pilot, says whether to pool or split the audit by coder label, and how many items to label next.
-
A pilot-allocated LLM audit looks better than the oracle and is wrong
Allocating an LLM-coded data audit from a 30% pilot gives intervals 25 to 48% narrower but coverage as low as 49%. A negative result on real GoEmotions data.
-
Audit by the coder's own label: when stratifying a hand-coded sample pays
Stratifying an LLM-coded audit by corpus group buys nothing; stratifying by the coder's own label cuts the interval 15 to 25% when the coder is informative.
-
Auditing LLM labels by the coder's own cells: no gain when the coder carries no signal
Splitting the human audit by a zero-shot Qwen coder's own label gains nothing on NaijaSenti, and a pilot floor widens intervals up to 22%.
-
Does stratifying a human audit of LLM codes pay when the category is common?
On two common GoEmotions categories, four audit strata moved interval width by about 1%, and the LLM coder's gain tracked its accuracy.
-
Can you plan a coder-assisted audit from a pilot? Mostly not
Two repairs for a pilot-estimated audit of LLM-coded text, a per-cell floor and an independent pilot, tested on GoEmotions with a real Qwen coder.
-
Should you stratify the human audit of an LLM-coded rare category? Not at 1% prevalence
On GoEmotions fear (1.15% prevalence), splitting a 400-item human audit into four strata lifted nothing and cost coverage; one pooled sample covered 95.8%.
-
A real LLM coder barely shrinks a rare-category audit
Qwen2.5-1.5B coded 6,683 GoEmotions comments for fear and sadness. With twice the informedness of naive Bayes it cut the human audit by at most 12%, not 3x.
-
A valid sequential audit that uses the coder costs more than one that ignores it
A betting confidence sequence makes a coder-assisted rare-category audit valid (99-100% coverage) but needs 3 to 14 times the oracle sample.
-
A Wilson interval for LLM-coded audits of rare emotions
On GoEmotions fear and sadness, a pooled PPI estimate with a Wilson interval on its effective sample size covers 92 to 97% at audits of 400 to 1,000 labels.
-
One PPI weight for all strata: what it fixes in a rare-category audit, and what it does not
With a real LLM coder on GoEmotions rare categories, one pooled PPI weight removes the undercoverage per-stratum tuning added; a variance floor over-corrects.
-
Estimating a 1% label from LLM codes: a valid sequential audit exists, and still costs more than labelling by hand
On GoEmotions fear and sadness a stratified coder-assisted sequential audit covers 100% but needs 4,500 to 5,800 labels; pooled human-only needs 1,100 to 5,000.
-
Stopping early at a rare category is survivable with the right interval
With pooled human labels, stopping an audit at 1 to 2% prevalence covers 90 to 99.6% with plug-in, Wilson and betting intervals; the PPI estimator collapsed.
-
A rare-category audit sized without any human label stays valid, at about twice the cost
Sizing a rare-category audit from the coder's own positive share covered 91.8 to 94.4% at 1.9 to 2.8 times the oracle size, where stopping rules failed.
-
Stopping an audit when the interval looks narrow fails at 1% prevalence
A sequential audit that worked at 45% prevalence covers 64 to 90% for fear and sadness in GoEmotions, and a minimum-positives guard makes it worse.
-
Making a thin pilot safe costs 1.5 to 3.8 times the audit
A conservative variance estimate fixes the undercoverage of pilot-sized human audits of LLM-coded rare categories, at a steep price.
-
Size an LLM-coding audit from a 400-item pilot
On 6,000 Nigerian tweets with three real Qwen coders, a 400-tweet pilot plans a hand-coding audit within 1% of the oracle size, and the coders save at most 5%.
-
Tune the control variate before you size the audit
On 6,683 Reddit comments, a control-variate weight per group stops an LLM coder from inflating the audit you need, but with a mediocre coder saves only 2 to 9%.
-
Widening the interval does not fix an early-stopping audit at a rare category
Stopping an LLM-coding audit when the interval looks narrow undercovers at 1 to 2% prevalence; a Bonferroni-wide interval still covers 64% in the worst cell.
-
Does calibrating an LLM coder pay for the human labels it uses?
Choosing a Yes/No cut for a small local LLM on 100 human labels, then charging those labels to the comparison, still loses to simply hand-coding more items.
-
How accurate must an LLM coder be to save you human labels?
A simulation on 6,000 real tweets shows an LLM coder's accuracy matters less than whether its mistakes track the text it codes.
-
How good must an LLM coder be before it shrinks your audit? The paper draft's answer
Draft 1.1 of the llm-coding-inference paper: about 0.80 agreement, errors unrelated to length or label, is needed to pay; small Qwen coders fall short.
-
Does the shape of an LLM coder's errors matter for valid inference?
Four synthetic LLM-coder error shapes at matched accuracy all still narrow PPI++ intervals; the real coders' null result is not explained by them.
-
A bigger zero-shot LLM coder did not narrow the interval: the threshold did
Qwen2.5 at 0.5B, 1.5B and 3B coded 6,000 Nigerian tweets; the valid PPI++ interval was narrowest for the middle model, because of where its Yes/No cut fell.
-
A real LLM as the coder: 2% narrower intervals, not 20%
Qwen2.5-1.5B as the coder in prediction-powered inference narrowed intervals by only 2%, far short of the hypothesised 20%.
-
Why small LLM coders didn't shrink our intervals: agreement was the wrong number
On 6,000 Nigerian tweets, three Qwen coders at 66 to 69% agreement gave almost no gain in prediction-powered inference. Youden's J, not agreement, explains it.
-
How many items to hand-code: the audit-size rule holds on a second corpus, and says a weak coder costs you
A closed-form rule for how many items to hand-check when an LLM codes the rest matched simulation to 0.2% on Reddit comments. A weak coder needs more.