How much hand-coding can an LLM coder replace, with a valid interval, when its error differs by group and the category is rare?

2026-10-31 · Milestone 7e: the 2,000-draw check moves into the paper and README (a writing step, no experiment)

• Why: 7d's "Next" said to fold the library audit path into the paper's recipe. The paper's Reproducibility section and the README quoted 7c's 92.0% lowest cell but not the 2,000-draw re-run that showed it was noise.
• Done: paper/paper.md (Section 10) and README.md now state 7d's result (coverage 94.1%, se 0.5; half-width 1.165 points) next to the 7c numbers. Every figure is copied from experiments/h7d_admiration_2000.txt. No claim changed; the caveat that the screen has no coverage theorem stays.
• Next: an informative coder on a second corpus once weights are reachable.

2026-10-30 · Milestone 7d: does the library path really cover below 94.6% on admiration at n = 1000? (planned before running)

• Why: 7c's "Next". 7c's 500 draws gave 92.0% against 6al's 94.6%, a 2.6-point gap that is about two sd of a difference. experiments/h7d_admiration_2000.py runs 2,000 fresh draws (seed B.SEED + 7) of the same library audit on that one cell; the standard error at 2,000 draws is about 0.5 points.
• Prediction (before running): coverage 93.0% or higher (noise, as 7c read it), half-width within 3% of 1.152. Refuted if coverage is under 93.0%, which would mean the library's plain-variance allocation really undercovers there and the screen's validity claim needs the lambda-tuned S2 back.

### Outcome (experiments/h7d_admiration_2000.py, about 5 min; experiments/h7d_admiration_2000.txt)

• Coverage 94.1% (se 0.5) over 2,000 draws; by block of 500: 94.6, 94.6, 93.8, 93.4%. Half-width 1.165 points against 7c's 1.152 (1.1% wider); cells used in 86% of draws. 6al's 94.6% is 1 se from this.
• Prediction held (coverage at least 93.0%, width within 3%): 7c's 92.0% was Monte Carlo noise; the library path does not undercover on this cell. The last block reaching 93.4% shows 500 draws can wander by more than a point either way.
• Limits: one cell, one coder; still no theorem for the data-dependent screen; no informative coder on a second corpus (weights unreachable).
• Next: an informative coder on a second corpus once weights are reachable; fold plan_audit / audit_interval into the paper's recipe section.

2026-10-30 · Milestone 7c: audit_interval, and does the library path (plan_audit -> draw -> interval) reproduce 6al? (planned before running)

• Why: 7a and 7b put the design rule in the library, but the interval that closes the audit (pooled lambda, stratified variance, Wilson on n_eff; 6ac, 6ad) still lived in experiment scripts, and nothing had checked the library path end to end. Added audit_interval(plan, f_all, labels) and a test that it equals the stratified estimator with the pooled weight fixed plus the Wilson n_eff formula (20 tests pass). experiments/h7c_end_to_end.py runs the whole audit through the library only on 6al's corpora, cached Qwen codes, seed rule, n = 400 and 1000, 500 draws.
• Prediction (before running): coverage within 2 points and half-width within 3% of 6al's "screen + guard G=20" WE row in every cell. Known difference that could move it: plan_audit estimates a cell's variance as the plain sample variance of y - f, 6al used the lambda-tuned S2 with plus-one smoothing, so the allocation, and so the width in cells-draws, may differ a little. Refuted if any cell moves more than that.

### Outcome (experiments/h7c_end_to_end.py, about 4 min; experiments/h7c_end_to_end.txt; 500 draws per cell)

• Half-width: within 2.0% of 6al's G = 20 WE row in all 14 cells (largest: fear n = 400, 1.014 vs 1.035; admiration n = 1000, 1.152 vs 1.166, cells used in 87% of draws against 86%). The 3% bound held. Share of draws using cells matches 6al to a few points (NaijaSenti 1.5B n = 400: 18% vs 16%).
• Coverage: 92.0 to 96.6% across cells; 12 of 14 within 2 points of 6al's, and two outside it: admiration n = 1000 (92.0 vs 94.6, the library's lowest) and NaijaSenti 0.5B n = 400 (96.0 vs 93.2, the library higher). Independent 500-draw samples differ by about 1.4 points (sd of a difference), so 2.6 and 2.8 are about two sd among 14 comparisons; I read this as noise, not as a different estimator, but a sample of 2,000 draws at admiration n = 1000 would settle whether the library path really sits 1 to 2 points under 6al's 94.6%. Prediction: width held, coverage held in 12 of 14 cells, refuted as stated in two.
• Reading: the library path (plan_audit, then audit_interval) gives the 6al numbers within Monte Carlo noise, including with the plain-variance allocation rather than 6al's lambda-tuned S2 (no width difference visible). The coverage caveat of 6al stands: no theorem, one admiration cell at 92%.
• Limits: same corpora and coders as 6al, so nothing new about transfer; still no informative coder on a second corpus (weights unreachable: Hugging Face 403, no torch).
• Next: 2,000 draws at admiration n = 1000; an informative coder on a second corpus once weights are reachable.

2026-10-30 · Milestone 7b: plan_audit, one call from a pilot to the rest of the audit

• Why: 7a exported screen and cell_allocation, but a user still had to glue the pilot, the cell split and the per-cell variances together. plan_audit(f_all, pilot_y, n) returns the design ("pooled" or "cells"), the cells as item indices and how many more items to label in each, counting pilot items toward n.
• Test added (19 pass): a good coder (J about 0.85, 3,000 items, 120-item pilot) gets two cells covering every item once, with to_label summing to about 400 minus the pilot; a random-noise coder pools and gets 280 more items. No new experiment, so no new claim; the rule's status in the paper is unchanged (heuristic, no coverage theorem). Per-cell variance is the plain sample variance of y - f in the pilot, not the lambda-tuned S2 of the experiment code, so allocations can differ slightly from h6ai_floor_indep.est_s2; not compared.
• Next: an informative coder on a second corpus once model weights are reachable.

2026-10-30 · Milestone 7a: the screened-audit rule becomes library code (llmci/audit_design.py)

• Why: 6ak to 6am settled the rule (pilot 30%, J_hat >= 0.3, at least 20 pilot positives, else pool; coder cells get a 0.35 floor) but it lived only inside experiment scripts, so nobody could use it without copying them.
• Added informedness, screen, cell_allocation (stdlib) and 3 tests (known J values and degenerate pilots, the two-condition screen including the perfect-coder-but-10-positives case, allocation sums to budget / honours floor / caps at cell size). 18 tests pass.
• Check against the evidence code: cell_allocation equals h6ai_floor_indep.floored(h6r_rare_pilot.allocate(...)) on 2,000 random (sizes, variances, n in {400, 1000}) inputs, 0 differences. No new experiment, so no new claim; the rule keeps the status the paper gives it (heuristic, no coverage theorem).
• Next: an informative coder on a second corpus once model weights are reachable (Hugging Face still returns 403 here).

2026-10-30 · Milestone 6am: is the guarded screen's J_hat cutoff (0.3) a knife edge? (planned before running; ran straight through)

• Why: 6ak chose 0.3 before looking and never swept it (6al's "Limits"). experiments/h6am_tau.py reruns 6al's screen + guard G = 20 with TAU in {0.1, 0.2, 0.3, 0.4, 0.5}; same data, cached Qwen scores, seed, rules and 500 draws (+/- 2 points). A second corpus with a new informative coder was not possible this session (Hugging Face returns 403 through the proxy and torch is not installed), so this closes the cheaper limit instead.
• Prediction (before running): WE coverage stays at least 92.5% and half-width within 3% of the TAU = 0.3 row in every cell, except that TAU = 0.1 picks cells more often on NaijaSenti 1.5B and 3B and is up to 3% wider there. Refuted if any cell moves more.

### Outcome (experiments/h6am_tau.txt)

• GoEmotions admiration, fear and sadness are identical across TAU 0.1 to 0.4 (the guard decides: cells 0% at n = 400, 86% for admiration at n = 1000; fear and sadness 0%). Admiration at TAU 0.5 loses 2 draws in 100 to pooling, half-width 1.167 vs 1.166.
• Neutral (J about 0.17) is the only category where TAU matters: cells in 74% / 87% of draws at 0.1 (n = 400 / 1000), 13% / 2% at 0.3, 0% at 0.5, with half-width 3.928 to 3.955 (n = 400) and 2.370 to 2.381 (n = 1000), under 1% apart. Coverage 93.8 to 96.4%.
• NaijaSenti: 0.5B (true J 0.008) never uses cells at TAU 0.2 and up (0.1: 2% of draws at n = 400, half-width 4.435 vs 4.421). 3B: TAU 0.1 uses cells in 18% of draws at n = 400 and is 3.9% wider (4.534 vs 4.364); from 0.2 it is level with 0.3. 1.5B (J 0.208) goes from cells in 87% / 97% of draws at 0.1 to 0% at 0.4, with half-width 4.315 to 4.325 (n = 400) and 2.585 to 2.590 (n = 1000); coverage 94.0 to 96.6%.
• Lowest WE coverage in any cell is 92.6% (NaijaSenti 0.5B, n = 1000, identical at every TAU, so not a TAU effect). Prediction held except one cell: NaijaSenti 3B at TAU 0.1, n = 400 is 3.9% wider, over the 3% bound.
• Reading: with the prevalence guard in place, any TAU from 0.2 to 0.5 gives the same answer to within 1% in width; 0.3 is not a knife edge, and the one place the cutoff has any effect (a coder with J near 0.2) it costs under 1% either way. 0.1 is a little too eager on an empty coder.
• Limits: the same four coders and two corpora as 6al; no new informative coder on a second corpus (still open, needs model weights); 500 draws.
• Next: an informative coder on a second corpus once weights are reachable.

2026-10-30 · Milestone 6al: does a prevalence guard (cells only if the pilot holds at least G human positives) make 6ak's screen valid on rare categories? (planned before running)

• Why: 6ak's "Next". experiments/h6al_guard.py adds to 6ak's screen the condition "pilot human positives >= G", G in {10, 20}, else one pooled stratum. Same data, coders, rules, seed, n = 400 and 1000, 500 draws (+/- 2 points).
• Predictions (before running): (1) With G = 20 the screen covers at least 93% (WE) in fear and sadness at both n, since it almost always pools there. (2) On admiration at n = 1000 (about 24 pilot positives of 300) G = 20 still uses cells in over half of draws and keeps at least 70% of the gain; at n = 400 (about 10 pilot positives) G = 20 pools in over 90% of draws, so the gain is lost there. (3) G = 10 is a middle: admiration n = 400 uses cells in over half of draws, fear/sadness at n = 1000 may still undercover. (4) NaijaSenti and neutral are unchanged from 6ak (within 0.5 point and 1% width).
• Refuted: (1) any fear/sadness cell under 93%; (2) cells used in half of draws or fewer at n = 1000, or kept gain under 70%, or cells used in 10% or more at n = 400; (3) G = 10 admiration n = 400 uses cells in half of draws or fewer; (4) any difference beyond those bounds.
• Limits known in advance: the guard is a design-time heuristic with no coverage theorem; one coder per GoEmotions category; 500 draws.

### Outcome (experiments/h6al_guard.py, about 8 min; experiments/h6al_guard.txt; 500 draws per cell, coverage +/- 2 points; WE = Wilson on n_eff)

• The guard makes the screen valid on the rare categories by pooling. Fear WE coverage 96.0 / 94.2% (n = 400 / 1000) for both G, against 75.4 / 87.0% for the plain screen; sadness 93.2 / 96.0% (G = 20), 93.2 / 96.4% (G = 10), against 87.4 / 93.8%. Cells are used in 0 to 12% of draws there; widths are the pooled ones (fear 1.035 vs 1.024 at n = 400; 0.601 vs 0.605 at n = 1000).
• It costs the common-category gain at n = 400 and part of it at n = 1000. Admiration, n = 1000: G = 20 uses cells in 86% of draws, half-width 1.166 against 1.415 pooled and 1.122 unguarded (keeps 85% of the gain), coverage 94.6% (unguarded 93.4%). G = 10 at n = 1000 is identical to the unguarded screen. At n = 400, G = 20 pools every draw (cells 0%, half-width 2.354 vs 2.350 pooled, coverage 95.6% against 92.6% unguarded); G = 10 uses cells in 50% of draws, half-width 2.110 (10% narrower than pooled), coverage 95.4%.
• Neutral, the three NaijaSenti coders: identical to 6ak to three digits for both G (the guard never binds there).
• (1) held: fear and sadness at least 93.2% at both n. (2) held: 86% cells and 85% of the gain kept at n = 1000; 0% cells at n = 400. (3) refuted on a tie: G = 10 admiration n = 400 uses cells in exactly 50% of draws ("over half" was the bar); fear/sadness at n = 1000 covered 94.2 / 96.4%. (4) held: no difference at all.
• Reading: "J_hat >= 0.3 and at least 20 pilot positives" is the screen to use: it is never invalid where pooling is valid, keeps 85% of the coder-cell gain at 8% prevalence and n = 1000, and gives the gain up (with a gain in coverage, 92.6 to 95.6%) at n = 400. G = 10 buys back half the n = 400 gain.
• Limits: G chosen before looking (two values, no sweep); one informative coder and category (admiration) carries the cells-win half; 500 draws; no coverage theorem for the data-dependent choice.
• Next: an informative coder on a second corpus; fold the guarded screen into the paper.

2026-10-30 · Milestone 6ak: can a pilot-estimated informedness screen (J_hat >= 0.3) choose between pooled and coder-cell audits, keeping 6ai's gain and avoiding 6aj's waste? (planned before running)

• Why: 6aj's "Next". experiments/h6ak_screen.py: the 30% pilot (pilot-in-audit) gives J_hat = sensitivity + specificity - 1; if J_hat >= 0.3 use two coder cells with a 0.35 floor (6ai), else one pooled stratum (pilot plus fresh items to n). Compared with always-pooled and always-cells(0.35) on GoEmotions admiration / neutral / fear / sadness (Qwen2.5-1.5B, score>0) and NaijaSenti negative share (three Qwen coders), n = 400 and 1000, 500 draws (+/- 2 points), same seed and rules as 6ai/6aj.
• Predictions (before running): (1) On NaijaSenti (true J 0.06 or below) the screen picks cells in under 10% of draws and its WE half-width is within 3% of pooled. (2) On admiration at n = 1000 (true J about 0.65) the screen picks cells in over 95% of draws and keeps at least 90% of the always-cells gain over pooled. (3) Screen WE coverage is at least 91% in every admiration/neutral/NaijaSenti cell. (4) In fear and sadness the screen is no more valid than always-cells at n = 400 (the pilot rarely sees positives, so J_hat is noisy); coverage there stays under 93%.
• Refuted: (1) cells chosen in 10% or more of draws, or half-width over 3% wider; (2) cells chosen in 95% or fewer of draws, or kept gain under 90%; (3) any such cell under 91%; (4) screen covers 93% or more in fear or sadness at n = 400, or beats always-cells there.
• Limits known in advance: one threshold (0.3) chosen before looking, two corpora, zero-shot coders, 500 draws, no coverage theorem for the data-dependent choice.

### Outcome (experiments/h6ak_screen.py, about 6 min; experiments/h6ak_screen.txt; 500 draws per cell, coverage +/- 2 points; WE = Wilson on n_eff)

• Where the coder is empty the screen is pooled: NaijaSenti 0.5B and 3B (true J 0.008 / 0.061) choose cells in 0% of draws and match pooled half-width to three digits (4.421 vs 4.421, 4.364 vs 4.360 at n = 400; 2.650 vs 2.649, 2.613 vs 2.613 at n = 1000), where always-cells was 22% wider at n = 400. 1.5B (J 0.208) picks cells in 6 to 16% of draws, half-width level with pooled. Coverage 92.6 to 96.4%.
• Where the coder is informative and the category is common the screen keeps the gain: admiration (J 0.652) picks cells in 99% / 100% of draws; half-width 1.875 / 1.122 against 2.350 / 1.415 pooled (20% narrower), the same as always-cells (1.863 / 1.122). Coverage 92.6 / 93.4%. Neutral (J 0.173) mostly pools (cells in 13% / 2%), coverage 94.2 to 96.2%.
• Rare categories remain invalid. Fear (J 0.506) picks cells in 47% / 70% of draws: coverage 75.4 / 87.0%; sadness 68% / 81%: 87.4 / 93.8%. Pooled covers 94.0 to 95.6% there. The screen is better than always-cells (50.4 / 81.8, 83.4 / 92.2) but not valid at n = 400, and a J_hat from a pilot with one or two positives cannot tell an informative coder from luck.
• (1) held for 0.5B and 3B (0% cells; widths within 0.1%). (2) held: 99 to 100% cells; the whole gain kept. (3) held: lowest WE coverage in those cells is 92.6%. (4) refuted in its second clause: coverage stays under 93% as predicted, but the screen covers 25 and 4 points better than always-cells in fear and sadness at n = 400.
• Reading: a J_hat >= 0.3 screen from the 30% pilot gives the one rule that is never worse than pooled in width and valid where prevalence is 8% or more: pool when the coder is empty, use coder cells with a 0.35 floor when it is informative. For categories near 1 to 2% prevalence the screen does not rescue validity; pool.
• Limits: threshold 0.3 fixed before looking, no sweep; two corpora, but the informative-coder evidence is still the one admiration category on one model (the NaijaSenti coders are all weak); 500 draws; no coverage theorem for the data-dependent choice.
• Next: an informative coder on a second corpus (a tuned threshold or fine-tuned small model on NaijaSenti) so the "cells when informative" half is not resting on admiration alone; a prevalence guard (pool unless the pilot holds at least ~20 positives); fold the screen rule into the paper.

2026-10-29 · Milestone 6aj: does 6ai's coder-cell floor rule transfer to a second corpus (NaijaSenti, negative share, three Qwen coders)? (planned before running)

• Why: 6ai's "Next". A second common-prevalence category with cached real-LLM codes: the 6,000-tweet NaijaSenti subsample of h3_llm_width.corpus(), target = negative share, coders = Qwen2.5 0.5B / 1.5B / 3B at score > 0. experiments/h6aj_naija_floor.py reuses 6ai's sim_floor (30% pilot, floor 0.2 or 0.35 of n) and 6ae's sim (pooled; two coder cells with oracle Neyman allocation), n = 400 and 1000, 500 draws (+/- 2 points).
• Predictions (before running): (1) a 35% floor gives WE coverage of at least 93% for every coder at both n (prevalence is far above 8%, so no cell starves). (2) The oracle two-cell design is at least 5% narrower than pooled only for a coder whose agreement with the human label is clearly above chance-level informedness; the 0.5B coder gains under 5%. (3) Pilot floor 0.35 keeps at least half of the oracle gain wherever the oracle gain is 5% or more. (4) Pilot floor 0.2 covers within 2 points of floor 0.35 here (no rare cell).
• Refuted: (1) any cell under 93%; (2) 0.5B gains 5% or more, or every coder gains under 5%; (3) kept gain under half in any such cell; (4) a gap over 2 points.
• Limits known in advance: one corpus subsample, zero-shot coders with a fixed threshold, a fixed corpus (only the audit is random), WE has no coverage theorem.

### Outcome (experiments/h6aj_naija_floor.py, about 6 min; experiments/h6aj_naija_floor.txt; 500 draws per cell, coverage +/- 2 points)

• Negative share 0.327 in the 6,000-tweet corpus. The three zero-shot coders agree with the human label on 66.8 / 66.1 / 68.9% of tweets (0.5B / 1.5B / 3B), against 67.3% for always answering "not negative": they carry almost no signal for this target. The 0.5B and 3B coders flag only 151 and 185 tweets as negative; 1.5B flags 1,794.
• No two-cell gain exists. The oracle two-cell WE half-width equals the pooled one to three digits for 0.5B and 3B (4.421 vs 4.421, 2.649 vs 2.649 at n = 400 / 1000) and is 0.2% wider for 1.5B (4.325 vs 4.317; 2.585 vs 2.588). Oracle coverage is 93.4 to 97.4%.
• The pilot floors cost width. At n = 400 floor 0.35 is 22% wider than pooled for 0.5B (5.396 vs 4.421), 22% for 3B (5.314 vs 4.360) and level for 1.5B (4.328 vs 4.317); at n = 1000, 7% wider (2.837 / 2.857 vs 2.649 / 2.613) for 0.5B and 3B. A floor that forces 35% of n into a 2.5%-of-corpus cell spends audit items where almost nothing is learned.
• Coverage of the floor designs is 93.8 to 97.4% in all 12 cells (pooled WE at n = 400 is 91.6 to 92.6%, so the pilot designs are, if anything, over-covering: wider intervals).
• (1) held: floor 0.35 covers 94.2 to 96.0% at n = 400 and 94.6 to 96.0% at n = 1000. (2) refuted by its second clause: every coder gains under 5% (all about 0%), the 0.5B coder included. (3) vacuous: no oracle gain to keep. (4) refuted: floor 0.2 covers 3.2 points above floor 0.35 for 0.5B at n = 400 (97.4 vs 94.2); the other five pairs are within 1.1 points.
• Reading: 6ai's rule does not "transfer" because on this corpus there is nothing to transfer: the coder-cell gain needs a coder with real informedness, and these zero-shot Qwen coders have none at score > 0 (the same finding as 5h). The positive result of 6ag to 6ai therefore stands on one corpus, one category (admiration) and one model; a second corpus with an informative coder is still open. Practical rule: check the coder's informedness before stratifying on its cells; with an uninformative coder use one pooled stratum, and never spend a floor on a tiny coder-positive cell.
• Limits: one corpus subsample, uninformative zero-shot coders at a fixed threshold, 500 draws, fixed corpus, WE has no coverage theorem.
• Next: an informative coder on a second corpus (a tuned threshold on NaijaSenti, or a fine-tuned small model) to separate "the rule does not transfer" from "this coder is empty"; a screening step (informedness estimated from the pilot) that chooses pooled or coder cells.

2026-10-29 · Milestone 6ai: do a floor on each coder cell's allocation, or an independent pilot, repair 6ah's pilot-estimated two-cell audit? (planned before running)

• Why: 6ah's "Next". experiments/h6ai_floor_indep.py: pilot of 0.3 n estimates per-cell S2 as in 6ah; (a) floor: Neyman allocation but no cell below 20% or 35% of n; (b) independent pilot: pilot items only estimate S2 and are not in the estimate, the other 0.7 n is a fresh audit (same floors); total humans stay n. Same categories, coders, rules, seed, 500 draws, n = 400 and 1000 as 6ah.
• Predictions (before running): (1) A 35% floor restores WE coverage to at least 91% in admiration and neutral at both n, and at fear/sadness at n = 1000 (the starved cell was the cause). (2) A 35% floor keeps at least 8% of the WE half-width gain over 1 pooled in admiration at n = 1000. (3) The independent pilot at the same floor is no more valid than the pilot-in-audit design (the allocation, not the reuse, was the problem) but loses 10 to 20% of the gain from the smaller audit. (4) Fear and sadness at n = 400 stay under 93% for every pilot design.
• Refuted: (1) any of those cells under 91%; (2) kept gain under 8%; (3) independent pilot covers 3 or more points better, or its gain loss is outside 10 to 20%; (4) any such cell 93% or more.
• Limits known in advance: one corpus, one 1.5B model; 500 draws (+/- 2 points); one pilot share; WE has no coverage theorem.

### Outcome (experiments/h6ai_floor_indep.py, about 25 min; experiments/h6ai_floor_indep.txt; 500 draws per cell, coverage +/- 2 points)

• A 35% floor rescues the common category and nothing rarer. Admiration (8%), WE coverage with the pilot-in-audit floor-0.35 design: 90.8 / 92.2% at n = 400 (score>0 / 3x cut) and 93.4 / 93.6% at n = 1000, against 83.8 / 86.6% and 92.6 / 91.4% at floor 0.2 (6ah: 80.8 / 83.8% and 91.2 / 92.4%). Half-width at n = 1000 is 1.122 / 1.121 against 1.415 / 1.423 pooled: 21% narrower, the oracle's 19 to 20%. Neutral is valid everywhere (93.0 to 95.8%) and gains nothing.
• Fear and sadness stay invalid: fear WE coverage 50.4 / 66.6% at n = 400 and 81.8 / 90.6% at n = 1000; sadness 83.4 / 68.0% and 92.2 / 89.6%. The floor only helps at n = 1000 with the weaker coder cut.
• The independent pilot buys validity by giving up the gain: at admiration, n = 1000, half-width 1.426 / 1.436 against 1.415 / 1.423 pooled (no gain; the 0.3 n spent on the pilot cancels the allocation gain), coverage 94.4%. In fear and sadness it is still invalid (53 to 70% at n = 400, 86.6 to 93.6% at fear and 89.2 to 89.6% at sadness, n = 1000) and wider than the pooled interval (sadness 0.897 / 0.779 against 0.793 / 0.810).
• (1) refuted: admiration at n = 400 is 90.8% (score>0), under 91%; fear n = 1000 is 81.8 / 90.6% and sadness 3x cut 89.6%. (2) held: 21% kept. (3) refuted: the independent pilot loses all the gain, not 10 to 20%, and it covers 4.8 points better in fear n = 1000 (score>0). (4) held: every rare n = 400 cell is under 70% for the pilot designs.
• Reading: the 6ag gain survives estimation only for a category near 8% with a floor of about a third of n and n of 1000 or more; below that, use one pooled stratum. A fresh pilot costs as much as it gains. So the advice stands: pool for rare categories; for an 8%-prevalence category with an informative coder, coder cells with a 35% floor and a pilot-in-audit are the one working pilot-based design, 21% narrower at n = 1000.
• Limits: one corpus, one 1.5B model; 500 draws; one pilot share (30%) and two floors; the pilot-in-audit design has no coverage theorem; a single category (admiration) carries the positive result.
• Next: a second common category on another corpus (for example Reddit subreddit groups) to see if the floor rule transfers; fold into the paper (done, one sentence).

2026-10-29 · Milestone 6ah: if the two-coder-cell audit's allocation is estimated from a 30% pilot rather than an oracle, how much of 6ag's 15 to 25% width gain survives, and does coverage hold? (planned before running)

• Why: 6ag's "Next". 6ag used oracle Neyman allocation (true per-cell S2). experiments/h6ah_pilot_cells.py: a simple random pilot of 0.3 n items from the pool is hand-coded, per-cell S2 estimated from it (with one pseudo-pair per label so S2 > 0), the rest allocated Neyman on those estimates, pilot items counted in their cell. Compared with oracle allocation, size-proportional allocation over the same two cells, and one pooled stratum. Same four categories, coders, rules, seed, 500 draws, n = 400 and 1000 as 6ag. Mean humans used is reported per design (allocation is capped by cell size and rounded, so the totals can differ from n).
• Predictions (before running): (1) pilot allocation keeps at least half of the oracle gain in WE half-width over 1 pooled in admiration and fear at n = 1000 (oracle 19 to 21%, so pilot at least 10%). (2) Size-proportional over two cells gains under 5% over pooled everywhere (the gain comes from allocating by S2, not from the split). (3) WE coverage under pilot allocation at n = 400 is below 93% in fear and sadness (few positives in the pilot, noisy S2), and at least 93% in admiration and neutral. (4) Pilot totals stay within 5% of n.
• Refuted: (1) kept gain under 10%; (2) any cell 5% or more; (3) fear or sadness 93% or more, or admiration or neutral under 93%; (4) a total off by more than 5%.
• Limits known in advance: one corpus, one 1.5B model; 500 draws (+/- 2 points); the pilot is not reused as an independent sample, so the allocation depends on the same data the estimate uses (the estimator is still conditionally unbiased given cell sample sizes, but WE has no coverage theorem here); pilot share fixed at 30%.

### Outcome (experiments/h6ah_pilot_cells.py, about 4 min; experiments/h6ah_pilot_cells.txt; 500 draws per cell, coverage +/- 2 points)

• Pilot allocation looks like a bigger win than the oracle and is not valid. WE half-width against 1 pooled stratum at n = 400: fear 47% / 34% narrower (score>0 / 3x cut), sadness 22% / 48%, admiration 26% / 25%; the oracle gave 17 to 24%. WE coverage there: fear 48.8 / 64.0, sadness 78.0 / 51.4, admiration 80.8 / 83.8. At n = 1000 coverage is 84.6 / 89.4 (fear), 91.0 / 86.4 (sadness), 91.2 / 92.4 (admiration); the oracle was 91.6 to 95.2. In neutral (no gain available) pilot allocation is harmless: gain 0 to 1%, coverage 93.0 to 95.8%.
• Size-proportional allocation over the same two cells gains nothing (-3% to +0%) and covers 93.0 to 96.8% in all 16 cells: the split alone is worthless, all of the oracle's gain comes from allocating by S2, and that is exactly what a pilot cannot estimate when the coder-negative cell has one or two positives in the pilot (its S2 is then underestimated, the cell is starved to a handful of items, and the rare positives it hides are missed in some draws).
• Pilot totals: 934 / 929 against n = 1000 in fear, 375 / 410 against 400 (fear 3x cut, sadness 3x cut); others within 1.5%.
• (1) held on its letter, and misleading: pilot gain is 21 to 28% at n = 1000 in fear and admiration, but its coverage is 84.6 to 92.4%, so the gain is partly false precision. (2) held: proportional gains are -3% to 0%. (3) refuted in part: fear and sadness are below 93% as predicted, and so is admiration (80.8 / 83.8 at n = 400); neutral holds. (4) refuted: three totals are off by 2.5 to 7%.
• Reading: a negative result that closes a branch. 6ag's coder-cell advice (15 to 25% narrower at admiration-like prevalence) rests on knowing each cell's S2; estimated naively from a 30% pilot, the allocation starves the low-prevalence cell and coverage collapses to 49 to 92%. So 6ag's rule stands only with a pilot-free allocation: at 8% prevalence or more with an informative coder, either pool (valid, wider) or use a floor on each cell's allocation. The naive Neyman-on-pilot design must not be recommended.
• Limits: one corpus, one 1.5B model; 500 draws; one pilot share (30%) and one smoothing choice; the pilot items are also in the audit (a design with an independent pilot was not run); WE has no coverage theorem.
• Next: floor the coder-negative cell's allocation (for example at least 20% of n or a shrunken S2) and re-run; try an independent pilot; fold into the paper (done, one sentence, this run).

2026-10-29 · Milestone 6ag: when audit strata are the coder's own cells (coder-positive vs coder-negative), whose prevalences differ by a large factor, does stratifying finally pay? (planned before running)

• Why: 6ae and 6af's "Next". Corpus groups had prevalences within 1.5x of each other, so stratifying could not help. Cells defined by the coder's label differ by 10x or more when the coder is informative, which is the case where Neyman allocation should pay. experiments/h6ag_coder_strata.py reuses 6ae's simulation (oracle Neyman allocation, pooled-lambda PPI Wilson on n_eff, human normal, human Wilson, 500 draws, same seed) on fear, sadness, admiration and neutral with the cached Qwen2.5-1.5B codes. Designs: 1 pooled, 4 corpus groups, 2 coder cells, 8 (group x cell). Coders: score>0 and 3x cut. n = 400 and 1000.
• Predictions (before running): (1) WE half-width with 2 coder cells is at least 10% narrower than with 1 pooled stratum in admiration at n = 1000 (J 0.65, prevalence 8%). (2) In neutral (J 0.20) the gain is under 5%. (3) In fear and sadness (1 to 2%) the gain is positive but under 25%, and human normal coverage at n = 400 falls below 93% in at least one of the two coder-cell cells (the coder-positive cell holds few positives). (4) WE covers at least 93% in at least 90% of all cells.
• Refuted: (1) gain under 10%; (2) gain 5% or more; (3) gain 25% or more or negative, or both coverages 93% or higher; (4) fewer than 90% of cells at 93% or more.
• Limits known in advance: one corpus, one 1.5B model; 500 draws (+/- 2 points); oracle allocation (a pilot would estimate S2, 6r to 6aa); coder cells are not randomisation strata fixed before seeing the coder, but the coder is a deterministic function of the text so the design is valid; WE has no coverage theorem.

### Outcome (experiments/h6ag_coder_strata.py, about 3 min; experiments/h6ag_coder_strata.txt; 500 draws per cell, coverage +/- 2 points)

• Coder cells do separate prevalence by a large factor: fear (score>0) 13.8% in the 304 coder-positive items against 0.55% in the rest; sadness 35.0% vs 1.2%; admiration 29.1% vs 1.8%; neutral 31.3% vs 18.3%.
• WE half-width, 2 coder cells vs 1 pooled stratum, n = 1000 (score>0 / 3x cut): fear 0.477 vs 0.605 (21% narrower) / 0.480 vs 0.593 (19%); sadness 0.675 vs 0.793 (15%) / 0.605 vs 0.810 (25.3%); admiration 1.141 vs 1.415 (19%) / 1.132 vs 1.423 (20%); neutral 2.374 vs 2.383 (0.4%) / 2.361 vs 2.377 (0.7%). The 8-cell design (group x coder cell) adds 1 to 4% more in the rare categories and nothing elsewhere.
• WE coverage at n = 400 with 2 coder cells: fear 88.8 / 88.0, sadness 90.8 / 92.4, admiration 96.2 / 94.6, neutral 93.8 / 95.0; at n = 1000 fear 94.0 / 91.6, sadness 93.2 / 94.0. Human normal at 2 coder cells is no better (fear 88.6 / 90.0 at n = 400). Across all 64 cells WE covers at least 93% in 51 (80%); every miss is a rare category (fear, sadness) or the 4-group design in fear at n = 400.
• (1) held: admiration gain 19 to 20% at n = 1000. (2) held: neutral gain 0.4 to 0.7%. (3) refuted, narrowly: gains are 15 to 25% in fear and sadness (sadness 3x cut at n = 1000 is 25.3%), and coverage falls to 88 to 90.8% at n = 400 (that half held). (4) refuted: 80%, not 90%.
• Reading: this completes the audit-design picture of 6ae and 6af. Stratifying by corpus group does not pay because groups barely differ in prevalence; stratifying by the coder's own label does, by 15 to 25% of the half-width whenever the coder separates prevalence strongly (admiration, fear, sadness) and by nothing when it does not (neutral, J 0.20). The price is validity at a rare category: with about 300 coder-positive items and 4 to 5 positives per audit cell at n = 400, coverage drops to 88 to 91%, as 6ad found for corpus strata. So the rule is: at prevalence of about 8% or more with an informative coder, audit by coder cell; at 1 to 2%, either pool (valid, wider) or use coder cells only with n of about 1,000 or more, where coverage is 92 to 94%.
• Limits: one corpus, one 1.5B model and prompt each; oracle allocation (a pilot-estimated S2 would lose some of the gain); 500 draws; WE has no coverage theorem; coder cells are not a fixed design before the coder runs, though the coder is deterministic given the text; coder cells here were not tested with a 7B coder.
• Next: pilot-estimated allocation over coder cells (6r to 6aa's pilot, now with the cell split); fold 6ab to 6ag into the paper (done in section 6 in this run, one paragraph).

2026-10-29 · Milestone 6af: on common GoEmotions categories (admiration 8%, neutral 23%), where does stratifying the audit start to pay, and does 6ae's "pool the audit" advice survive? (planned before running)

• Why: 6ae's "Next". At 1 to 2% prevalence four strata bought nothing. experiments/llm_code_goemotions.py now also codes admiration and neutral (same Qwen2.5-1.5B-Instruct, one fixed zero-shot yes/no prompt each, written before any score was seen, no prompt search; cached in gollm_<cat>_*.tsv). experiments/h6af_common_category.py runs 6ae's design comparison (4, 3, 2, 1 strata; human normal, human Wilson, pooled-lambda PPI Wilson on n_eff; coders LLM score>0 and LLM 3x cut; n = 400 and 1000; 500 draws; oracle Neyman allocation).
• Predictions (before running): (1) with 20+ positives per audit group the normal interval is adequate: human normal covers at least 93% at n = 400 for every design. (2) WE covers at least 93% in every cell. (3) 4 strata is at most 5% narrower than 1 stratum in WE half-width in every cell (groups differ in prevalence by under 3x, so allocation gains little). (4) the coder's width gain over human Wilson is larger than in fear and sadness (6ab to 6ae: 3 to 9%): at least 10% in admiration at n = 1000, 1 stratum. (5) coder quality: Youden J of the 3x cut is below 0.5 for neutral (a poorly posed question for a 1.5B model).
• Refuted: (1) any cell under 93%; (2) any WE cell under 93%; (3) 4 strata narrower by more than 5% in a cell; (4) gain under 10% there; (5) J of 0.5 or more.
• Limits known in advance: one corpus, one 1.5B model; 500 draws (+/- 2 points); "neutral" is an annotation convention more than an emotion; oracle allocation.

### Outcome (experiments/llm_code_goemotions.py about 31 min per category on 2 threads each; experiments/h6af_common_category.py, experiments/h6af_common_category.txt; 500 draws per cell, coverage +/- 2 points)

• Coders (sens / spec): admiration (prevalence 8.1%) score>0 0.830 / 0.821, 3x cut 0.843 / 0.806 (J 0.65); neutral (23.3%) score>0 0.517 / 0.656, 3x cut 0.641 / 0.554 (J 0.20).
• Human normal coverage at n = 400, 4 / 3 / 2 / 1 strata: admiration 95.0 / 94.2 / 92.6 / 94.8 (score>0), neutral 96.2 / 95.4 / 93.8 / 95.8. WE covers 93.4 to 96.8% in all 32 cells. WE half-width at n = 1000, 4 vs 1 strata: admiration 1.410 vs 1.417, neutral 2.375 vs 2.383 points (4 strata about 0.5% narrower). WE against human Wilson at n = 1000, 1 stratum: admiration 1.417 / 1.425 against 1.566 (9.5% and 9.0% narrower), neutral 2.383 / 2.377 against 2.411 (1.2 to 1.4%).
• (1) refuted, narrowly: admiration at 2 strata covers 92.6% at n = 400 (both coders; the 93% line is inside the Monte Carlo error). (2) held (minimum 93.4%). (3) held: stratification changes the half-width by about 1%, i.e. it does not pay at 8 or 23% prevalence either. (4) refuted, narrowly: the gain is 9.0 to 9.5% in admiration, above fear and sadness (3 to 9%) but under 10%. (5) held: neutral J is 0.20.
• Reading: 6ae's advice generalises. With groups whose prevalences differ by under 1.5x (admiration 6.4 to 9.3%, neutral 18.7 to 27.8%), four strata give no usable precision, so "pool the audit" holds from 1% to 23% prevalence; stratification should only pay where group prevalences differ by a large factor (not tested). The coder's width gain tracks its informedness (J 0.65 gives about 9%; J 0.20 gives about 1%), consistent with the rectifier-variance account of 6ab. Where the normal interval is fine (20+ positives per group), WE is as valid and a little narrower.
• Limits: one corpus, one 1.5B model, one fixed prompt per category; "neutral" is an annotation convention; 500 draws; oracle allocation; the groups here happen to have similar prevalences, which is the reason stratifying cannot help, so this does not test a corpus with strongly unequal groups.
• Next: a corpus or category whose group prevalences differ by 5x or more (to find where stratification starts to pay); fold 6x to 6af into the paper's audit-design section.

2026-10-29 · Milestone 6ae: does merging the rarest GoEmotions groups into one audit stratum repair the n = 400 fear undercoverage that the estimator could not? (planned before running)

• Why: 6ad's "Next". Human Wilson itself covers only 91.8 to 94.2% in fear at n = 400, so the residual is the design (about 4 positives per audit group), not the estimator. experiments/h6ae_merge_strata.py reuses 6ad's corpus, cached Qwen2.5-1.5B codes, allocation rule (oracle Neyman on the merged strata), seed and 500 draws. Designs: 4 strata (6ad), 3 (the two groups with fewest corpus positives merged), 2 (the three rarest merged), 1 (no stratification). Rules per design: human normal, human Wilson, P Wilson on n_eff (6ad's WE). Coders: LLM score>0 and LLM 3x cut. n = 400 and 1000.
• Predictions (before running): (1) at n = 400 in fear, human normal coverage rises monotonically (within 2 points) as strata are merged, reaching at least 91% with 1 stratum. (2) WE at 1 stratum covers at least 93% in every n = 400 cell. (3) Merging costs under 10% in WE half-width between 4 strata and 1 stratum in every cell (oracle allocation gains little at 1 to 2% prevalence). (4) The coder's width gain over human Wilson shrinks with merging but stays non-negative in at least 6 of 8 n = 1000 and n = 400 cells at the 1-stratum design.
• Refuted: (1) coverage not rising, or under 91% at 1 stratum; (2) any n = 400 WE cell under 93% at 1 stratum; (3) a width increase of 10% or more; (4) WE wider than human Wilson in more than 2 of 8 cells at 1 stratum.
• Limits known in advance: one corpus, one 1.5B model and prompt; 500 draws (+/- 2 points); oracle allocation; n_eff has no coverage theorem; merged groups differ in the covariate the strata were meant to capture, so 1 stratum gives up whatever stratification buys in precision.

### Outcome (experiments/h6ae_merge_strata.py, under 1 min; experiments/h6ae_merge_strata.txt; 500 draws per cell, coverage +/- 2 points)

• n = 400 fear, human normal coverage at 4 / 3 / 2 / 1 strata: score>0 87.8 / 91.4 / 91.2 / 95.8; 3x cut 85.0 / 91.2 / 91.4 / 95.8. WE (P Wilson on n_eff): score>0 93.8 / 94.4 / 93.8 / 97.2; 3x cut 92.0 / 92.8 / 91.2 / 95.8. Sadness WE at n = 400: 94.6 to 95.8 at 1 stratum. WE half-width at n = 400 in fear: 1.007 (4 strata) to 1.026 (1 stratum) for score>0, 0.971 to 0.996 for the 3x cut; sadness 1.31 to 1.35 throughout. At n = 1000 WE covers 93.8 to 95.8% at every design.
• (1) held (fear human normal rises 85 to 88 -> 95.8, one 0.2-point dip inside Monte Carlo error). (2) held: WE at 1 stratum covers 94.6 to 97.2% at n = 400. (3) held, with room: merging costs 2 to 3% of the half-width in fear and under 1% in sadness. (4) held: WE is narrower than human Wilson in all 8 cells at 1 stratum (by 3 to 9%).
• Reading: at 1 to 2% prevalence the four strata buy nothing the audit can use. Oracle Neyman allocation over them moves the half-width by 1 to 3%, while the design costs about 4 positives per group and with them the normal interval's coverage (85 to 88% at 4 strata, 95.8% pooled). So for a rare category: audit one pooled random sample, use a pooled lambda and the Wilson-on-n_eff interval; that is cheaper in validity than any stratified variant tried in 6r to 6ad, and the coder's 3 to 9% width gain over human Wilson survives. The last row of 6x to 6ad's ranking (stratify, then repair) is replaced by "do not stratify at this prevalence".
• Limits: one corpus (GoEmotions, 77 fear positives in the test part), one 1.5B model and prompt; 500 draws; the coder's small gain was found, not tuned; the 4-strata row differs from 6ad's by up to 1 point because strata are ordered by positives here, which changes the random draws; stratification may still pay where prevalence differs by 10x between groups, not tested; WE has no coverage theorem.
• Next: the same on a corpus with a common category (prevalence 10 to 30%), to see where stratification starts to pay; fold 6x to 6ae into the paper's rare-category section.

2026-10-28 · Milestone 6ad: does a Wilson interval on the PPI estimate's effective sample size repair the n = 400 rare-category undercoverage that no PPI weight could? (planned before running)

• Why: 6ac's "Next". With a pooled lambda the point estimator is no worse than human-only, but fear still covers 88 to 89% at n = 400 because the normal interval is wrong for about 4 positives per audit group, and the plus-four floor over-covers (99 to 100%) at 1.4 to 1.7 times human-only's width. experiments/h6ad_wilson_ppi.py reuses 6ac's corpus, cached Qwen2.5-1.5B codes, allocation, audit simulation and seed (500 draws) and adds, on the same draws, intervals around the pooled-lambda estimate P: (WE) Wilson with n_eff = p(1 - p) / se^2, where p is the estimate clipped to [0.0005, 0.9995] and se is P's plug-in standard error (the PPI estimate is treated as a proportion measured on n_eff independent labels); (WH) the same Wilson interval for stratified human-only. Compared with the normal intervals for the same two estimators.
• Predictions (before running): (1) WE covers at least 93% in every cell at n = 400 (it inherits Wilson's repair of the rare-event interval). (2) WE's mean half-width at n = 400 is below the plus-four floor's 1.65 and 1.93 points (fear, sadness, score>0) in every cell, and within 25% of human-only's normal width. (3) WE at n = 1000 covers at least 93% and is narrower than WH in at least half of the cells (the coder still buys something once the normal approximation holds). (4) WE is not simply WH shifted: its coverage differs from WH's by under 3 points in at least half of the cells (the coder rarely moves the interval at 1 to 2% prevalence).
• Refuted: (1) any WE n = 400 cell under 93%; (2) a WE n = 400 half-width at or above the floor's, or more than 25% over human-only's normal; (3) coverage under 93% at n = 1000, or WE narrower than WH in fewer than half of the cells; (4) coverage differs by 3 points or more in at least half of the cells.
• Limits known in advance: one corpus, one 1.5B model and prompt; 500 draws (+/- 2 points); n_eff is a heuristic with no coverage theorem; lambda estimated from the same audit (ignored, as in 6ac); oracle allocation.

### Outcome (experiments/h6ad_wilson_ppi.py, under 2 min; experiments/h6ad_wilson_ppi.txt; 500 draws per cell, coverage +/- 2 points)

• n = 400 coverage, human normal / P normal / human Wilson / WE: fear score>0 88.8 / 89.2 / 93.6 / 92.4; fear 3x cut 88.0 / 88.2 / 94.2 / 92.2; sadness score>0 94.0 / 91.6 / 94.2 / 94.2; sadness 3x cut 92.6 / 92.8 / 96.0 / 93.0. WE mean half-width (points) at n = 400: 1.028 / 0.994 (fear), 1.303 / 1.336 (sadness), 3 to 8% narrower than human Wilson and 3 to 4% wider than human normal in fear. At n = 1000 WE covers 92.2 / 94.8 (fear) and 96.0 / 97.0 (sadness), 0.583 to 0.793 points wide.
• (1) refuted, narrowly. WE covers 92.4 and 92.2% in the two fear cells at n = 400 (sadness 94.2 and 93.0); the 93% line is inside the +/- 2 point Monte Carlo error but the prediction said at least 93%. (2) held: WE is 1.03 to 1.34 points against the floor's 1.65 to 1.93, within 4% of human normal in fear and narrower in sadness. (3) half refuted: narrower than the human Wilson interval in all 8 cells, but fear score>0 at n = 1000 covers 92.2% (human Wilson 91.8%, human normal 92.4%). (4) held: WE and human Wilson differ by under 3 points in 7 of 8 cells.
• Reading: the Wilson-on-n_eff interval is the cheap repair 6ac asked for. It lifts fear's n = 400 coverage from 88 to about 92 (the floor reached 100% at 1.7 times the width), keeps the coder's 3 to 8% width gain over human Wilson, and is no more variable than the human-only interval. It does not fully close fear's gap: the human Wilson interval itself covers 91.8 to 94.2% there, so what is left is the stratified design (about 4 positives per audit group) rather than the estimator, and a design-level fix (fewer strata, or pooling the rarest groups) is the open lever. The coder's contribution at 1 to 2% prevalence stays small, as 6ab found.
• Limits: one corpus, one 1.5B model and prompt; 500 draws; n_eff has no coverage theorem; lambda estimated on the audit it is applied to; oracle allocation.
• Next: collapse the rarest groups into one stratum for the audit and repeat; a 7B coder; fold the six rare-category findings (6x to 6ad) into the paper's rare-category section with this ranking: pooled lambda, Wilson on n_eff, no per-stratum tuning.

2026-10-28 · Milestone 6ac: does a pooled PPI weight, with or without a plus-four variance floor, repair 6ab's undercoverage of tuned stratified PPI at n = 400? (planned before running)

• Why: 6ab's "Next" and its refuted prediction (3): with the real LLM coder, tuned stratified PPI covered 85.2 to 89.0% (fear) and 90.0 to 90.2% (sadness) at n = 400, below human-only, because lambda and the residual variance are estimated per stratum on a handful of positives. experiments/h6ac_pooled_lambda.py reuses 6ab's corpus, split, cached Qwen2.5-1.5B codes, allocation and audit simulation (500 draws, same seed) and adds two estimators on the same audit draws: (P) one lambda per audit (cov(y, f) / var(f) pooled over all audited items, clipped to [0, 1]) used in every stratum; (PF) P plus a variance floor: each stratum's residual variance is at least the plus-four variance p~(1 - p~) of y in that stratum's audit, p~ = (k + 2) / (n_s + 4) (deliberately conservative). Compared with stratified human-only and 6ab's per-stratum tuned PPI. Coders: LLM score>0 and LLM 3x cut (NB omitted; it was not the problem). n = 400 and 1000.
• Predictions (before running): (1) P covers at least 91% in every cell at n = 400 (pooling stabilises lambda). (2) PF covers at least 93% in every cell at n = 400. (3) PF's mean half-width stays below human-only's at n = 1000 in every cell (the coder still buys a little). (4) PF's n = 400 half-width is within 15% of human-only's in at least half of the cells, i.e. the repair costs most of the PPI gain.
• Refuted: (1) any P cell under 91%; (2) any PF cell under 93%; (3) PF at or above human-only's width at n = 1000 in any cell; (4) PF within 15% in fewer than half of the cells.
• Limits known in advance: one corpus, one 1.5B model, one prompt; 500 draws (+/- 2 points); the floor is a heuristic with no coverage theorem; allocation uses the oracle S2 as in 6ab.

### Outcome (experiments/h6ac_pooled_lambda.py, about 5 min; experiments/h6ac_pooled_lambda.txt; 500 draws per cell, coverage +/- 2 points)

• n = 400 coverage, human-only / per-stratum tuned / P / PF: fear score>0 88.8 / 89.0 / 89.2 / 100.0; fear 3x cut 88.0 / 85.2 / 88.2 / 100.0; sadness score>0 94.0 / 90.0 / 91.6 / 99.4; sadness 3x cut 92.6 / 90.2 / 92.8 / 99.8. Mean half-width (points) at n = 400, same order: fear score>0 0.994 / 0.921 / 0.949 / 1.651; sadness score>0 1.397 / 1.244 / 1.259 / 1.933. At n = 1000, PF is 0.784 to 0.990 points against 0.599 to 0.842 for human-only, covering 97.6 to 99.0%.
• (1) refuted. P is 89.2 and 88.2% in the two fear cells at n = 400. (2) held: PF covers 99.4 to 100.0% at n = 400. (3) refuted: PF is 31 to 32% wider than human-only at n = 1000 (for example 0.788 vs 0.601, fear score>0). (4) refuted: PF is 1.4 to 1.7 times human-only's width at n = 400, never within 15%.
• Reading: pooling lambda removes the extra loss that per-stratum tuning added (fear 3x cut 85.2 to 88.2%, sadness score>0 90.0 to 91.6%, and at n = 1000 sadness score>0 90.8 to 94.6%), and P is then no worse than human-only, with a 3 to 10% narrower interval. But fear's 88% at n = 400 is human-only's own shortfall (about 4 positives per audit group), so no PPI weight can repair it. The plus-four floor does repair it, at the price of an interval wider than human-only, so it buys validity by giving the coder's gain away and then some. Net for a rare category with this coder: use a pooled lambda, not per-stratum; below about 1,000 items the shortfall is the rare-event normal interval, and a Wilson-type interval on the PPI estimate (not the floor tried here) is still untried.
• Limits: one corpus, one 1.5B model and prompt; 500 draws; the floor is a heuristic whose over-coverage shows it is too blunt; oracle allocation; no pooled-lambda variance correction for lambda being estimated from the same audit (it has an influence-function term the plug-in ignores).
• Next: an influence-function variance for pooled lambda and a Wilson-type interval for the PPI estimate; a 7B coder.

2026-10-28 · Milestone 6ab: with a real zero-shot LLM coder on GoEmotions fear and sadness, does tuned stratified PPI beat human-only, and by how much? (planned before running)

• Why: 6z and 6aa's "Next". Every GoEmotions result so far used a naive Bayes coder whose rectifier variance is about twice the prevalence. experiments/llm_code_goemotions.py codes the whole test part (6,683 items, the 6r to 6aa split) with Qwen2.5-1.5B-Instruct, one zero-shot yes/no prompt per category (fixed before any score was seen, no prompt search), cached in gollm_<cat>_*.tsv. experiments/h6ab_llm_goemotions.py compares three coders on the same items: NB (6r), LLM with the fixed cut score > 0, and LLM with a label-free cut so coder-positives are 3x the train prevalence. For each: sensitivity, specificity, the oracle Neyman audit size at 1.0 and 0.5 point for tuned stratified PPI against human-only (lambda = 0), then fixed-size audits (n = 400, 1000; 500 draws) for coverage and width. Oracle S2 allocation in the simulations (an upper bound on what a pilot achieves).
• Predictions (before running): (1) the LLM 3x-cut coder has higher Youden J (sens + spec - 1) than NB in both categories. (2) its tuned oracle audit size is at least 10% below human-only at the 0.5-point target in both categories. (3) tuned PPI coverage at n = 400 and 1000 is at least 91% in every cell. (4) the saving stays under 3x (H3's bar is a 3x smaller audit), i.e. a 1.5B zero-shot coder does not meet H3 at a rare category.
• Refuted: (1) J not higher in either category; (2) saving under 10% in a category; (3) any cell under 91%; (4) a saving of 3x or more.
• Limits known in advance: one corpus and one 1.5B model, one prompt; oracle allocation and oracle lambda in the sizing table; 500 draws (coverage +/- 2 points); nominal normal intervals.

### Outcome (experiments/llm_code_goemotions.py, about 45 min for 2 x 6,683 items on 4 CPU cores; experiments/h6ab_llm_goemotions.py, experiments/h6ab_llm_goemotions.txt; 500 draws per cell)

• Coders (sens / spec): fear NB 0.247 / 0.969, LLM score>0 0.545 / 0.960, LLM 3x cut 0.481 / 0.980; sadness NB 0.396 / 0.922, LLM score>0 0.494 / 0.978, LLM 3x cut 0.682 / 0.941. Oracle audit size at 0.5 point, human-only to tuned: fear 1,368 to 1,347 (NB), 1,291 (LLM >0), 1,253 (LLM 3x); sadness 2,242 to 2,197, 1,971, 2,033. Best saving is 12.1% (sadness, LLM score>0), 1.14x smaller.
• (1) held. Youden J of the LLM 3x-cut coder is 0.46 (fear) and 0.62 (sadness) against 0.22 and 0.32 for NB.
• (2) refuted. At 0.5 point the 3x-cut coder saves 8.4% (fear) and 9.3% (sadness), under 10%. (The score>0 coder saves 5.6% and 12.1%.)
• (3) refuted, and the failure is not the PPI's. Coverage under 91% in several cells: fear human-only at n = 400 covers only 86 to 88% (77 positives in the corpus, about 4 in an audit group), tuned PPI 85.2 to 89.0% at n = 400; sadness tuned PPI with the score>0 coder 90.0% and 90.8% at n = 400 and 1,000 (human-only 94.0%). At n = 1,000 tuned PPI is 92.0 to 94.8% except that cell.
• (4) held. No saving reaches 3x; the largest is 1.14x.
• Reading: a coder with twice the informedness of NB moves the audit size by under 13%, because at 1 to 2% prevalence the rectifier variance is dominated by the few positives the coder misses or falsely flags, so H3's 3x bar is far from met by a 1.5B zero-shot model at a rare category. The width gain is a few percent, and tuned PPI sometimes undercovers more than human-only (lambda estimated on a handful of positives), so the coder buys little and costs some validity at n = 400.
• Limits: one corpus, one 1.5B model and one prompt (no few-shot, no larger model); oracle lambda and allocation in the sizing table; 500 draws (+/- 2 points); nominal normal intervals; coder cuts chosen label-free but "score>0" is the model's own boundary.
• Next: a stronger coder (7B or an API model with the owner's key; Aya for the Nigerian corpus), or few-shot prompts on the same cache design; the audit-size rule with a Wilson-type interval for the tuned estimator.

2026-10-28 · Milestone 6aa: does freezing the PPI weight from a pilot stage make the stratified betting confidence sequence cheaper than 6z's lambda = 1, and does it beat the pooled human-only sequence? (planned before running)

• Why: 6z's "Next" (a predictable lambda). 6z's CS used lambda = 1 and so got no variance reduction. experiments/h6aa_pilot_lambda.py: 6z's setup (GoEmotions fear and sadness, four groups, targets 1.0 and 0.5 point, NB coder at about 3x prevalence, 6v's adaptive allocation, start 400, batches of 100). In each stratum the first pilot items (about 100) fix lambda = clip(cov(y, f) / var(f), 0, 1) once and for all; the pilot rectifier sum enters exactly; the betting CS (level 0.05/4 per stratum, union bound) covers the mean of the unseen rest of the stratum for (y - lambda f + lambda) / (1 + lambda), sampled without replacement. Lambda depends only on the pilot items, so it is fixed before the CS sees any item and validity is unaffected. 300 draws per cell. (A 3-draw smoke run, run before writing these predictions, printed stopped n of 5,033 / 5,600 / 5,100 / 5,933, so predictions 2 and 3 are informed by 3 draws and are weak tests; coverage with 3 draws means nothing.)
• Predictions (before running): (1) coverage at least 94% in all four cells (valid by construction, conservative union). (2) mean stopped n is below 6z's lambda = 1 sequence (5,484 / 6,035 / 5,951 / 6,394 in the order fear 1.0, fear 0.5, sadness 1.0, sadness 0.5) in at least three of four cells. (3) it still needs more items than 6y's pooled human-only CS in at least three of four cells, i.e. the coder still does not pay at this prevalence.
• Refuted: (1) any cell under 94%; (2) fewer than three cells cheaper than 6z; (3) the stratified sequence cheaper than 6y's pooled CS in two or more cells.
• Limits known in advance: one corpus and coder; 300 draws (coverage +/- 2.5 points); pilot lambda noisy at about 100 items per stratum with a handful of positives; union bound; the grid is a window of +/- 0.08 around the pilot estimate in the scaled rectifier (a truth outside it would be a validity leak, checked via coverage).

### Outcome (experiments/h6aa_pilot_lambda.py, about 10 min; experiments/h6aa_pilot_lambda.txt; 300 draws per cell, coverage +/- 2.5 points)

• Coverage 100.0% in all four cells. Mean stopped n: fear 1.0 pt 4,543 (11.41x oracle), fear 0.5 pt 5,387 (4.00x), sadness 1.0 pt 4,932 (6.73x), sadness 0.5 pt 5,839 (2.66x), against 6z's lambda = 1: 5,484 / 6,035 / 5,951 / 6,394. No audit ran to the full corpus.
• (1) held. (2) held, in all four cells (saving 555 to 1,019 items, 8 to 17%). (3) held: 6y's pooled human-only CS needs 1,075 / 3,437 / 2,692 / 4,984, cheaper in every cell, so the stratified coder-assisted sequence is still dominated.
• Reading: freezing lambda from a 100-item pilot recovers some of the coder's value but not enough; at a rare category the audit still covers 68 to 87% of the corpus. The union bound and the 4-fold stratification, not lambda alone, make the sequence expensive. The recommendation from 6z stands (label-free plan, or pooled human-only Wilson or CS).
• Limits: one corpus and coder; 300 draws; coverage 100% everywhere means the sequence is very conservative, not that its nominal level is tight; the pilot lambda was not tuned; no pooled-then-poststratified sequence tried.
• Next: H3 with a sharper coder (the real Qwen coders on GoEmotions), where the rectifier variance is smaller and PPI should pay.

2026-10-28 · Milestone 6z: does a stratified, coder-assisted betting confidence sequence give a valid sequential audit at a rare category, and at what cost? (planned before running)

• Why: 6y's "Next". 6y showed the pooled human-only betting CS is valid at the stop (96.6 to 98.0%) at 2.2 to 3.5 times the fixed-n size, and that the 6t to 6w collapse belongs to the stratified tuned-PPI variance estimate. experiments/h6z_strat_cs.py: 6r's setup (GoEmotions groups, fear and sadness, targets 1.0 and 0.5 point, NB coder at about 3x prevalence), the same adaptive Neyman-style allocation as 6v (start 400, batches of 100). Rule A = tuned stratified PPI plug-in at z = 1.96 (repeat of 6t/6v). Rule C = per stratum a hedged betting CS (Waudby-Smith and Ramdas) for the mean of the PPI rectifier (y - f + 1)/2 sampled without replacement, at level 0.05/4, combined with the known stratum means of the coder and the stratum weights; valid by a union bound over strata, and per-stratum validity holds under adaptive allocation because each stratum is read in a fixed random order. Rectifier weight lambda = 1 (untuned, so no estimated lambda enters). 300 draws per cell, both rules on the same audit stream, each stopping at its own time.
• Predictions (before running): (1) C covers at least 94% in all four cells (valid by construction; the union bound makes it conservative, so I expect 97 to 100%). (2) A repeats 6v: under 90% at 0.5 point for both categories. (3) C's mean stopped n is more than 3 times the oracle n in every cell (the rectifier is not variance-reduced by tuned lambda, the CS pays about 2 to 3x over fixed-n in 6y, and the union costs another factor), and more than the pooled human-only CS of 6y needs in absolute terms in at least one cell, i.e. the coder does not pay for itself inside a betting CS at this rare prevalence.
• Refuted: (1) any C cell under 94%; (2) A at or above 90% at 0.5 point in either category; (3) C under 3x oracle in some cell, or C cheaper in absolute n than 6y's CS in all four cells.
• Limits known in advance: one corpus and coder; 300 draws (coverage +/- 2.5 points); lambda not tuned and betting constants not tuned; the union bound is crude; the audit may run to the full corpus.

### Outcome (experiments/h6z_strat_cs.py, about 25 min; experiments/h6z_strat_cs.txt; 300 draws per cell, coverage +/- 2.5 points)

• Plug-in tuned stratified PPI (A) coverage at the stop: fear 89.7 (1.0 pt) / 65.3 (0.5), sadness 83.3 / 67.7. Betting CS (C): 100.0 / 99.3 / 100.0 / 100.0. Mean stopped n: A 460, 905, 660, 1,822; C 5,484, 6,035, 5,951, 6,394 of a 6,683-item corpus (0.3% of sadness 0.5-point audits ran to the whole corpus). C as a multiple of the oracle n: 13.78, 4.48, 8.12, 2.91.
• (1) held. C covers 99.3 to 100.0%, far above 94% (the union bound over four strata is conservative).
• (2) held. A covers 65.3 and 67.7% at 0.5 point, repeating 6t to 6w.
• (3) refuted by one cell, held in spirit. C is above 3x oracle in three cells but 2.91x in sadness at 0.5 point. The second half held: C needs more items than 6y's pooled human-only CS in every cell (for example sadness 0.5 point 6,394 vs 4,984; fear 0.5 point 6,035 vs 3,437).
• Reading: a valid stratified coder-assisted sequential audit exists, but at this prevalence it is not worth running: it audits 80 to 96% of the corpus. The rectifier (y - f) at an NB coder that over-flags by 3x has variance of about twice the prevalence, and the untuned lambda = 1 PPI gives no variance reduction over the human-only labels, so the coder costs us (union over four strata, a two-fold range on the rescaled rectifier) rather than helps. The practical ranking at a rare category is now: label-free planned size (6x, 1.9 to 2.8x oracle, covers 91.8 to 94.4%), pooled human-only Wilson at the stop (6y, 1.2 to 1.5x fixed-n, empirically valid), pooled betting CS (6y, 2.2 to 3.5x, guaranteed); stratified PPI CS is dominated.
• Limits: one corpus and coder; 300 draws; lambda untuned (a CS for the tuned estimator would need a predictable lambda sequence, not tried); union bound (not a joint betting process); the half-width at stop for C undershoots the target slightly because it is checked every 100 items.
• Next: a predictable-lambda variant (lambda_t from past strata data) and a pooled-then-poststratified CS; or move on to H3, a sharper coder, where PPI should help.

2026-10-27 · Milestone 6y: does an always-valid confidence sequence repair the rare-category sequential audit? (planned before running)

• Why: the paper's limit 5 ("an always-valid interval was not compared") and 6w's close ("a valid sequential audit needs a bound that does not depend on the stopped normal approximation"). experiments/h6y_confseq.py: GoEmotions, fear and sadness, one pooled corpus (test 70% of the seeded split), uniform random audit order without replacement, human labels only (no coder, no strata, so this isolates the stopping problem; the stratified/PPI version is not tried). Looks every 100 items from 400, stop when the interval half-width is at most the target (0.5 and 1.0 point), cap 20,000. Three intervals computed on the same audit stream: (N) plug-in normal with finite-population correction (the rule 6t to 6w showed failing), (W) Wilson, and (C) a hedged betting confidence sequence for sampling without replacement (Waudby-Smith and Ramdas 2020/2024; grid of candidate means, predictable bets with clip 0.5, alpha = 0.05, running intersection). 500 draws per cell.
• Predictions (before running): (1) coverage at the stop of (C) is at least 94% in all four cells (it is valid by construction, so I expect 98 to 100%). (2) (N) covers under 90% at the 0.5-point target for both categories, as in 6t. (3) (W) at the stop also undercovers at 0.5 point (under 93%). (4) (C)'s mean stopped n is at most 4 times the fixed-n Wald size with finite-population correction, and more than 1.3 times it.
• Refuted: (1) any (C) cell under 94%; (2) (N) at or above 90% at 0.5 point; (3) (W) at or above 93% in both 0.5-point cells; (4) ratio outside 1.3 to 4 in any cell. A ratio above 4 would mean validity costs too much to use.
• Limits known in advance: one corpus, human-only (no PPI control variate: a sequential PPI confidence sequence is the next step), the grid resolution (0.0001 up to 12%) caps the upper end, betting constants not tuned.

### Outcome (experiments/h6y_confseq.py, 2.5 min; experiments/h6y_confseq.txt; 500 draws per cell, coverage +/- 2 points)

• Coverage at the stop, normal+fpc / Wilson / betting CS: fear 1.0 pt 95.0 / 97.6 / 98.0; fear 0.5 pt 90.6 / 95.2 / 96.6; sadness 1.0 pt 90.4 / 96.8 / 97.6; sadness 0.5 pt 93.0 / 99.6 / 96.8. Mean stopped n as a multiple of the fixed-n Wald size: normal 0.99 to 1.17, Wilson 1.21 to 1.54, CS 2.19 to 3.52. No audit reached the cap.
• (1) held. CS covers 96.6 to 98.0% in all four cells.
• (2) refuted. Normal+fpc at 0.5 point covers 90.6% (fear) and 93.0% (sadness), not under 90%. It does undercover at 1.0 point for sadness (90.4%) and at fear 0.5 point, but mildly.
• (3) refuted. Wilson at the stop covers 95.2 to 99.6%, no undercoverage.
• (4) held. CS costs 2.19 to 3.52 times the fixed-n size, inside 1.3 to 4.
• Reading: with human labels only and one pooled stratum, stopping on a plug-in interval is far less damaging than in 6t to 6w (there 64 to 72% at 0.5 point). Wilson, whose width does not collapse when few positives are seen, is valid at the stop in these four cells, and cheaper than the CS (1.2 to 1.5x vs 2.2 to 3.5x). The betting CS is the only interval here with a guarantee for all stopping times, and it costs about 2 to 3 times the fixed-n size. So the collapse in 6t to 6w is specific to the stratified tuned-PPI estimator's variance estimate (per-stratum counts of a few positives, tuned lambda), not to stopping at a rare category in general. This narrows the earlier advice: a stratified or PPI audit needs a stratum-safe interval, and the pooled human-only Wilson stop is a usable baseline.
• Limits: one corpus, human-only (no coder or PPI control variate), Wilson's validity here is empirical (500 draws, and it has no sequential guarantee), CS constants untuned, grid caps the upper end at 12%. Not tested: a sequential PPI confidence sequence and the stratified version.
• Next: a stratified, coder-assisted confidence sequence (betting on the PPI residual per stratum), and Wilson-type per-stratum intervals inside the stratified rule.

2026-10-27 · Milestone 6x: a label-free audit-size rule for the rare category (planned before running)

• Why: 6w closed with "a valid sequential audit at a rare category needs a stopping rule that uses no audit labels; none is given". The simplest such rule is a fixed size planned from what the corpus already holds without human labels: the coder's own prevalence per group. experiments/h6x_label_free.py: 6r's setup (GoEmotions groups, fear and sadness, targets 1.0 and 0.5 point, tuned stratified PPI, 500 draws). Per group g the proxy variance is S2_g = c * pf_g (1 - pf_g), pf_g = the coder's positive share over the whole group (no human label used); n = neyman_total(sizes, S2, target), Neyman allocation, one fresh audit. c = 1 and c = 2. The coder is thresholded at about 3x the train prevalence, so pf is about 3x the true prevalence and the proxy should over-plan.
• Predictions (before running): (1) at c = 1 coverage is at least 91% in all four cells (the size is independent of the audit labels, and fixed-n audits covered 91 to 94% in 6t). (2) mean planned n is 1.5 to 4 times the oracle n at c = 1 (proxy variance is about 3x the residual variance). (3) c = 2 costs about twice c = 1 and covers no worse (within 2 points).
• Refuted: (1) any c = 1 cell under 91%; (2) ratio outside 1.5 to 4 in any cell; (3) c = 2 more than 2 points below c = 1 in some cell, or cost ratio outside 1.7 to 2.3.
• Limits known in advance: one corpus and coder family, 500 draws, the proxy is a fixed formula I chose, not tuned; a rule that over-plans by 3x may not be useful even if valid.

### Outcome (experiments/h6x_label_free.py, under 2 min; experiments/h6x_label_free.txt; 500 draws per cell, coverage +/- 2 points)

• c = 1: planned n is 2.58x (fear 1.0 pt), 2.08x (fear 0.5), 2.79x (sadness 1.0), 1.94x (sadness 0.5) the oracle n; coverage 91.8, 92.8, 93.6, 94.4%. c = 2: 4.46x, 2.92x, 4.27x, 2.36x; coverage 92.6, 95.2, 94.6, 95.0%. No audit exceeded 1.25 times its target half-width (share 0.000 everywhere).
• (1) held. Lowest c = 1 cell is 91.8% (fear, 1.0 point), the same 1 to 2 point normal-interval shortfall that fixed-n audits show at about 1,000 items (6t).
• (2) held. All four ratios are inside 1.5 to 4 (1.94 to 2.79).
• (3) refuted on cost, held on coverage. c = 2 covers at least as well as c = 1 in every cell, but costs 1.73x, 1.40x, 1.53x and 1.22x the c = 1 audit, outside 1.7 to 2.3 in three cells: the finite-population term in neyman_total makes n grow less than linearly in c when the audit is a large share of the corpus (my prediction ignored it).
• Reading: a size planned from the coder's positive share alone is valid at a rare category (91.8 to 94.4% at c = 1) because no audit label enters the size, which is what 6t to 6w showed a sequential rule cannot offer; the price is 1.9 to 2.8x the oracle audit, against 1.5 to 3.8x for the Wilson-upper pilot (6s) that also needs a labelled pilot. So at a rare category the supported advice is now a label-free plan or a fixed audit; no sequential rule tried is valid.
• Limits: one corpus and one coder (NB thresholded at about 3x prevalence, so the proxy over-plans by design; a sharper coder would over-plan less or may under-plan, untested), 500 draws, c not tuned, nominal 95% intervals.

2026-10-27 · Milestone 6w: is the sequential rare-category audit's failure an understated standard error at the stop? (planned before running)

• Why: 6v's reading (plug-in se too small at the stop) was not tested. experiments/h6w_stop_z.py reuses 6v's rule A (z = 1.96, same corpus, coder, strata, start, batches, cap, seed, 500 draws) and records, per stopped audit, the standardised error t = (estimate - truth) / reported se, and the same t for a fixed-n audit of that audit's stopped size drawn independently (same estimator, no stopping). If stopping is the cause, sd(t) should be near 1 for fixed n and well above 1 at the stop.
• Predictions (before running): (1) fixed-n sd(t) between 0.9 and 1.15 in all four cells. (2) stopped sd(t) is at least 1.3 in the two 0.5-point cells and exceeds fixed-n sd(t) in all four. (3) among stopped audits, the correlation between reported se and |error| is lower than for fixed n.
• Refuted: (1) any fixed-n sd(t) outside 0.9 to 1.15; (2) stopped sd(t) under 1.3 in a 0.5-point cell or not above fixed in some cell; (3) correlation not lower.
• Limits known in advance: one corpus and coder family, 500 draws (sd(t) +/- about 0.06), fixed-n comparator draws one audit per stopped size.

### Outcome (experiments/h6w_stop_z.py, under 1 min; experiments/h6w_stop_z.txt; 500 draws per cell)

• t = (estimate - truth) / reported se. Stopped: sd t 1.08 (fear 1.0 pt), 1.44 (fear 0.5), 1.18 (sadness 1.0), 1.58 (sadness 0.5); mean t -0.56, -1.36, -0.79, -1.13. Fixed n at the same sizes: sd t 1.21, 1.17, 1.15, 1.14; mean t -0.35, -0.34, -0.20, -0.18.
• (1) refuted. Fixed-n sd t is 1.14 to 1.21, above the 0.9 to 1.15 bar in three cells (the fixed-n normal interval itself runs a little narrow at 1% prevalence, consistent with its 91 to 94% coverage).
• (2) half held. Stopped sd t is above 1.3 in both 0.5-point cells (1.44, 1.58), but at fear 1.0 point it is below the fixed-n value (1.08 vs 1.21).
• (3) held. Corr(se, |error|) among stopped audits is -0.67, -0.52, -0.52, -0.35 against -0.06 to +0.33 at fixed n: audits that stop have the small se because their estimate is low, and the error is large exactly when se is small.
• Reading: the mechanism is mostly a downward bias at the stop (mean t -1.4 and -1.1 at 0.5 point), not only a spread understatement: stopping selects audits that saw few positives, whose estimate and se are both low. This matches the 6u reading and is now measured. A wider z cannot fix a shifted centre (6v). A valid sequential audit at a rare category would need a bound that uses no audit labels in the stopping decision.
• Limits: as predicted; mean t is an average over 500 draws (se of mean t about 0.05).

2026-10-27 · Milestone 6v: does an optional-stopping correction (a wider critical z at the stop) repair the rare-category sequential audit? (planned before running)

• Why: 6u's open item. experiments/h6v_alpha_spend.py: 6t's setup (GoEmotions groups, fear and sadness, targets 1.0 and 0.5 point, start 400, batches of 100, cap 6,000, 500 draws, seed as 6t). Rule A = plug-in at z = 1.96 (repeat of 6t). Rule E stops and reports at z = 2.58. Rule F uses the Bonferroni z over the at most 57 looks (z = 3.33 for a 5% total level). The estimator is the tuned stratified PPI throughout.
• Predictions (before running): (1) F covers at least 93% in all four cells (a union bound over looks is valid whatever the stopping time). (2) E covers less than F in every cell. (3) F's mean n is 1.8 to 3.2 times A's (z ratio squared is 2.9).
• Refuted: (1) any F cell under 93%; (2) E at or above F in some cell; (3) ratio outside the range in any cell.
• Limits known in advance: one corpus and coder family, 500 draws, the union bound is only valid if the interval is valid at each fixed look, which 6t's fixed-n check supports at about 1,000 items and above but not at n = 457.

### Outcome (experiments/h6v_alpha_spend.py, about 4 min; experiments/h6v_alpha_spend.txt)

• (1) refuted, badly. F covers fear 89.2% (1.0 pt) and 63.6% (0.5), sadness 92.6% and 79.2%. A repeats 6t exactly (89.8 / 63.8 / 86.2 / 72.0%).
• (2) refuted by one cell. E covers 91.6, 61.6, 88.2, 77.4%; E is below F in three cells and above it in fear at 1.0 point (91.6 vs 89.2).
• (3) held except one cell. F/A mean n: fear 1.65x and 1.90x, sadness 2.24x and 1.79x (bar 1.8 to 3.2; fear 1.0 point misses at 1.65x).
• Reading: a z of 3.33 should cover whatever the stopping time if the interval were valid at every look; it does not, and fear at 0.5 point stays near 64% at any z tried. So the failure is not mainly multiple looks. At the stop the plug-in standard error is too small relative to the true error (few positives seen, coder agreement high), and widening z by 70% does not offset it. This reading is not tested directly; a test would compare the estimate error with the reported se among stopped audits.
• Limits: as above; z not tuned beyond two values.
• Next: if the sequential audit at a rare category is revisited, use a variance bound that uses no audit labels, or a fixed-n audit at the oracle size; the five-plus rules tried (6t, 6u, 6v) are all invalid or inefficient. Close this branch; return to the paper and the 7B coder.

2026-10-27 · Milestone 6u: stopping the rare-category sequential audit on a Wilson-upper-bound half-width (planned before running)

• Why: 6t's open item. Plug-in stopping undercovers at 1 to 2% prevalence (optional stopping on a low variance estimate); a positives guard made it worse. experiments/h6u_wilson_stop.py: same corpus, coder, strata, start, batches, cap and 500 draws as 6t. Rule A = 6t plain plug-in. Rule D stops when the half-width computed from a human-only variance with per-group 95% Wilson-upper prevalence (capped 0.5) is at most the target; the reported interval is still the tuned stratified PPI one.
• Predictions (before running): (1) D covers at least 92% in all four cells. (2) D's mean n is 1.5 to 4 times the oracle n (it is conservative, as the 6s pilot rule was). (3) D hits the 6,000 cap in over 10% of fear audits at the 0.5-point target.
• Refuted: (1) any D cell under 92%; (2) mean n outside that range in any cell; (3) cap share 10% or less.
• Limits known in advance: one corpus and coder family, 500 draws, human-only bound ignores lambda, interval still normal-approximation.

### Outcome (experiments/h6u_wilson_stop.py, 3.4 min; experiments/h6u_wilson_stop.txt; 500 draws per cell, coverage +/- 2 points)

• (1) refuted, badly. Rule D covers fear 51.4% (1.0 point) and 63.6% (0.5), sadness 68.4% and 74.4%. Rule A (plug-in, repeat of 6t) covers 89.8 / 63.8 / 86.2 / 72.0%.
• (2) refuted. D's mean n against the oracle: fear 2,175 vs 398 (5.5x) and 4,310 vs 1,347 (3.2x); sadness 2,005 vs 733 (2.7x) and 4,133 vs 2,197 (1.9x). It audits far past the target (mean half-width 0.20 to 0.51 point against 0.5 to 1.0) and still undercovers.
• (3) refuted. No D audit reached the 6,000 cap in any cell (0.000).
• Reading: stopping on a conservative variance bound does not repair the sequential audit. Unlike a pilot, the bound is computed from the same human labels the estimate uses, so the stopping time still depends on the positives seen so far: audits that have met the bound are the ones that saw few positives early, and their estimate is biased low. This mechanism is my reading, not tested here; a test would stop on a bound that uses no audit labels (for example, size only) and check coverage. The fixed-n audit of 6t covers 94% at the same sizes, so the loss is from the stopping, not the estimator.
• Limits: one corpus and coder family, 500 draws, human-only bound, normal-approximation final interval.
• Next: a confidence sequence (or alpha-spending) for the stratified PPI estimate; keep the fixed-n coverage check as an experiment file; the 7B coder. Closing the branch for now: of the five stopping rules tried at a rare category (plug-in, two positive guards, Wilson-upper), none is valid and efficient; the supported advice is a Wilson-upper pilot (6s) or a fixed-n audit at the oracle size.

2026-10-27 · Milestone 6t: a sequential audit at a rare category, with and without a minimum-positives stopping guard (planned before running)

• Why: 6s's open item. Its pilot rules either undercover (plug-in) or over-plan 1.5 to 3.8x (Wilson upper). The 6c sequential audit needs no pilot variance, but at 1 to 2% prevalence its plug-in half-width can look small, or be exactly zero, after a few hundred items that hold almost no positives, so it may stop too early. experiments/h6t_rare_sequential.py: same GoEmotions subreddit-group corpus, coder, split and strata as 6r/6s; fear and sadness; half-width targets 1.0 and 0.5 point; tuned stratified PPI (lambda from the audit); start 400 items allocated by stratum size, batches of 100 split by the Neyman shares of the current plug-in variances (floor 1e-4), cap 6,000; 500 draws per cell. Rules: (A) stop when the half-width is at most the target; (B) as A and also at least 20 human-coded positives in the audit; (C) as A and at least 40. Oracle n from neyman_total with whole-corpus S2 as reference.
• Predictions (before running): (1) A undercovers fear at the 0.5-point target (coverage under 92%) and stops with fewer than 20 positives in more than 10% of audits. (2) B covers at least 92% for both categories at both targets. (3) B's mean stopped n is within 0.8 to 1.3 times the oracle n for sadness and 0.8 to 1.6 times for fear (against 1.5 to 3.8x for the Wilson-upper pilot rule of 6s). (4) C is no better than B on coverage (within 2 points) and costs more audits.
• Refuted: (1) fear coverage at 0.5 point for A is 92% or more, or under 10% of A audits stop with fewer than 20 positives; (2) any B cell under 92%; (3) mean n outside those ranges; (4) C more than 2 points above B in any cell, or not costlier.
• Limits known in advance: one corpus and coder family, 500 draws (coverage +/- 2 points), no optional-stopping correction, the final interval is the normal-approximation one, the 20 and 40 guards are my choices and not tuned.

### Outcome (experiments/h6t_rare_sequential.py, about 3.5 min; experiments/h6t_rare_sequential.txt; 500 draws per cell, coverage +/- 2 points)

• (1) held. Rule A (stop at the plug-in half-width) covers fear 89.8% at 1.0 point and 63.8% at 0.5 point; sadness 86.2% and 72.0%. At fear, 100% (1.0) and 89% (0.5) of A audits stop with fewer than 20 human positives. Mean stopped n is near the oracle (fear 457 vs 398, 911 vs 1,347; sadness 669 vs 733, 1,874 vs 2,197): the rule does not stop absurdly early, it stops on low variance estimates.
• (2) refuted, badly. The guard makes coverage worse, not better, for fear: B covers 47.2% and 47.6% (1.0 / 0.5 point), C 40.6% at both, and they audit 1,700 to 3,450 items (up to half the corpus) to reach 20 or 40 positives, ending with half-widths of 0.19 to 0.31 point against the 0.5 to 1.0 targets. Sadness: B 85.2 / 72.0%, C 76.0 / 75.6%.
• (3) refuted. B's mean n against oracle: fear 4.4x at 1.0 point, 1.31x at 0.5; sadness 1.25x and 0.86x (the bar was 0.8 to 1.3x for sadness, 0.8 to 1.6x for fear; fear at 1.0 point misses it, the other three cells pass on size but their coverage fails).
• (4) refuted by one cell. C costs more audited items than B in every cell. C is below B for fear (40.6 vs 47.2 / 47.6%) and sadness at 1.0 point (76.0 vs 85.2%), but 3.6 points above B for sadness at 0.5 point (75.6 vs 72.0%), past the 2-point bar.
• Not a coding bug. A fixed-size audit with the same estimator, corpus and strata covers fear 94.0% at n = 1,000, 94.0% at 3,454 and 91.3% at 1,738 (300 draws, /tmp check, not a kept experiment; the 87% at n = 457 is the known small-audit normal-interval shortfall). So the loss at stopping is optional stopping: a data-dependent stopping time that waits for a narrow plug-in interval (or for a positive count) selects audits whose variance estimate is low, and with 77 fear positives in the corpus the estimate at that moment is biased and under-dispersed.
• Reading: the 6c sequential audit, which covered 93 to 96% at a 45% prevalence (3 points and 1 point), does not carry to 1 to 2% prevalence. At this prevalence neither a pilot (6r/6s) nor a plain sequential rule gives an efficient valid audit with the normal-approximation interval; the safe pieces found are the Wilson-upper pilot (valid, 1.5 to 3.8x the size) and a fixed-n audit at the oracle size. A minimum-positives guard is not a remedy.
• Limits: one corpus and coder family, 500 draws, the fixed-n check is 300 draws and was not kept as an experiment file, no confidence sequence or alpha-spending correction tried, the batch size 100 and start 400 were not varied.
• Next: a stopping rule with an optional-stopping correction (a confidence sequence for the stratified PPI estimate, or stopping on the Wilson-upper half-width); keep the fixed-n coverage check as an experiment file.

2026-10-27 · Milestone 6s: does a Wilson-upper-bound S2 in the pilot remove the low-n bias of the rare-category audit-size rule? (planned before running)

• Why: 6r's open item. At 1 to 2% prevalence the plug-in pilot S2 is noisy and mostly low (planned n 25 to 30% under the oracle; fear coverage 73% at the 0.5-point target with pilot 400). experiments/h6s_wilson_pilot.py, same GoEmotions subreddit-group corpus, coder, strata, targets, pilot sizes, 500 draws and seed family as 6r, fear and sadness. Three pilot S2 rules per cell: (A) plug-in, as 6r; (B) human-only Wilson-upper: per group p_up = upper end of the 95% Wilson interval for the pilot prevalence (capped at 0.5), S2 = p_up (1 - p_up), no lambda (tuned S2 is never above the human-only S2, so B is conservative for tuned estimation); (C) pooled shrinkage: S2_h = (n_h s2_h + 30 s2_pool) / (n_h + 30) with s2_pool the plug-in S2 on the whole pilot, lambda per group as in A. The audit still uses the tuned estimator with lambda estimated from the audit, so only the planned n changes.
• Predictions (before running): (1) B removes the low bias: mean planned n at pilot 400 is 1.0 to 1.6 times the oracle n for both categories and targets (it over-plans, since a Wilson upper bound at 0 to 4 positives is large), and the share of audits wider than 1.25x target is under 5%. (2) B at pilot 1,500 is 1.0 to 1.25 times oracle. (3) B's fear coverage at the 0.5-point target with pilot 400 is at least 90% (A: 73%). (4) C reduces the bias in A but not to within 10% of oracle for fear at pilot 400 (shrinkage towards a pooled S2 that is itself low-biased).
• Refuted: (1) mean planned n outside 1.0 to 1.6x oracle in any pilot 400 cell, or 5% or more of audits over-wide; (2) outside 1.0 to 1.25x; (3) coverage under 90%; (4) C within 10% of oracle in fear pilot 400.
• Limits known in advance: one corpus and coder family, 500 draws (coverage +/- 2 points), the final interval is still the normal-approximation one, the prevalence in B is pooled over lambda so B cannot show any tuned saving, the shrinkage weight 30 and the 0.5 cap are my choices, not tuned.

### Outcome (experiments/h6s_wilson_pilot.py, about 100 s; experiments/h6s_wilson_pilot.txt; 500 draws per cell, coverage +/- 2 points)

• (1) refuted on size, held on over-width. B over-plans far more than predicted: at pilot 400 the mean planned n is 3.80x / 2.65x the oracle for fear (1.0 / 0.5 point) and 2.51x / 1.82x for sadness (bar 1.0 to 1.6x). It does remove the failure: 0% of audits end wider than 1.25x target in every cell (bar under 5%).
• (2) refuted. B at pilot 1,500 is 2.25x / 1.88x (fear) and 1.73x / 1.47x (sadness) the oracle (bar 1.0 to 1.25x).
• (3) held. B's fear coverage at the 0.5-point target with pilot 400 is 94.4% (A: 70.2% in this run, 73.0% in 6r); at the 1.0-point target 92.0% (A 69.6%). Sadness B covers 93.6 to 94.4%.
• (4) held. C (shrinkage, weight 30) lifts planned n from 0.72 / 0.74x (A) to 0.86 / 0.87x of the oracle for fear at pilot 400, still 13 to 14% low, with coverage 78.4% (1.0 point) and 91.6% (0.5 point). At pilot 1,500 A and C are indistinguishable (0.95 to 0.97x).
• Reading: the 6r failure is a variance underestimate, and a Wilson upper bound removes it completely but at a price of 1.5 to 3.8x the audit, so it is a safe rule, not an efficient one. Bias correction in the plan cannot be had for free from a 400-item pilot at 1%: the pilot holds 4 to 5 fear positives in total. The efficient route is a larger pilot (A at 1,500 is 0.95x of oracle with 22% of audits over-wide at 1.0 point), or the sequential audit of 6c, which needs no pilot variance. C is a small improvement and still undercovers.
• Limits: one corpus and coder family, 500 draws, normal-approximation final interval (fear at the oracle plan already covers 90.8% at 1.0 point), B ignores lambda so it cannot show a tuned saving, shrinkage weight and the 0.5 cap not tuned, a bound at a less conservative level (80% Wilson, say) not tried.
• Next: sequential audit (6c design) at a rare category, which should avoid the pilot problem; a Wilson-type final interval for stratified PPI; the 7B coder; paper Section 6 update (done for 6s).

2026-10-26 · Milestone 6r: does the pilot-estimated audit-size rule survive a rare category (fear, sadness) on the GoEmotions groups? (planned before running)

• Why: 6q's open item. The pilot rule (400 random items -> per-group lambda and S2 -> neyman_total -> fresh audit) worked at a 45% prevalence. At 1 to 2% a pilot of 400 holds 4 to 8 positives in total and some groups none, so the estimated S2 is noisy or zero and the normal-approximation interval is already known to undercover here (milestone 4). experiments/h6r_rare_pilot.py: GoEmotions subreddit-group corpus of 6o, target = a majority of raters marked fear (and, second, sadness), coder = pooled NB trained on the 30% part with cut so coder-positives are about 3x prevalence (milestone 4 recipe), strata = the four groups, tuned stratified PPI, half-width targets of 0.5 and 1.0 point, pilot sizes 400 and 1,500, 500 draws per cell, an oracle plan (S2 from the whole corpus) as reference. S2 floor 1e-4 as in 6q.
• Predictions (before running): (1) Oracle plan: mean half-width within 0.05 point of target, coverage 93 to 96.5% for sadness; for fear coverage may fall below 93% (normal interval). (2) Pilot 400: the sd of planned n over the oracle n exceeds 0.3 for fear (against 0.05 to 0.1 in 6q), and at least 15% of audits end wider than 1.25 times the target. (3) Pilot 1,500 cuts that sd to under 0.2 of oracle n and under 10% of audits wider than 1.25x target. (4) Coverage of the pilot version is below 93% for fear at the 0.5-point target with pilot 400.
• Refuted: (1) a gap over 0.05 point; (2) sd ratio under 0.3 or under 15% over-wide; (3) either limit missed; (4) coverage at or above 93%.
• Limits known in advance: one corpus, one coder family, 500 draws (coverage +/- 2 points), the truth is the finite corpus share, pilot items are excluded from the audit pool.

### Outcome (experiments/h6r_rare_pilot.py, about 1 min; experiments/h6r_rare_pilot.txt; 500 draws per cell, coverage +/- 2 points)

• Corpus 6,683 comments. Fear: prevalence 1.15% (77 positives, 12 to 27 per group), coder agreement 0.961, oracle lambda 0.05 to 0.13. Sadness: 2.30% (154 positives), agreement 0.910, lambda 0.06 to 0.11.
• (1) held. Oracle plan mean half-width 0.965 / 0.500 (fear, 1.0 / 0.5 target) and 0.985 / 0.498 (sadness); coverage sadness 93.6 / 94.2%. Fear covers 90.8% at the 1-point target (normal interval, rare) and 93.8% at 0.5.
• (2) refuted at one cell. Pilot 400, fear: sd of planned n over oracle n is 0.41 and 0.36 (bar 0.3, met), but only 14.8% of audits end wider than 1.25x target at 1.0 point (bar 15%, missed by 0.2 point) and 25.8% at 0.5. The planned n is biased low: mean 291 against 398 and 953 against 1,347.
• (3) refuted for fear, held for sadness. Pilot 1,500, fear: sd/oracle 0.21 and 0.19 (bar 0.2, one miss) and 22 / 22% over-wide (bar 10%). Sadness: 0.14 / 0.11 and 9.8 / 9.0% (held).
• (4) held. Fear at the 0.5-point target with pilot 400 covers 73.0% (pilot 1,500: 93.6%). Sadness with pilot 400 covers 86.8 / 85.0%.
• Reading: at 1 to 2% prevalence a 400-item pilot is not enough to run the rule. It sees few positives, so the estimated S2 is noisy and mostly low, the plan is 25 to 30% under the oracle, the interval is wide and undercovers. A pilot of 1,500 (about 17 fear positives in the pilot) fixes sadness (2.3%) but not fear (1.15%). Rule of thumb from these two cells: the pilot needs on the order of 35 or more positives in the rarest group-pool before the plan is trusted. That number is read off two cells, not derived.
• Limits: one corpus and coder family, a stand-in coder, 500 draws, two categories, no small-count-safe interval used for the final estimate (Wilson-type would be the next fix for the fear undercoverage), no pilot-size rule derived.
• Next: a bias-corrected pilot S2 (plug-in with an upper confidence bound, or a Wilson-type floor) and test whether it removes the low-n bias; the 7B coder; paper Section 6 update.

2026-10-26 · Milestone 6q: the lambda-aware audit-size rule on NaijaSenti with the real Qwen coders, oracle and pilot-estimated (planned before running)

• Why: 6p's open items. 6p showed on the Reddit-group corpus that tuned lambda never needs more audit than human-only and saved 2 to 9% with stand-in coders; it named NaijaSenti with the real Qwen coders (informedness J 0.06 to 0.21 from 5h) and a pilot-estimated version as the next checks. experiments/h6q_naija_sizing.py: the 6,000-tweet subsample with cached Qwen scores (coder = score > 0), strata = language (hau, ibo, pcm, yor), three plans per coder (human-only, lambda = 1, oracle lambda) at 3- and 2-point half-width targets, 1,000 draws; then the pilot version (400 random tweets give lambda and S2 per language, n from neyman_total, a fresh audit from the other tweets, 500 draws).
• Predictions (before running): (1) The oracle-lambda plan is never larger than the human-only plan and saves at most 5% for every Qwen coder (J is low); the lambda = 1 plan needs more than the human-only plan for at least two of the three coders. (2) Oracle plan with the oracle lambda fixed, and with lambda estimated from the audit: mean half-width within 0.05 point of target, coverage 93 to 96.5%. (3) Pilot version: mean planned n within 15% of the oracle n, mean realised half-width within 10% of target, and at least 70% of audits at or under 1.05 times the target; coverage 92 to 97%. (4) The 3B coder saves more than the 0.5B coder.
• Refuted: (1) any oracle plan above human-only, a saving over 5%, or fewer than two lambda = 1 plans above human-only; (2) a gap over 0.05 point or coverage outside 92 to 97%; (3) any of the stated limits missed; (4) the 3B saving not above the 0.5B one.
• Limits known in advance: one corpus subsample (corpus = the 6,000 tweets, truth is their finite mean), zero-shot cut score > 0, oracle S2 from the whole corpus, the pilot's items are excluded from the audit pool (a small design deviation: audit items are drawn from a corpus minus the pilot) while the estimator still uses all N_h coder codes.

### Outcome (experiments/h6q_naija_sizing.py, about 90 s; experiments/h6q_naija_sizing.txt; 1,000 draws per cell, 500 for the pilot version)

• Oracle per-language lambda: 0.5B 0.00 to 0.79 (erratic: Yoruba 0.79, Nigerian Pidgin 0.00), 1.5B 0.01 to 0.38, 3B 0.28 to 0.79. Coder agreement 0.668, 0.661, 0.689.
• Audit size at 3 / 2 points: human-only 760 / 1,474; plain PPI (lambda = 1) 820 / 1,574 (0.5B), 1,120 / 2,042 (1.5B), 785 / 1,517 (3B); oracle lambda 758 / 1,472, 724 / 1,415, 748 / 1,454.
• (1) held. The oracle plan is never above human-only. Savings: 0.5B 0.3% / 0.1%, 1.5B 4.7% / 4.0%, 3B 1.6% / 1.4% (all under 5%, the largest is close to the line). Plain PPI needs more than human-only for all three coders (1.03 to 1.47 times).
• (2) held. Oracle plan with oracle lambda or audit-estimated lambda: mean half-width 2.99 to 3.00 and 2.00; coverage 93.7 to 95.9% (lowest: 1.5B, 2 points, estimated lambda, 93.7%).
• (3) held. Pilot of 400 tweets: mean planned n is within 0.5% to 1.1% of the oracle n (sd 31 to 62); mean realised half-width 3.01 to 3.02 and 2.01 to 2.02; 88.4 to 92.8% of audits are at or under 1.05 times the target; coverage 94.6 to 96.8%.
• (4) held, by a small margin. 3B saves 1.6% / 1.4% against 0.5B's 0.3% / 0.1%. But the saving is not monotone in model size: the 1.5B coder saves the most (4.7%) while its plain-PPI plan is the worst (+47%), so informedness and calibration of the cut matter, not size (consistent with 5h).
• Reading: the rule behaves on the real-coder corpus as it did on the stand-ins: use tuned lambda, size the audit from a pilot of about 400, expect a saving of 0 to 5% from a zero-shot 0.5B to 3B Qwen coder, and never use lambda = 1 with them.
• Limits: one corpus subsample, zero-shot cut at 0, oracle S2 from the whole corpus, the pilot was a simple random sample (not language-stratified), audit pool excludes the pilot, groups are languages, so a different group definition could change the lambda spread. The 7B coder is not run.
• Next: the 7B coder; a rarer-category check of the pilot version; paper Section 7 update; owner submission.

2026-10-26 · Milestone 6p: a lambda-aware audit-size rule, so the plan can choose its control variate (planned before running)

• Why: 6o found the lambda = 1 stratified PPI plan needs 1.23 to 1.58 times the human-only plan's audit on weak coders, and said "PPI++ (tuned lambda) is the fix and is not covered by neyman_total". With the stratum's own weight lambda_h = cov(y, f) / var(f) (f is known on every item, so no 1 + n/N term) the residual variance is var(y) (1 - rho_h^2), which is never above either endpoint (lambda = 0 human-only, lambda = 1 plain PPI). New library code: stratified_ppi_tuned (per-stratum lambda, estimated from the audit, clipped to [0, 1]) and stratum_lambda_s2 (oracle lambda and S2 for sizing), with tests. experiments/h6p_lambda_sizing.py, same Reddit-group corpus, coders, seed family and 3- and 2-point targets as 6o; per coder three plans (human-only, lambda = 1, oracle-lambda), each sized by neyman_total on its own S2 with Neyman allocation, 1,000 draws each; the oracle-lambda plan is also run with the audit-estimated lambda (what a practitioner can do).
• Predictions (before running): (1) The oracle-lambda plan's n is below both other plans' n for every coder and target, and saves 15 to 35% against human-only for the pooled coder and 3 to 15% for the sports-only and personal-only coders (their S2 inflation in 6o was largest, so their correlation with y is lowest). (2) Run with the oracle lambda fixed, the plan hits its target: mean half-width within 0.05 point, coverage 93 to 96.5%. (3) Run with the audit-estimated lambda at the same n, the mean half-width exceeds the target by at most 0.10 point at 3 points and 0.08 at 2 points (estimating 4 lambdas costs some width, less at larger n), coverage 93 to 96.5%. (4) At the oracle-lambda plan's n and allocation, the estimated-lambda estimator's mean half-width is below the human-only estimator's for every coder (i.e. practitioners lose nothing by tuning).
• Refuted: (1) any plan ordering violated, or a saving outside the stated bands; (2) a gap over 0.05 point or coverage outside 92 to 97%; (3) an excess over the stated limits or coverage outside the band; (4) any coder where tuned is not narrower.
• Limits known in advance: one corpus (the second one for this rule; the first, NaijaSenti, has real LLM coders and is the next check), stand-in coders, oracle lambda and S2 from the whole corpus, groups made by hand, lambda estimated in-sample from the same audit that estimates the mean.

### Outcome (experiments/h6p_lambda_sizing.py, about a minute; experiments/h6p_lambda_sizing.txt; 1,000 draws per cell)

• Oracle per-group lambda: pooled coder 0.28 to 0.35, sports-only 0.11 to 0.17, personal-only 0.14 to 0.23. Residual S2 against human-only (about 0.17 to 0.19): pooled 0.155 to 0.172, the others 0.161 to 0.188; plain PPI 0.23 to 0.33.
• Audit size at 3 / 2 points (human-only 688 / 1,371; plain PPI 873 / 1,689 pooled, 1,088 / 2,033 sports-only, 980 / 1,862 personal-only): oracle-lambda plan 629 / 1,265 (pooled), 673 / 1,345 (sports-only), 670 / 1,339 (personal-only). Against human-only: 0.914 / 0.923, 0.978 / 0.981, 0.974 / 0.977. Against plain PPI: 0.62 to 0.75.
• (1) half held, half refuted. The ordering held in all six cells (oracle below both others). The savings bands were wrong, in the pessimistic direction: pooled 7.7 to 8.6% (predicted 15 to 35%), sports-only and personal-only 1.9 to 2.6% (predicted 3 to 15%). I overrated how much a 0.7-agreement coder correlates with y.
• (2) held. Oracle lambda fixed: mean half-width 3.00 and 2.00 in all six cells, coverage 94.8 to 96.1%.
• (3) held. Lambda estimated from the audit, same n: mean half-width 2.98 to 2.99 at 3 points and 2.00 at 2 points (no excess; slightly under target, so estimating lambda in-sample cost no width here), coverage 94.8 to 96.0%.
• (4) held. At the oracle plan's n and allocation the human-only estimator is wider: 3.15 / 2.10 (pooled), 3.03 / 2.02 and 3.04 / 2.03 (others), against 2.98 to 3.00 / 2.00 for tuned.
• Reading: the 6o puzzle is resolved. Plain PPI with lambda = 1 costs 23 to 58% more audit than human-only on these coders; tuned lambda cannot lose to human-only and here saves 2 to 9%, i.e. with coders of agreement 0.69 to 0.77 the honest recommendation is "tune lambda, expect a small saving, do not expect the coder to pay for itself". The sizing rule needs only per-stratum S2 of y - lambda f, which stratum_lambda_s2 gives and a pilot audit can estimate (6b).
• Limits: one corpus, stand-in coders, oracle lambda and S2 from the whole corpus (the pilot version, with lambda estimated on a pilot, is not run here), lambda estimated in-sample from the audit that also estimates the mean (coverage stayed in band, n = 150 to 440 per group).
• Next: the same check on NaijaSenti with the real Qwen coders (informedness J 0.06 to 0.21 predicts about 0 to 4% saving); the pilot-estimated version of the rule; the 7B coder; owner submission.

2026-10-26 · Milestone 6o: does the audit-size rule (6a to 6f) hold on a second corpus, GoEmotions subreddit groups? (planned before running)

• Why: Section 7 / the sizing result rests on NaijaSenti only (the "second corpus for sizing" item open since 5). The Reddit-group corpus of group_shift_reddit.py (6,683 comments, four hand-fixed groups, target "negative") is smaller (so finite-population terms bite harder), has a different error structure (one language, coder agreement 0.69 to 0.77 for every group) and a base rate of 23.5%. experiments/h6o_reddit_sizing.py: same three coder kinds as group_shift_reddit.py (pooled LR, sports-only LR, personal-only LR), library neyman_total, stratified PPI, no change to the rule.
• Design: Part 1, plug-in (predicted) vs simulated mean half-width of stratified PPI at three allocations (equal quota 100 each, 70% sports, Neyman at n = 400), 1,000 draws. Part 2, the rule's n for a 3-point and a 2-point half-width target, Neyman allocation, 1,000 draws, report mean and 95th percentile half-width, coverage, and the human-only stratified plan's n for the same target. Seed 20261026.
• Predictions (before running): (1) simulated / predicted half-width ratio is 0.97 to 1.03 for every coder and design (it was 0.99 to 1.01 on NaijaSenti). (2) At the rule's n the mean simulated half-width is within 0.1 point of the target at 3 points, and within 0.15 at 2 points. (3) Coverage is 93 to 96.5% in every Part 2 cell (binomial sd about 0.7 points). (4) The weak coders do not pay: the PPI plan's n is at least 0.9 of the human-only plan's n for every coder (agreement is 0.69 to 0.77, informedness low), i.e. no more than a 10% saving.
• Refuted: (1) any ratio outside 0.95 to 1.05 (0.97 to 1.03 is the prediction; outside 0.95 to 1.05 is a failure); (2) a gap over 0.2 point; (3) any cell outside 92 to 97%; (4) any PPI plan more than 10% smaller than the human-only plan.
• Limits known in advance: groups defined by hand, one corpus per family, stand-in coders (not an LLM), the corpus is the test split so truth is the finite corpus, plug-in variance computed from the full corpus (an oracle, as in 6a; the pilot version is 6b).

### Outcome (experiments/h6o_reddit_sizing.py, 8 s; experiments/h6o_reddit_sizing.txt; 1,000 draws per cell)

• (1) held. Simulated / predicted half-width is 0.999 to 1.002 in all nine coder x design cells (equal quota 4.7 to 5.3 points, 70% sports 6.7 to 7.7, Neyman n = 400 4.6 to 5.2).
• (2) held. At the rule's n the mean simulated half-width is 3.00 and 2.00 (to two decimals) in all six cells; the 95th percentile is 3.10 to 3.14 and 2.05 to 2.06.
• (3) held. Coverage 94.4 to 96.3% in the six Part 2 cells (the nine Part 1 cells: 93.5 to 96.7%, so one is at the edge of the 93 to 96.5 band I wrote for Part 2 only).
• (4) refuted, and in the opposite direction. The PPI plan needs more audited items than the human-only stratified plan, not at most 10% fewer: 1.23 to 1.27x (pooled coder), 1.48 to 1.58x (sports-only), 1.36 to 1.42x (personal-only). At 3 points: 873, 1,088 and 980 against 688. With coder agreement 0.69 to 0.77 the residual r = y - f has more variance than y itself (S2 0.23 to 0.33 against about 0.18 for y), and stratified_ppi uses lambda = 1 (no tuning), so the control variate adds variance. The rule is right about its own estimator and also says that estimator is the wrong choice here; PPI++ (tuned lambda) is the fix and is not covered by neyman_total.
• Limits: one second corpus, hand-made groups, stand-in coders, oracle plug-in variances from the whole corpus, lambda = 1 only; the corpus is 6,683 items so a 2-point audit is 25 to 30% of it (finite-population correction large, and the formula still held).
• Next: a lambda-aware sizing rule (variance of y - lambda f) so the plan can choose PPI++ or human-only; the 7B coder; owner submission.

2026-10-26 · Paper draft 1.2: Section 8 and the abstract corrected for 5h

• No experiment. The paper still said the real null is "what a 0.66 to 0.69 coder with this error structure should give" and that a gain needs agreement near 0.80; 5h showed agreement was the wrong axis. Rewrote the abstract sentence and added the 5h paragraph to Section 8 (numbers checked against h5h_informedness.txt). Still open: 7B coder, second corpus for sizing, LaTeX template, owner submission.

2026-10-26 · Milestone 5h: is the real coders' null explained by informedness, not agreement? (planned before running)

• Why: 5f's reading called the real null "not an anomaly" because 5f's synthetic coders at agreement 0.60 to 0.70 landed near the real ratios. But the 3B fixed-cut coder has sensitivity 0.072 and specificity 0.989 (h3_llm-Qwen2.5-3B-Instruct.txt): it is almost a constant "not negative" code, and its agreement 0.689 is barely above the 0.673 a constant gets on this corpus. A symmetric coder at agreement 0.69 has Youden J = 0.38; the real one has J = 0.06. So 5d to 5f compared coders of very different information at the same agreement, and 5f's reading may be right for the wrong reason. experiments/h5h_informedness.py: for each real coder (0.5B, 1.5B, 3B; fixed cut score > 0, and the cut maximising J over the whole corpus, an in-sample diagnostic with no label cost charged) a synthetic coder with the same sensitivity and specificity and independent errors, plus a symmetric a = 0.69 coder; same corpus, truth, tuned PPI++ + HC3 + t and human-only comparator as 5f; n = 100, 200, 400; 600 draws; seed 20261105; one coder draw per cell.
• Predictions (before running): (1) The matched-(sens, spec) independent coder reproduces each real coder's mean width ratio at n = 400 within 0.02, for all six real coders (so informedness, not error correlation, carries the gain). (2) The symmetric a = 0.69 coder gives a ratio at n = 400 at most 0.95, while the 3B fixed-cut coder gives at least 0.98: a gap of 0.03 or more at the same agreement. (3) Every real coder's ratio at n = 400 is at least 0.90 (none reaches H3's 20% gain bar; with J this small the control variate is weak). (4) Matched-synthetic PPI++ coverage mean stays 93 to 96.5% at n >= 200.
• Refutes: (1) any difference of 0.02 or more; (2) symmetric ratio above 0.95, or a gap under 0.03; (3) any ratio under 0.90; (4) any cell outside.
• Limits known in advance: the best-J cut is chosen on the same items it is scored on; one corpus and one prompt; one coder draw per cell; matched synthetic coders have errors independent of text, so a difference from the real coder is the measured effect of correlation beyond sens/spec.

### Outcome (experiments/h5h_informedness.py, h5h_informedness.txt; 600 draws per cell)

• Real coders: 0.5B fixed cut sens 0.031 / spec 0.977 (J 0.008); 1.5B fixed 0.439 / 0.769 (J 0.208); 3B fixed 0.072 / 0.989 (J 0.061). Best-J cuts (in-sample): 0.5B J 0.033, 1.5B 0.212, 3B 0.194. A symmetric a = 0.69 coder has J 0.403.
• Mean tuned PPI++ / human-only half-width ratio at n = 400, real against matched-(sens, spec) independent-error synthetic: 0.5B fixed 0.996 / 0.996; 0.5B bestJ 0.993 / 0.996; 1.5B fixed 0.978 / 0.980; 1.5B bestJ 0.979 / 0.977; 3B fixed 0.993 / 0.987; 3B bestJ 0.976 / 0.972. Symmetric a = 0.69: 0.937.
• (1) held. Largest real-minus-matched difference at n = 400 is 0.006 (3B fixed). (2) held. 0.937 against 0.993 for the 3B fixed cut, a gap of 0.056 at agreement 0.69 to 0.70. (3) held. The smallest real ratio is 0.976. (4) held. Matched-synthetic PPI++ coverage 0.943 to 0.955 at n >= 200.
• Reading: this corrects the reading of 5f (and the README paragraph that repeated it). The real coders' null is not "what a 0.6 to 0.7 coder with length- or label-dependent errors gives": agreement was the wrong axis. On this corpus (32.7% negative) a constant "not negative" code agrees 0.673, so the 3B fixed-cut coder at 0.689 is almost a constant (J 0.06) and the 0.5B one is constant (J 0.008). What sets the width gain is informedness: a coder with the real J and independent errors reproduces the real ratio to within 0.006 at n = 400, so error correlation with the text, beyond sensitivity and specificity, costs nothing measurable here. The synthetic symmetric coders of 5d to 5f at "agreement 0.70" had J 0.40, twice the best real coder's 0.21. To reach the 20% gain bar a coder needs J of roughly 0.5 or more (5d: a = 0.75, J 0.5, ratio 0.90).
• Relation to 5g (landed in parallel, h5g_fitted): its fitted human-label term (+7.38 logit for 3B, almost every negative missed) is the low sensitivity seen here; 5g shows the fitted errors reproduce the real ratio, 5h shows sens/spec alone suffice, and that agreement 0.69 hid a J of 0.06.
• Limits: the best-J cut is chosen on the items it is scored on; one corpus (NaijaSenti, negative sentiment) and one prompt; one coder draw per cell, so the matched coder's realised sens/spec differ from the real by up to 0.02 (3B bestJ J 0.173 against 0.194), which is a rough match; the 5e and 5f results about error shape stand as results about synthetic coders but do not explain the real null.
• Next: the 7B or a fine-tuned coder (J above 0.5) when a GPU or longer budget is available; report J, not agreement, in the paper's coder table.

2026-10-26 · Milestone 7 (part): paper draft 1.1 adds Section 8, what a coder must be to pay

• No experiment this step. paper/paper.md stopped at milestone 6 plus the first real-LLM run; 5b to 5g (model size, charged calibration, break-even accuracy, correlated error, fitted error) were only in this log. Added Section 8, updated the abstract, limits numbering and status table. Numbers for 5b, 5d and 5g were checked against experiments/*.txt; the rest are copied from the entries below.
• Correction found while writing: the 5d outcome says a 20% gain needs a = 0.75, but its own table gives a width ratio of 0.898 there (a 10% cut); 0.811 at a = 0.80 is the 19% cut. The paper says roughly 0.80.
• Still open: read Gligorić et al. and Fisch et al. in full, a 7B or multilingual model, a second corpus, the LaTeX template, owner submission.

2026-10-26 · Milestone 5g: does a synthetic coder with error fitted to the real 1.5B and 3B errors reproduce their width ratios?

• Process deviation, disclosed: I ran the experiment before writing this entry, so the hypotheses below are stated after seeing the numbers and are not pre-registered. Read the outcome as descriptive.
• Question (5f's "Next"): experiments/h5g_fitted.py. Per model, logistic regression of the real coder's error (code != human label, score > 0 rule, 6,000 items) on group dummies, standardised log length, human label and "@" presence; synthetic coder flips the human label with the fitted per-item probability (one draw, seed 20261105). Then tuned PPI++ + HC3 + t against human-only for the real and the fitted coder, n = 100, 200, 400, 600 draws, same corpus, truth and estimators as 5b to 5f.
• Intended hypotheses: (1) the fitted coder's n = 400 ratio is within 0.01 of the real one for both models; (2) its coverage stays 93 to 96.5%.

### Outcome (experiments/h5g_fitted.txt)

• Fitted error coefficients (logit): 1.5B: human-negative +1.70, group ibo -0.22, hau -0.46, pcm -1.28, length +0.11, "@" -0.35. 3B: human-negative +7.38 (almost every missed negative), pcm -0.86, length -0.14, "@" -0.32. Realised agreement of the fitted coder 0.658 and 0.686 against real 0.661 and 0.689.
• Ratio PPI++/human-only at n = 100 / 200 / 400. 1.5B: real 0.959 / 0.971 / 0.978; fitted 0.966 / 0.978 / 0.985. 3B: real 0.974 / 0.983 / 0.993; fitted 0.973 / 0.983 / 0.993. Coverage 0.933 to 0.953 in every cell.
• (1) holds for 3B (identical at n = 400) and misses for 1.5B by 0.007 (inside 0.01). (2) holds.
• Reading: the real coders' null is reproduced by a coder whose errors depend only on the human label, group, length and "@". The 3B result is almost entirely the label term (it misses nearly every negative), the 1.5B a milder version. The fitted coder is slightly worse for 1.5B, so the real 1.5B has a little text-specific signal beyond these features, worth about 0.007 of width ratio. Together with 5f this closes the "why no gain" question for these two models: low agreement with label-dependent errors, no exotic cause needed.
• Limits: one corpus; one draw of the fitted coder; the fit is in-sample on the same items; the 7B model is untested.
• Next: a model with real signal (7B or few-shot) when compute allows; otherwise move to milestone 6, the closed-form audit-size rule.

2026-10-26 · Milestone 5f: at the real coders' accuracy (0.70), and with errors tied to a text feature, does synthetic PPI++ still beat human-only? (planned before running)

• Why: 5e's "Next". Its coders at 0.80 and 0.90 were far better than the real 1.5B and 3B (agreement 0.66 to 0.69), so it could not say whether correlated error explains the real null. experiments/h5f_text_error.py: same corpus, truth, estimators and comparator as 5e, accuracy a = 0.70 and 0.60, mechanisms indep, group, length, class (weights unchanged from 5e) plus mention: flip probability 3 times larger when the text contains "@" (47% of items; the feature is fixed now, picked from four candidates by prevalence only, not by its effect on any result). Mean flip rate pinned to 1 - a as in 5e. n = 100, 200, 400; 600 draws; seed 20261104; one coder draw per cell.
• Predictions (before running): (1) At a = 0.70, n = 400 indep ratio is 0.93 to 0.94 (5d: 0.934). (2) No mechanism at a = 0.70 pushes the n = 400 ratio above 1.00. (3) mention costs less than 0.03 over indep at a = 0.70, n = 400 (a binary feature the regression does not see, but present in 47% of items, so error stays spread out). (4) At a = 0.60 the range of ratios over mechanisms at n = 400 stays under 0.06. (5) Tuned PPI++ coverage mean for a = 0.70 at n >= 200 stays 93 to 96.5% in every mechanism.
• Refutes: (1) outside [0.92, 0.95]; (2) any ratio above 1.00; (3) a gap of 0.03 or more; (4) a range of 0.06 or more; (5) any cell outside.
• Limits known in advance: synthetic coders; weights illustrative; one corpus and one text feature; one coder draw per cell.

### Outcome (experiments/h5f_text_error.py, h5f_text_error.txt; 600 draws per cell; realised agreement 0.697 to 0.706 at a = 0.70, 0.591 to 0.608 at 0.60)

• Mean tuned PPI++ / human-only half-width ratio at n = 100 / 200 / 400. a = 0.70: indep 0.914 / 0.929 / 0.936; group 0.858 / 0.875 / 0.884; length 0.944 / 0.958 / 0.965; class 0.929 / 0.946 / 0.954; mention 0.877 / 0.892 / 0.902. a = 0.60: indep 0.959 / 0.972 / 0.979; group 0.923 / 0.939 / 0.946; length 0.978 / 0.988 / 0.992; class 0.977 / 0.987 / 0.992; mention 0.942 / 0.958 / 0.965.
• (1) held (indep 0.936). (2) held (largest at a = 0.70 is 0.965, length). (4) held (range at a = 0.60, n = 400 is 0.946 to 0.992 = 0.046). (5) held (PPI++ coverage 0.937 to 0.954 at a = 0.70, n >= 200).
• (3) refuted by the letter. mention was 0.034 narrower than indep at a = 0.70, n = 400 (0.902 against 0.936), not within 0.03; I had the sign wrong again, as with group in 5e. Post hoc reading, untested: concentrating errors in a subset of items makes the coder exact elsewhere, which is where the control variate works.
• Reading: no error structure tested turns the gain negative at 0.60 to 0.70; the most unfavourable ones (length, class) land at 0.965 to 0.992, and the real coders' measured ratios (1.5B 0.978, 3B 0.993 at n = 400, 5b) sit inside that range at their agreement (0.66 to 0.69). So the real null is not an anomaly needing an exotic explanation: it is what a 0.6 to 0.7 coder with length- or label-dependent error is expected to give. This corrects 5d's reading that real errors must be correlated "with text content" to break the gain; the synthetic indep coder was just too optimistic, and a real gain needs agreement near 0.80 or more with errors that are not length- or label-dependent.
• Limits: synthetic; weights illustrative; one text feature; one coder draw per cell; the real coders were not fitted to any mechanism, only compared by range.
• Next: fit the mechanism weights to the real 1.5B and 3B errors (logistic regression of error on group, length, label, "@") and see whether the fitted synthetic coder reproduces their measured ratios; then the 7B model when a GPU or longer budget is available.

2026-10-26 · Milestone 5e: how much does correlated coder error cost PPI++ at matched accuracy? (planned before running)

• Why: 5d's "Next". Independent flips need almost no accuracy to beat human-only, but real 1.5B and 3B coders did not (5b, 5c), so agreement alone does not set the gain. experiments/h5e_correlated.py: same corpus, truth, tuned PPI++ + HC3 + t and comparator as 5d. The synthetic coder flips the human label with a per-item probability p_i whose mean over the corpus is set exactly to 1 - a (a = 0.80 and 0.90), so overall agreement is identical across mechanisms. Mechanisms: indep (p_i constant); group (p proportional to 0.5, 1, 1.5, 2.5 for the baseline group, hau, ibo, pcm); length (p proportional to exp(z), z the standardised log length, the covariate in X); class (p three times larger when the human label is 1 than 0). Weights fixed now, not tuned; p_i clipped to 0.5 after scaling, then rescaled once more to restore the mean. n = 100, 200, 400; 600 draws; seed 20261103; one coder draw per cell.
• Predictions (before running): (1) group and length errors are mostly absorbed by the regression covariates, so their width ratio is within 0.03 of indep at every n (the control variate still has the signal: errors are predictable from X). (2) class errors cost more: ratio at a = 0.80, n = 400 at least 0.05 above indep, because label-dependent error shifts the coder's mean away from the truth in a way X cannot explain. (3) Tuned PPI++ coverage mean stays 93 to 96.5% for a = 0.80 at n >= 200 in all four mechanisms. (4) No mechanism at a = 0.80 pushes the ratio above 1.00 at any n.
• Refutes: (1) any gap above 0.03; (2) a gap under 0.05; (3) any cell outside; (4) any ratio above 1.00.
• Limits known in advance: weights are illustrative, not fitted to a real coder; one corpus; one coder draw per cell, so a cell includes its realised error pattern.

### Outcome (experiments/h5e_correlated.py, h5e_correlated.txt; 600 draws per cell; realised agreement 0.791 to 0.802 at a = 0.80 and 0.898 to 0.908 at 0.90)

• Mean tuned PPI++ / human-only half-width ratio at n = 100 / 200 / 400. a = 0.80: indep 0.822 / 0.847 / 0.849; group 0.770 / 0.792 / 0.801; length 0.868 / 0.884 / 0.891; class 0.819 / 0.846 / 0.853. a = 0.90: indep 0.619 / 0.640 / 0.649; group 0.570 / 0.600 / 0.605; length 0.683 / 0.711 / 0.713; class 0.606 / 0.623 / 0.636.
• (1) refuted, in both directions. Group-dependent error made PPI++ narrower than independent error at the same accuracy (about 0.05 lower at a = 0.80) and length-dependent error made it wider (about 0.04 higher). The regression absorbs neither fully. I guessed wrong about the sign for group: concentrating errors in one group leaves the coder very accurate elsewhere, so the control variate is stronger there; this is a post hoc reading, not tested.
• (2) refuted. Class-dependent error cost 0.004 at a = 0.80, n = 400 (0.853 against 0.849), nowhere near 0.05.
• (3) held. Tuned PPI++ coverage at a = 0.80, n >= 200 is 0.942 to 0.958 in all four mechanisms. At a = 0.90, n = 100 it is 0.927 (indep) to 0.950; the small-audit undercoverage of 5d shows again for independent flips and length.
• (4) held. Largest ratio at a = 0.80 is 0.891 (length, n = 400).
• Reading: at matched accuracy, the structure of the errors moves the width ratio by up to about 0.1, but all four mechanisms still beat human-only by 10 to 23% at a = 0.80. So correlated error of these kinds does not explain why the real 1.5B and 3B coders (agreement 0.66 to 0.70) gained nothing: the synthetic coders at 0.80 are much better than the real ones, and the gap to 0.70 was not tested here. The open candidate is that real error is correlated with the text content, which none of these four mechanisms models.
• Limits: weights illustrative, not fitted; one coder draw per cell; accuracy 0.80 and 0.90 only; one corpus; the group and length results are single draws of one error pattern each.
• Next: re-run at a = 0.70 (the real coders' level) with the same four mechanisms, and a mechanism that flips by a text feature (a lexicon hit) rather than by group, length or label.

2026-10-26 · Milestone 5d: how accurate must a coder be before PPI++ beats human-only labelling? (planned before running)

• Why: 5, 5b and 5c found no gain from 0.5B, 1.5B or 3B zero-shot coders (best tuned width ratio 0.978 at n = 400, and a loss once the cut is charged). The next question for a reader is the target: what agreement does a coder need? A synthetic coder on the real human labels answers it without a bigger model: the coder returns the human label flipped independently with probability 1 - a (symmetric accuracy a, errors independent of covariates), swept over a. experiments/h5d_breakeven.py: same corpus, truth, regression, tuned PPI++ + HC3 + t as 5c, n = 100, 200, 400, a in {0.60, 0.70, 0.75, 0.80, 0.85, 0.90, 0.95}, 600 draws, seed 20261102. Comparator is human-only on the same n labels (the coder is free to run on all items).
• Predictions (before running): (1) The width ratio PPI++(n) / human-only(n) crosses 1.00 between a = 0.70 and 0.80 at every n (a 0.75-accurate independent-flip coder has little signal beyond what the covariates already give). (2) At a = 0.90 the ratio is below 0.90 at n = 400 (H3's 20% gain bar needs a coder better than 0.90). (3) Tuned PPI++ coverage mean stays within 93 to 96.5% in every cell. (4) The measured 3B zero-shot agreement (0.7) sits below the break-even, consistent with 5b.
• Refutes: (1) a crossing outside [0.65, 0.85] at any n; (2) a ratio at or above 0.90 at a = 0.90, n = 400 (then the bar is stricter than I think); (3) any cell outside; (4) not testable here beyond comparing numbers.
• Limits known in advance: symmetric independent flips are friendlier than a real LLM (whose errors are correlated with the text and the group), so the break-even here is a lower bound on the accuracy a real coder needs; one corpus; one outcome (negative sentiment), five coefficients.

### Outcome (experiments/h5d_breakeven.py, 600 draws per cell, experiments/h5d_breakeven.txt)

• Mean tuned PPI++ / human-only half-width ratio at n = 100 / 200 / 400: a = 0.60: 0.968 / 0.978 / 0.984; 0.70: 0.912 / 0.923 / 0.934; 0.75: 0.870 / 0.888 / 0.898; 0.80: 0.785 / 0.800 / 0.811; 0.85: 0.740 / 0.754 / 0.765; 0.90: 0.618 / 0.641 / 0.656; 0.95: 0.429 / 0.450 / 0.461.
• (1) refuted. There is no crossing between 0.70 and 0.80: the ratio is already below 1 at a = 0.60 (0.97 to 0.98). With independent flips the break-even sits between 0.50 (no information) and 0.60, so a coder needs almost nothing, provided its errors are independent of the text and group.
• (2) held. At a = 0.90, n = 400 the ratio is 0.656, well under 0.90; a 20% gain needs a = 0.75 (0.898) here.
• (3) refuted. Tuned PPI++ coverage falls below 93% at n = 100 for a = 0.90 (91.8) and 0.95 (89.0), and at a = 0.95, n = 200 (92.6); a = 0.85 is 93.4, inside the bar. Means over five coefficients. The high-accuracy coder shrinks the interval so much that the in-sample tuning of lam (which picks lam near 1 on the 21-point grid) undercovers at small audits; human-only holds 95.0 to 95.7.
• (4) the comparison is the finding. Real 1.5B agreement with the human label is 0.661; the independent-flip coder at 0.60 to 0.70 would give a ratio of about 0.92 to 0.98 at n = 400, but the real 1.5B and 3B coders gave 0.978 or worse (5b) and lost once the cut was charged (5c). Real coder errors are correlated with the text and the group, which breaks the control variate's gain; agreement alone is not the quantity that sets the gain.
• Reading: the useful targets are the correlation of the coder's error with the covariates, not accuracy alone, and a small-audit coverage fix for high-accuracy coders (cross-fitting 6i helped for the earlier coders; not re-run here). Limits: the synthetic coder is an upper bound on what a real one does; one fixed coder draw per accuracy level (not re-drawn per audit draw), so coverage includes that draw's realised error pattern; one corpus.
• Next: a coder whose flip probability depends on a covariate (group or length) at matched overall agreement, to measure how much correlated error costs; re-run the a = 0.90 and 0.95, n = 100 cells with the cross-fit estimator.

2026-10-26 · Milestone 5c: does a threshold chosen on c = 100 human labels, charged to PPI++, give a width gain over human-only with n + 100 labels? (planned before running)

• Why: 5b found the zero-shot cut score > 0 decides the gain (0.5B and 3B call almost nothing negative). The cached continuous scores let us pick the cut on human labels, but those labels cost audit budget. experiments/h5c_calibrated.py: per draw, sample n + c rows; the first c (c = 100) pick the cut maximising balanced accuracy (sens + spec) over the observed scores; PPI++ (tuned, HC3 + t, as 5b) uses the other n rows; the human-only comparator gets all n + c labels (so the cut's cost is charged). n = 100, 200, 400; 600 draws; seed 20261101; the three cached models.
• Predictions (before looking): (1) Calibrated agreement/balanced accuracy rises above the fixed-cut one for 0.5B and 3B (sens + spec > 0.5 + 0.2). (2) PPI++(n) width / human-only(n + 100) is above 1.00 for all three models at n = 100 (the 100 spent labels beat a 3 to 6% gain). (3) At n = 400 the ratio is below 1.00 for no model by more than 3%. (4) Tuned PPI++ coverage mean stays 93 to 96.5% at n >= 100 (selection of the cut on separate rows should not break it).
• Refutes: (1) sens + spec under 0.7 for 0.5B or 3B; (2) a ratio of 1.00 or below at n = 100 for any model; (3) a ratio below 0.97 at n = 400 for any model; (4) any cell mean outside.
• Not tested: other cut rules, a bigger c, a 7B model, few-shot.

### Outcome (experiments/h5c_calibrated.py, 600 draws per cell, 3 processes; experiments/h5c_<model>.txt)

• Mean calibrated sens + spec (a coder with no information scores 1.00; I wrote the prediction against 0.5 by mistake, so I judge it against the intended "clearly informative", above 1.2): 0.5B 0.998, 1.5B 1.19, 3B 1.18. The 0.5B cut collapses to a constant code: its score carries no usable signal on this task. 3B's score is informative (1.18) once the cut is moved, though its fixed-cut code (1.01) was not.
• PPI++(n) half-width / human-only(n + 100) half-width, mean over coefficients, n = 100 / 200 / 400: 0.5B 1.443 / 1.238 / 1.128; 1.5B 1.419 / 1.220 / 1.113; 3B 1.425 / 1.222 / 1.115. Tuned PPI++ coverage (mean): 93.9 / 94.8 / 94.4 (0.5B), 94.0 / 94.8 / 94.2 (1.5B), 93.9 / 95.0 / 94.1 (3B); human-only 95.6 / 95.3 / 94.9.
• (1) refuted as intended (3B 1.18, 0.5B 1.00, both under 1.2), and the stated bar was ill-posed.
• (2) held. All ratios at n = 100 are 1.42 to 1.44.
• (3) held. The best n = 400 ratio is 1.113, no cell under 1.
• (4) held. Every cell mean is 93.9 to 95.0.
• Reading: charging 100 human labels for the cut loses. 100 extra labels buy the human-only interval roughly 10% (n = 400) to 30% (n = 100) in width, far more than a calibrated 1.5B or 3B coder adds. With these three small models there is no regime in n = 100 to 400 where picking the cut on labels beats just labelling more. The honest route to a gain is a better coder, not a better cut.
• Limits: one cut rule (balanced accuracy), c fixed at 100, one corpus, one prompt; the comparator assumes the calibration labels could instead have been audit labels (they could).

2026-10-26 · Milestone 5b: does the PPI++ width gain grow with model size within one family? (planned before running)

• Why: milestone 5 used one 1.5B model and ended "a larger model is next". A 7B model is out of reach on this CPU, but Qwen2.5-0.5B and 3B-Instruct are not, and give three sizes of one family with the same prompt, items and truth (experiments/llm_code.py unchanged, experiments/h3_llm_width.py unchanged, same seed 20261031, 600 draws).
• Predictions (before looking at any score): (1) Agreement with the human label rises with size: 0.5B < 1.5B (0.661) < 3B. (2) Tuned PPI++ + HC3 + t half-width ratio at n = 400 falls with size: 0.5B above 0.978, 3B below 0.978. (3) The 3B ratio at n = 400 is still above 0.90, i.e. no 20% gain (H3's bar) from a 3B zero-shot model. (4) Tuned PPI++ mean coverage stays in 93 to 96.5% at n >= 100 for both new models.
• Refutes: (1) any non-monotone agreement; (2) a non-monotone ratio at n = 400; (3) a ratio at or below 0.90 for the 3B model; (4) any n >= 100 mean outside 93 to 96.5%.
• Not tested: a 7B or larger model, few-shot prompts, other families, a second corpus.

### Outcome (experiments/llm_code.py for 0.5B in 9 min and 3B in 45 min on 4 CPU cores; experiments/h3_llm_width.py; experiments/h3_llm-Qwen2.5-{0.5B,3B}-Instruct.txt; 600 draws per cell)

• Agreement / sensitivity / specificity / share coded negative (human share 0.327): 0.5B 0.668 / 0.031 / 0.977 / 0.025; 1.5B 0.661 / 0.439 / 0.769 / 0.299; 3B 0.689 / 0.072 / 0.989 / 0.031.
• Tuned PPI++ + HC3 + t half-width ratio to human-only, mean over coefficients, n = 60 / 100 / 200 / 400: 0.5B 0.963 / 0.982 / 0.991 / 0.996; 1.5B 0.942 / 0.960 / 0.971 / 0.978; 3B 0.955 / 0.974 / 0.986 / 0.993. Tuned PPI++ coverage (mean): 0.5B 93.8 / 93.3 / 93.9 / 94.7; 3B 93.4 / 93.1 / 93.9 / 94.7.
• (1) refuted. Agreement is not monotone in size (0.668, 0.661, 0.689).
• (2) refuted. The n = 400 ratio is 0.996, 0.978, 0.993: the 1.5B model is the narrowest, not the 3B.
• (3) held. The 3B ratio at n = 400 is 0.993, nowhere near 0.90.
• (4) held. Mean tuned PPI++ coverage at n >= 100 is at least 93.1% for both models (the lowest cell, 3B at n = 100).
• Reading: size within this family did not drive the gain; the threshold did. At the fixed rule score > 0 the 0.5B and 3B models say "negative" for only 2.5% and 3.1% of tweets (truth 32.7%), so the binary code carries almost no information (sensitivity 0.03 to 0.07) and agreement is just the base rate plus a little. The 1.5B model happened to be near-calibrated at 0, which is what bought its 2.2% gain. So in the zero-shot, no-tuning design the efficiency gain is set by where the Yes/No score is cut, not by parameters. This was not tested: a threshold or a continuous score (e.g. the cached scores binned, with the cut chosen on a held-out human-labelled split) would very likely help the 0.5B and 3B models; that is the next step and it spends human labels, so it needs its own audit-budget accounting.
• Limits: one family, one prompt, one corpus; threshold fixed at 0 by design; a 7B model not run.

2026-10-26 · Milestone 7 (part): paper draft 1 folds in regression, small-audit and real-LLM results

• No experiment this step. paper/paper.md was still draft 0 (milestone 6f): it said no real LLM had coded anything and had no regression or small-audit section. Added Section 7 (6g to 6n and milestone 5), updated the abstract, limits and status table; all numbers are copied from the entries below, not re-run. cite_check.py on references.bib: all checked ids ok, one host refused (not a verdict).
• Still open for the paper: read Gligorić et al. and Fisch et al. in full (Section 2 says so), a larger or multilingual LLM, a second corpus for Section 7, the venue's LaTeX template, owner submission.

2026-10-26 · Milestone 5: does a real local LLM coder narrow the valid PPI++ interval? (planned before running)

• Why: every efficiency result so far (6j to 6n) used stand-in coders (agreement 0.68 to 0.74) and ended "whether a real LLM changes that is milestone 5". This session, for the first time, Hugging Face downloads and PyPI worked through the proxy (earlier entries: HF 403, no API key), so a local model can run. experiments/llm_code.py codes a seeded 6,000-item subsample of the NaijaSenti test part (same 30/70 split and truth source as 6m/6n) zero-shot with Qwen2.5-1.5B-Instruct (CPU, bf16): score = logit(Yes) - logit(No) for "Is the sentiment of this tweet negative?", code = score > 0, no threshold tuned on human labels. experiments/h3_llm_width.py then runs the 6n design (human-only HC3 + t vs tuned PPI++ HC3 + t, n = 60, 100, 200, 400, 600 draws, fresh seed 20261031) on that corpus for the LLM and for the three stand-ins on the same items, so the coders are compared on identical data.
• Predictions (before looking at any score): (1) The zero-shot 1.5B model's agreement with the human label is below nb-in-language's 0.743 on the same items (it was never shown Hausa, Igbo or Yoruba training data in any quantity; Pidgin is near English). (2) Tuned PPI++ + HC3 + t half-width ratio to human-only at n = 400 is at least 0.96 for the LLM, i.e. no better than the best stand-in (0.961 in 6n). (3) Tuned PPI++ coverage is within 93 to 96.5% in the mean over coefficients at every n >= 100.
• Refutes: (1) LLM agreement of 0.743 or more; (2) a ratio below 0.96 at n = 400 (an LLM gain beyond the stand-ins); (3) any n >= 100 mean outside 93 to 96.5%.
• Not tested: larger models (a 7B model is out of reach on this CPU in a session), few-shot prompts, prompt variants (one prompt, fixed before the run; no prompt search on human labels), a second corpus, H3's break-even audit size (this step compares widths at fixed n only).

### Outcome (experiments/llm_code.py, 6,000 items in about 21 min on 4 CPU cores; experiments/h3_llm_width.py, experiments/h3_<coder>.txt; 600 draws per cell, numpy)

• Corpus: 6,000 NaijaSenti test items (human negative share 0.327). Agreement with the human label / sensitivity / specificity: Qwen2.5-1.5B-Instruct zero-shot 0.661 / 0.439 / 0.769; nb-in-language 0.735 / 0.575 / 0.813; nb-english 0.678 / 0.145 / 0.937; lexicon-en 0.680 / 0.038 / 0.992. The LLM is the worst on accuracy, and its errors are of a different kind (it over-calls negative at 29.9% against a true 32.7%, sensitivity high for a weak coder, specificity low) where the stand-ins under-call.
• Mean over five coefficients of tuned PPI++ + HC3 + t half-width ratio to human-only HC3 + t (n = 60 / 100 / 200 / 400): LLM 0.942 / 0.960 / 0.971 / 0.978; nb-in-language 0.918 / 0.941 / 0.956 / 0.966; nb-english 0.958 / 0.976 / 0.988 / 0.994; lexicon-en 0.961 / 0.980 / 0.990 / 0.996. Coverage, human-only vs tuned PPI++ (mean): 94.1 / 93.6 / 94.4 / 94.6 vs LLM 93.3 / 93.2 / 94.3 / 94.8; the other coders are within 0.2 points of the LLM row at n >= 200.
• (1) held. Agreement 0.661 is below 0.743 (and below every stand-in).
• (2) held. The ratio at n = 400 is 0.978, above the 0.96 bar: the zero-shot 1.5B model narrows the interval by 2.2%, more than the two weakest stand-ins (0.6 to 0.4%) and less than the in-language naive Bayes (3.4%). Width gain tracks the coder's information about the label (here nb-in-language, which saw labelled Nigerian text), not agreement alone: the LLM has the lowest agreement but a bigger gain than two coders with 0.68 agreement, because its errors are less one-sided.
• (3) held, narrowly. Mean PPI++ coverage is 93.2 to 94.9% for every coder at n >= 100 (bar 93 to 96.5). The lowest cell is 93.2% (LLM and nb-in-language at n = 100, nb-english 93.3).
• Reading: a real but small zero-shot instruction model does not change the picture of 6n: with valid intervals the PPI++ width gain at audits of 100 to 400 is 2 to 6%, nowhere near the 20% in H3's definition, so H3's "at least 3 times smaller audit for a 20% gain" cannot be assessed (no coder reaches a 20% gain at any size here). That is a statement about a 1.5B model on Hausa, Igbo and Yoruba, which it handles badly; it is not evidence about frontier models, which this session could not reach.
• Limits: one model, one prompt (fixed before the run, no search), zero-shot, one corpus and outcome, a 6,000-item subsample (truth is the subsample's OLS), 600 draws (coverage noise about 0.9 points per cell), HC3 leverage heuristic as in 6m. The score cache is experiments/llm_scores_Qwen2.5-1.5B-Instruct.tsv (index into the seeded test part, score).
• Next: a larger or multilingual-trained model (Aya, a 7B model on a GPU or via the owner's API key) and few-shot prompting on the same cache design; H3's break-even n once a coder reaches a 20% gain; a second corpus.

2026-10-26 · Milestone 6n: with valid intervals, is PPI++ narrower than human-only HC3 + t? (planned before running)

• Why: 6m ended with coverage near nominal for tuned and cross-fit PPI++ with an HC3-type influence, but coverage is half the question: if the corrected intervals are no narrower than human-only HC3 + t, PPI++ has no use at small audits. experiments/h6n_width.py records, on the same audit, the mean ratio of each PPI++ interval's half-width (t quantile x se) to human-only HC3 + t's, per coefficient, at n = 60, 100, 200, 400 (the 6m estimators unchanged), 600 draws per cell, fresh seed 20261029, with coverage as a by-product. Coders: lexicon-en, nb-english, nb-in-language (stand-ins, not LLMs); same corpus and truth as 6m.
• Predictions (before running): (1) Tuned PPI++ + HC3 + t is narrower than human-only HC3 + t in the mean over coefficients at n = 400 for every coder (ratio below 1.00), by at most 8% (agreement of the stand-ins with the human label is modest, and 6j found 6 to 8% se gains at n = 400 on the unleveraged scale). (2) At n = 60 the ratio is at least 0.97 for every coder: no gain at small audits, since lam must be chosen from few rows. (3) Cross-fit with t(n/2 - 5) is wider than human-only at n = 60 and 100 for every coder (ratio above 1.00), because of the smaller df. (4) The ordering of coders by gain follows their agreement with the human label (best coder narrowest) at n = 400.
• Refutes: (1) a mean ratio of 1.00 or more for any coder at n = 400, or a gain over 8%; (2) a ratio under 0.97 for any coder at n = 60; (3) cross-fit narrower than human-only at n = 60 or 100 for any coder; (4) a different ordering.
• Not tested: a real LLM coder, a second corpus, a derived or bootstrapped leverage term.

### Outcome (experiments/h6n_width.py, about 15 min, 3 processes; experiments/h6n_<coder>.txt; 600 draws per cell, numpy)

• Mean over the five coefficients, of (coverage human-only HC3+t / tuned PPI++ HC3+t / cross-fit HC3+t(n/2-5); half-width ratio tuned / human-only; half-width ratio cross-fit / human-only), on the same audits. Stand-in agreement with the human label: lexicon-en 0.678, nb-english 0.680, nb-in-language 0.743.
• n = 400: lexicon-en 95.6 / 95.3 / 95.7%, ratios 0.995 / 1.027; nb-english 95.6 / 95.4 / 95.9%, 0.994 / 1.028; nb-in-language 95.6 / 94.7 / 95.2%, 0.961 / 0.994 (cross-fit 0.994).
• n = 200: tuned ratio 0.990, 0.989, 0.953; cross-fit 1.052, 1.051, 1.017. n = 100: tuned 0.979, 0.977, 0.932; cross-fit 1.105, 1.106, 1.056. n = 60: tuned 0.957, 0.956, 0.908 (3 of 600 draws dropped, singular design).
• Cross-fit at n = 60 is not usable from this run: its width ratio is nan (at least one draw per cell gave a non-finite se, which the script does not trap) and its coverage there (95.7 / 95.6 / 90.8%) counts those draws as misses or hits depending on the cell, so I do not report it. Fix before reuse: drop non-finite draws and count them.
• (1) held. Tuned PPI++ with HC3 + t is narrower than human-only at n = 400 for every coder (0.995, 0.994, 0.961), and by far less than 8%. For the two weak coders the gain is 0.5%: nothing.
• (2) refuted. At n = 60 the ratio is 0.957, 0.956 and 0.908, under the 0.97 bar. The tuned interval is narrower at small n than at large n, and its coverage there is 93.5 / 93.9 / 92.7% against human-only 94.0%. Narrow while under-covering by 0.5 to 1.3 points means the width gain is partly an optimistic se from picking lam on the audit (min over a grid), not free efficiency. I read the n = 60 ratio as a warning, not a gain.
• (3) held at n = 100 (cross-fit 1.105, 1.106, 1.056, all wider); not evaluated at n = 60 (see above).
• (4) held. The coder with the highest agreement (nb-in-language, 0.743) is the narrowest at every n; the other two are indistinguishable.
• Reading: with a valid interval, PPI++ with a 0.68-agreement coder buys under 1% width at n = 400 and the cross-fit option costs 3% (n = 400) to 10% (n = 100). Only the 0.743-agreement coder gives a real gain (3.9% narrower at n = 400, 4.7% at n = 200), and its tuned coverage (94.7%) is the lowest of the three there. So on this corpus the validity repair of 6m works, and the efficiency gain that motivates PPI is small with the coders we have; whether a real LLM coder changes that is milestone 5.
• Limits: one corpus and outcome; stand-in coders; 600 draws (width ratios are tight, coverage noise about 0.9 points per cell); the HC3 leverage for the blended equation is heuristic; the tuned se is a minimum over a grid so its small-n narrowness is partly selection; cross-fit at n = 60 invalid (above).
• Next: milestone 5 with a real LLM coder (the only thing that can move the efficiency gain); trap non-finite se in h6n_width.py and rerun n = 60 for cross-fit.

### Addendum to 6n: cross-fit at n = 60 with non-finite draws dropped (experiments/h6n_nanfix.py, experiments/h6n_nanfix_<coder>.txt; independent rerun, same seed and design, ratios only)

• The nan above came from draws where a half-audit row has leverage 1 (a degenerate group), so the HC3 se is non-finite. The rerun drops those draws with the singular-design ones (10 to 12 of 600 at n = 60, none at n >= 100) and the n >= 100 ratios reproduce the entry above to three digits.
• Cross-fit + HC3 + t(n - 5) half-width ratio to human-only at n = 60: lexicon-en 1.147, nb-english 1.153, nb-in-language 1.089 (narrower in 23, 23, 34% of draws); with t(n/2 - 5): 1.178, 1.184, 1.119. So prediction (3) also holds at n = 60: cross-fit is 9 to 18% wider than human-only there.
• Process note: this rerun was launched in parallel with the run that landed 6n (PR #10550); the duplicate was found at merge time.

2026-10-26 · Milestone 6m: does an HC3-type leverage correction on the PPI++ influence close the small-audit gap? (planned before running)

• Why: 6l found human-only HC3 + t near nominal at n >= 100 (94.4 to 94.9%) and 94.0% at n = 60, while tuned and cross-fit PPI++ with t sit 3 to 4 points lower at n = 60, but that compares a leverage-corrected interval with an uncorrected one. experiments/h6m_ppi_hc3.py divides each audit row's PPI++ influence by 1 - h_i, with h_i = x_i' (n M_lam)^-1 x_i (M_lam the blended design matrix of the estimating equation; lam = 0 reproduces human-only HC3), for in-sample tuned PPI++ and two-fold cross-fit (the correction is applied both when choosing lam and in the reported se). Same corpus, coders, truth and sizes as 6l, 1,500 draws per cell, fresh seed 20261028. Cross-fit is scored with t on n - 5 df and, conservatively, on n/2 - 5 df.
• Predictions (before running): (1) Tuned PPI++ with HC3 + t covers at least 2 points more than tuned PPI++ with t at n = 60 (mean over coefficients, every coder). (2) It is within 1.5 points of human-only HC3 + t at n = 60 and within 1 point at n >= 100. (3) Cross-fit with HC3 + t(n/2 - 5) is at least as high as with t(n - 5) and is within 1.5 points of 95% at n >= 100. (4) The correction costs width: mean se ratio against 6l's uncorrected se is above 1 (not measured here; reported only if added), so this step reports coverage only.
• Refutes: (1) a gain under 2 points for any coder at n = 60; (2) a gap over 1.5 points at n = 60 or over 1 point at n >= 100 in the mean; (3) cross-fit HC3 + t(n/2 - 5) below 93.5% at any n >= 100 mean, or below the t(n - 5) version; (4) is not tested.
• Not tested: a real LLM coder, a bootstrap interval, a second corpus, interval width (so no claim that PPI++ is worth using at small n, only whether its interval is valid).

### Outcome (experiments/h6m_ppi_hc3.py, about 25 min, 3 processes; experiments/h6m_<coder>.txt; 1,500 draws per cell, numpy)

• Mean coverage over the five coefficients, n = 60 / 100 / 200. Human-only HC3 + t: 93.8 / 94.3 / 94.3% (same audits for all coders). Tuned PPI++ + t without leverage (6l): lexicon-en 90.7 / 92.4 / 93.3, nb-english 91.4 / 92.8 / 93.5, nb-in-language 90.1 / 92.0 / 93.3. Tuned PPI++ with HC3 + t: 93.1 / 94.0 / 93.8, 93.5 / 94.1 / 94.1, 92.4 / 93.3 / 94.0. Cross-fit with HC3 + t(n - 5): 94.3 / 95.2 / 94.9, 94.4 / 95.5 / 94.9, 94.1 / 94.7 / 94.9. With t(n/2 - 5): 94.8 / 95.5 / 95.1, 94.9 / 95.8 / 95.1, 94.5 / 95.1 / 95.0. 16 of 1,500 draws at n = 60 and 1 at n = 100 dropped (singular design), as in 6l.
• (1) held. The leverage correction adds 2.4, 2.1 and 2.3 points for tuned PPI++ at n = 60 (lexicon-en, nb-english, nb-in-language), against a 2-point bar.
• (2) held, with no margin at one cell. Tuned HC3 + t is 0.7, 0.3 and 1.4 points below human-only HC3 + t at n = 60 (bar 1.5); at n >= 100 the gap is 0.2 to 0.5 points except nb-in-language at n = 100, which is 1.0 (rounded to the printed digits; at the bar, not clearly inside it).
• (3) held. The t(n/2 - 5) cross-fit is 0.2 to 0.6 points above the t(n - 5) version everywhere, and both are within 93.5 to 96.5% in every n >= 100 mean (94.7 to 95.8%). Cross-fit with HC3 covers at or above human-only HC3 + t in every cell, by up to 1.2 points at n = 100, so it is conservative here. Why was not tested; the cross-fit se is built from two half-audits with their own leverage, so a wider interval is plausible, but width was not measured in this step.
• (4) not tested (no se ratio recorded).
• Reading: the 3 to 4 point shortfall of PPI++ at small audits in 6k and 6l was mostly the missing leverage correction, not selection of lam. With an HC3-type influence all three estimators are within about 1.5 points of nominal for n >= 60 on this corpus; the cross-fit estimator with HC3 is the safest of them (94 to 96%). What this does not show is that PPI++ is worth using at n < 400: coverage is only half the question and the interval widths against human-only HC3 were not compared here.
• Limits: one corpus and outcome; stand-in coders, not LLMs; the HC3 leverage for the blended equation is a heuristic (the audit part of M_lam), not derived, and the t df choices are not derived; no multiplicity correction; 1,500 draws (about 0.3 points of noise in a five-coefficient mean, more per coefficient); 1% of n = 60 audits dropped.
• Next: width of the HC3 intervals against human-only HC3 at n = 60 to 400 (is there still an efficiency gain once the intervals are valid?); a derivation or bootstrap check of the leverage term; milestone 5 with a real coder.

2026-10-25 · Milestone 6l: does a t quantile (and HC3 for human-only) restore coverage at audits of 60 to 200? (planned before running)

• Why: 6k found human-only OLS itself under-covers with a normal quantile (mean over cells 91.6 / 92.9 / 94.0% at n = 60 / 100 / 200) and tuned PPI++ a further 0.3 to 0.7 points below; its next step was a small-sample correction. experiments/h6l_small_sample_interval.py re-runs the 6k estimators (human-only, in-sample tuned PPI++, two-fold cross-fit) on the same corpus, coders and truth, 1,500 draws per cell (seed 20261027, fresh audits, so not a copy of 6k), with the Student-t 97.5% quantile on n - 5 df in place of 1.96 for all three, and an HC3 leverage-corrected sandwich for human-only. The t quantile is a Cornish-Fisher expansion (no scipy in this image). The cross-fit estimator averages two half-audit estimates, so n - 5 df is a rough choice for it, not a derived one.
• Predictions (before running): (1) t raises human-only mean coverage at n = 60 from the normal's by at least 1 point. (2) Human-only HC3 + t is within 1.5 points of 95% in the mean over coefficients at every n >= 100 for every coder. (3) Tuned PPI++ with t still covers below human-only with t at n = 60 by at least 0.3 points in the mean (the selection cost is not a quantile problem). (4) No estimator reaches 94% mean coverage at n = 60 for nb-in-language.
• Refutes: (1) a gain under 1 point; (2) any coder-n mean outside 93.5 to 96.5%; (3) tuned-t at or above human-t at n = 60; (4) any estimator at 94% or more there.
• Not tested: a real LLM coder, a bootstrap interval, a ridge or stratified design for the singular half-audits, a second corpus.

### Outcome (experiments/h6l_small_sample_interval.py, about 20 min, 3 processes; experiments/h6l_<coder>.txt; 1,500 draws per cell, numpy)

• Mean coverage over the five coefficients (human-only columns are identical across coders: same audits). Human-only normal / t / HC3+t: 91.1 / 91.7 / 94.0% at n = 60, 92.5 / 92.8 / 94.4% at n = 100, 94.2 / 94.4 / 94.9% at n = 200. 17 of 1,500 draws at n = 60 dropped (singular design), 1 at n = 100.
• Tuned PPI++ normal / t, by coder at n = 60: lexicon-en 90.5 / 91.2, nb-english 90.7 / 91.3, nb-in-language 90.2 / 90.8%; at n = 200: 93.9 to 94.2 / 94.0 to 94.4%. Cross-fit normal / t at n = 60: 89.6 / 90.2, 89.7 / 90.2, 89.9 / 90.6%.
• (1) refuted. The t quantile (2.004 against 1.960 at n = 60) adds 0.6 points for human-only, under the 1-point bar. The normal quantile is not the main problem.
• (2) held. Human-only HC3 + t is 94.4 to 94.9% at n >= 100 (bar 93.5 to 96.5). The leverage correction does the work: +2.3 points at n = 60 over t alone. The intercept (yor, the smallest group) stays lowest at 91.0% at n = 60.
• (3) held. Tuned PPI++ with t is below human-only with t by 0.5, 0.4 and 0.9 points at n = 60 (lexicon-en, nb-english, nb-in-language). Against human-only HC3 + t the gap is about 3 points, but that compares a leverage-corrected interval with an uncorrected one.
• (4) refuted, at the line. Human-only HC3 + t reaches 94.0% for nb-in-language at n = 60; no PPI estimator comes within 3 points of it (best 90.8%).
• Reading: the shortfall at n <= 100 is a heteroskedastic-sandwich leverage problem more than a quantile problem, and the PPI++ variants here have no equivalent correction, so at n = 60 to 100 they under-cover by 3 to 4 points and the human-only HC3+t interval is the only one near nominal. Whether PPI++ is worth using at audits this small is therefore not answered: its se has no leverage correction yet, so its comparison with human-only is unfair to PPI++. The 6j/6k recommendation (report in-sample tuned PPI++ at n >= 400 only) stands.
• Limits: one corpus and outcome; stand-in coders; the t df (n - 5) for the cross-fit is not derived; HC3 applied to human-only only; 1,500 draws (binomial sd about 0.6 points per coefficient cell, about 0.3 for the five-coefficient mean treating coefficients as independent, which they are not); the t quantile is a series, accurate to about 1e-4.
• Next: a leverage-corrected (HC3-type) influence for the tuned and cross-fit PPI++ estimators, then repeat at n = 60 to 200; milestone 5 with a real coder.

2026-10-25 · Milestone 6k: do selection effects show at small audits (n = 60, 100, 200)? (planned before running)

• Why: 6i's null (cross-fitting buys no measurable coverage at n = 400, since bounded by the paired 6j) named smaller audits, where choosing lam on 21 grid points from fewer rows should overfit more, as the untested case. experiments/h6k_small_audit.py runs human-only, in-sample tuned PPI++ and two-fold cross-fit on the same audits (paired), 3,000 draws per cell, n = 60, 100, 200, same corpus, three stand-in coders and covariates as 6g to 6i. Paired coverage differences get a normal interval from the per-draw differences, which is tighter than comparing two coverages with their own binomial noise.
• Predictions (before running): (1) in-sample tuned coverage falls below human-only's at n = 60 by at least 1 point in some cell (paired interval excluding 0) and the loss shrinks with n. (2) Cross-fit coverage is within 1.5 points of nominal in all cells at n >= 100. (3) At n = 60 the cross-fit se is not smaller than in-sample's (lam from 30 rows is noisy), and cross-fit coverage is at least as high as in-sample in the mean over cells.
• Refutes: (1) no cell with an in-sample loss over 1 point at n = 60; (2) any cell at n >= 100 outside 93.5 to 96.5%; (3) cross-fit mean coverage below in-sample's at n = 60.
• Not tested: a real LLM coder, more folds, stratified audits, other corpora.

### Outcome (experiments/h6k_small_audit.py, about 80 min; experiments/h6k_small_audit.txt; 3,000 paired draws per cell, numpy)

• Draws dropped: at n = 60, 29 of 3,000 audits (1%) have a half-audit with no observation of some language, so the design matrix is singular; they are dropped for all three estimators (none at n >= 100). A real small audit would need a stratified design or a ridge, not a silent drop.
• The base problem is not tuning. Human-only OLS with a normal interval already under-covers: mean over the 15 cells 91.6% at n = 60, 92.9% at n = 100, 94.0% at n = 200 (lowest 89.3%). The normal quantile ignores the small-sample uncertainty of the se; a t or HC3-type correction would be the fix and was not tested. This also explains the absolute failures below.
• (1) held. In-sample tuned PPI++ covers less than human-only on the same audits: mean over cells 90.9 / 92.4 / 93.6% (a loss of 0.7, 0.5, 0.35 points at n = 60, 100, 200). At n = 60 five cells lose over 1 point with a paired interval excluding 0 (up to 2.6, nb-in-language pcm). The loss shrinks with n on average but not everywhere: nb-in-language at n = 200 still loses 1.1 (intercept), 1.1 (hau), 1.4 (pcm) and 0.7 (loglen) points, intervals excluding 0, which 6i's n = 400, 1,000-draw run could not resolve. So the one-point loss seen in 6h is probably real for the weakest-signal coder, and 6i's "within noise" was a power failure.
• (2) refuted. Cross-fit coverage at n >= 100 is 90.2 to 94.6% (mean 92.3% at n = 100, 93.7% at n = 200), outside the 93.5 to 96.5% bar in most n = 100 cells; human-only is outside it too, so the cause is the interval, not the estimator.
• (3) refuted. Cross-fit does not help: mean coverage 90.2 / 92.3 / 93.7% against in-sample's 90.9 / 92.4 / 93.6%. Paired differences are mostly inside +-1 point; at n = 60 in-sample is ahead by 1 to 2.3 points on lexicon-en and nb-english pcm/ibo, cross-fit ahead by about 1 point on nb-in-language n = 100 (intercept, pcm). Cross-fit se is 1 to 4% wider than in-sample's, so it buys no validity and costs width. The lam from 30 rows is no better than lam tuned in-sample.
• Reading: below n = 400, every normal-interval estimator here under-covers, tuned PPI++ by a further 0.3 to 0.7 points on average and up to 2.6 in single cells; cross-fitting does not repair it. Recommendation for the programme: report a small-sample correction (t quantile with audit df, or bootstrap) before claiming validity at n < 200, and re-test PPI++ with it. The 6h/6i default (in-sample tuned) stands at n >= 400 only.
• Limits: one corpus and outcome; stand-in coders; normal intervals without small-sample correction; se of tuned PPI++ is a minimum over a grid (optimistic); 1% of n = 60 draws dropped.
• Next: t-quantile / HC3 correction for all three estimators at n = 60 to 200; milestone 5 with a real coder.

2026-10-26 · Milestone 6j: a paired test of the selection effect that 6i could not resolve (planned before running)

• Why: 6i ended with a null it could not interpret: in-sample-tuned PPI++ and cross-fitted PPI++ both covered at about 95.15% but cells differ by less than the binomial sd (0.7 points) of 1,000 unpaired draws, so "no selection effect" was not distinguishable from "an effect of up to a point". 6i's own limits section says a paired difference over more draws is needed to bound the effect at under 0.5 points. This step does that: the same three estimators (human-only, in-sample-tuned PPI++, two-fold cross-fit) on the same audits, 4,000 paired draws per coder (experiments/h6j_paired.py, seed 20261026), same corpus, coders and n = 400 as 6g to 6i. The per-draw coverage differences are paired, so their sampling error is that of discordant draws only.
• Predictions (before running): (1) For each coder, the coverage difference in-sample minus cross-fit, pooled over the five coefficients, has a 95% interval inside [-0.5, +0.5] points. (2) In-sample minus human-only coverage is within [-0.5, +0.5] too (6h saw up to a point of loss unpaired; 6i's replicate saw none). (3) Paired se ratios reproduce 6i: cross-fit at most 1.03 of human-only in every cell, and 1 to 3% wider than in-sample tuning on average.
• Refutes: (1) or (2) if an interval's upper or lower end lies outside +-0.5 points (the effect is then not bounded below half a point at this n); (3) if any cell's cross-fit ratio exceeds 1.03.
• Not tested: a real LLM coder (milestone 5; no model host reachable and no API key in this session), other audit sizes, a second corpus.

### Outcome (experiments/h6j_paired.py, about 25 min, 3 processes; experiments/h6j_paired.txt; 4,000 paired draws per coder)

• (1) held. In-sample minus cross-fit coverage, pooled over the five coefficients: -0.02 (+-0.15), +0.03 (+-0.15) and -0.11 (+-0.16) points for lexicon-en, nb-english, nb-in-language. Cross-fitting buys no measurable coverage; the effect is bounded under 0.3 points.
• (2) refuted, narrowly, and it corrects 6i. In-sample minus human-only, pooled: -0.16 (+-0.13), -0.07 (+-0.13), -0.31 (+-0.24). The nb-in-language interval reaches -0.55, outside the -0.5 bar. Per cell, the paired loss is real in two: lexicon-en intercept -0.88 (+-0.38) and nb-in-language pcm -0.88 (+-0.41); in-sample coverage there is 93.8% and 93.7%. So tuning lam on the audit it is scored on costs up to about 0.9 points in some cells. 6i's "did not reproduce" was a power failure (unpaired sd 0.7 points), not evidence of absence.
• (3) held. Cross-fit se over human-only is 0.944 to 1.017 (bar 1.03), and 0.8 to 3.0% wider than in-sample tuning (largest on the nb-in-language intercept, 0.967 to 0.996).
• Reading: the in-sample selection cost is small (pooled under 0.35 points) but not zero; cross-fitting removes part of it (cross-fit minus human-only pooled -0.14, -0.10, -0.20; pooled in-sample vs cross-fit differences are inside noise) at a 1 to 3% width cost. Both stay within about 1 point of nominal. For a paper: report in-sample tuned PPI++ as the default and cross-fitting as the conservative option.
• Limits: same single corpus, outcome and stand-in coders; n = 400 only; coverage is against the corpus OLS, and the pooled interval treats draws as independent (coefficients share an audit, so pooling is over draw-level means); no multiplicity correction over 15 cells (two of the cells above would not survive Holm over 15).
• Next: milestone 5 with a real coder; a second corpus; a smaller audit (n = 100) where selection effects should be larger.

2026-10-25 · Milestone 3b: does H1 replicate on a second real corpus whose groups differ little in coder error? (planned before running)

• Why: every result so far except the rare-category ones rests on NaijaSenti (a different language per group, so coder error differs by group by construction). A reviewer will ask whether the pooled-PPI failure under audit-mix shift is a property of that corpus or of the estimator. GoEmotions comments grouped by the kind of subreddit are a milder case: same language, and the smoke run already shows coder agreement with the human label within 5 points across groups.
• Data and design (experiments/group_shift_reddit.py): GoEmotions parts 1 to 3 (Demszky et al. 2020), comments from 51 of the 80 largest subreddits, put by hand into four groups (sports, tv-dating, personal, video-cringe) before any coder was run; corpus = the 70% test part (6,683 comments), seeded 30/70 split. Target = a majority of raters marked at least one negative emotion (the 11 negative GoEmotions categories); true corpus share 23.5%, 21.1 to 26.2% by group. Coders (logistic regression on word presence, label threshold at the train quantile matching train prevalence, stand-ins, not LLMs): lr-pooled (trained on all train), lr-sports-only, lr-personal-only (trained on one group, so applied out of domain to the others). Audit n = 400, designs A random, B equal quota (100 each), C 70% sports (280/40/40/40), 1,000 draws per cell, same estimators as group_shift.py.
• Predictions (before running): (1) Under A every estimator covers 93.5 to 96.5%. (2) The effect is smaller than on NaijaSenti: pooled PPI under C covers at least 90% for lr-pooled, because that coder's agreement is flat across groups (0.755 to 0.769) and the group prevalences differ by at most 5 points. (3) The shift still bites for at least one out-of-domain coder: pooled PPI (lambda = 1) under B or C covers under 92% for lr-sports-only or lr-personal-only (agreement gaps up to 5 points, plus a bias of up to 1.5 points under A). (4) Stratified PPI covers 93.5 to 96.5% in every cell. (5) Stratified PPI is not narrower than stratified human-only for any coder (agreement 69 to 77%, as with the earlier stand-ins).
• Refutes: (1) any A cell outside 93.5 to 96.5%; (4) any stratified PPI cell outside; (3) all pooled PPI cells for the two domain-trained coders at 92% or more, which would say H1's failure mode needs larger group differences than a same-language corpus gives (that would narrow H1's claim, not drop it).
• Not tested: a real LLM coder, groups with a large prevalence gap, a rarer target.

### Outcome (experiments/group_shift_reddit.py, under a minute; experiments/group_shift_reddit.txt; 1,000 draws per cell)

• Corpus: 6,683 comments, true negative share 23.5% (21.1 to 26.2% by group). Coder agreement with the human label: lr-pooled 0.755 to 0.769 across groups; lr-sports-only 0.675 to 0.721; lr-personal-only 0.720 to 0.745.
• (1) held. Under A every cell covers 93.6 to 95.2%.
• (2) held. Pooled PPI with lr-pooled under C covers 95.2% (bias -0.56 points).
• (3) held, narrowly, in one cell of 6. Pooled PPI with lr-sports-only under C covers 91.4% with bias +1.74 points; every other pooled-PPI cell for the two domain-trained coders is 94.7 to 95.6%. Pooled human-only under C is 93.7% (bias -0.72 from the audit's mix), PPI++ 93.6 to 95.6%.
• (4) held at the bar. Stratified PPI covers 93.5 to 95.0% in all nine cells (two cells at 93.5 and 93.6%). Stratified human-only covers 93.7 to 94.3%.
• (5) held. Stratified PPI is wider than stratified human-only in all nine cells (for example 9.2 vs 8.1 points under A with lr-pooled; 13.4 to 15.4 vs 11.8 under C).
• Reading: H1's failure mode is real here but small. When coder error is nearly flat across groups (agreement within 5 points) and the groups' prevalences are within 5 points, the pooled PPI interval is mostly valid, with one undercovering cell (91.4%). The 67 to 69% of NaijaSenti needs groups whose coder error differs by tens of points. So the claim should be stated as "undercoverage grows with between-group differences in coder error", not as a property of PPI under any mix; stratification is safe in both corpora but costs width here (about 15 to 30% wider than stratified hand-coding in A and C).
• Limits: one corpus, groups I defined by hand from subreddit names, stand-in coders, a prevalence gap of at most 5 points, a small corpus (the audit is 6% of it), one tested bar missed by no more than a hair (93.5%). Predictions were written before the run.
• Next: milestone 5 (no model reachable; no API key); a corpus with a large between-group prevalence gap.

2026-10-25 · Milestone 6i: does cross-fitting the PPI++ weight remove the coverage cost of tuning it on the audit? (planned before running)

• Why: 6h's next item. 6h chose lam per coefficient on the same audit that is then used for the estimate; coverage was 93.3 to 95.6% (up to about one point under human-only), which that run could not separate from noise or from the selection effect. Two-fold cross-fitting removes the selection effect by construction: lam for each half of the audit is chosen on the other half (llmci.regression.ppi_ols_crossfit), the two half estimates are averaged and the se is sqrt(se_A^2 + se_B^2)/2 (the half-audit se is sqrt(2) times the full one, so the average has about the full width; the cost is a lam estimated from n/2 = 200 rows). Same corpus, coders, n = 400, 1,000 draws and audit draws as 6g/6h (experiments/h6i_crossfit.py; the split permutation is drawn from the same generator after each audit, so the audits after the first differ from 6h's and the in-sample column below is a fresh replicate of 6h, not a copy).
• Predictions (before running): (1) cross-fit coverage is 93 to 96.5% in all 15 cells. (2) Mean over the 15 cells of coverage is within 0.3 points of 6h's in-sample figure (I expect no visible gain: 6h's loss was at most 1 point and the binomial sd per cell is 0.7). (3) Mean se over human-only is at most 1.03 in every cell. (4) nb-in-language keeps at least 3 of 5 coefficients below ratio 0.99 (the cost of estimating lam on 200 rows may erase part of 6h's 1 to 7% gain; I expect the gain to shrink, not vanish).
• Refutes: any cell under 92% coverage; any se ratio over 1.03; fewer than 3 nb-in-language coefficients under 0.99.
• Not tested: a real LLM coder (milestone 5), more folds, a cross-fitted lam that also uses the corpus (unlabelled rows), stratified audits.

### Outcome (experiments/h6i_crossfit.py, about 4 min; experiments/h6i_crossfit.txt; 1,000 draws per cell, numpy)

• (1) held. Cross-fit coverage is 94.4 to 95.9% in all 15 cells (human-only 95.0 to 95.8%).
• (2) held, and the premise of the milestone weakens. Mean coverage over the 15 cells: cross-fit 95.15%, in-sample tuned 95.15% (fresh replicate; cells 94.3 to 95.8%), human-only 95.30%. The one-point loss seen in 6h (lowest 93.3%) did not reproduce in the replicate, so it was within the noise of 1,000 draws (binomial sd 0.7 points); I cannot show that in-sample tuning costs coverage at this n, and cross-fitting therefore buys no measurable coverage here.
• (3) held. Cross-fit se over human-only is 0.943 to 1.021 (the worst cell is the lexicon-en intercept, 1.021; bar 1.03).
• (4) held, narrowly. nb-in-language: ratios 1.000 intercept, 0.943 hau, 0.977 ibo, 1.003 pcm, 0.956 loglen_z, i.e. 3 of 5 under 0.99 (in-sample tuning: 0.969, 0.935, 0.965, 0.988, 0.943, so 5 of 5). Estimating lam on 200 rows gives back about 2 points of the gain (hau, loglen_z) and all of it on the intercept and pcm.
• Reading: cross-fitting is not needed for validity at n = 400 and costs 1 to 3% in width relative to in-sample tuning; keep the in-sample tuned PPI++ of 6h as the default, and treat cross-fitting as the option for smaller audits or a finer lam grid, where selection effects are larger (untested). The honest finding is a null: the selection effect is below what 1,000 draws can resolve.
• Limits: the same single corpus, outcome and stand-in coders; coverage cells differ by less than the binomial noise, so the comparison between estimators is underpowered by design (a paired difference test over more draws would be needed to bound the effect at under 0.5 points); two folds only.
• Next: milestone 5 with a real coder (no model reachable in this session: no API key set); a second corpus.

2026-10-25 · Milestone 6h: does power-tuned PPI++ for regression at least match human-only OLS? (planned before running)

• Why: 6g's last line. PPI with lam = 1 was wider than human-only OLS on all 15 coefficient-by-coder cells (se ratio 1.08 to 1.41) because the three stand-in coders agree with the human label only 68 to 74%. Angelopoulos et al. (PPI++, 2023) tune lam to minimise variance; lam = 0 recovers human-only, so a tuned estimator should never be much wider. The question is whether the tuned lam is estimated well enough from n = 400 to keep coverage, and whether any cell gets narrower.
• Construction (experiments/h6h_ppipp_regression.py, llmci.regression.ppi_ols_tuned): per coefficient j, g = (M^-1 x)_j, lam_j = Cov_audit(g y, g f) / Var_audit(g f), estimated in-sample on the audit, then the coefficient-j estimate and design-based se of 6g with that lam (variance of g (y - lam_j f) with the finite-population correction). Two variants: unclipped, and clipped to [0, 1]. (Superseded by the correction below.) Same corpus, coders, n = 400, 1,000 draws, seed as 6g (the draws are paired with 6g's).
• Predictions (before running): (1) coverage of the corpus OLS coefficient is 93 to 96% in every cell for both variants (estimated lam costs a little; no under-coverage below 92%). (2) Mean se of tuned PPI++ is at most 1.03 times human-only's in every cell. (3) For at least one coefficient of nb-in-language the mean se is below human-only's (ratio under 0.98); for lexicon-en none is. (3) is the one I expect to fail.
• Refutes: any cell with coverage under 92%; any se ratio above 1.03; no nb-in-language coefficient under 0.98.
• Not tested: a real LLM coder, cross-fitted lam, stratified audits, other covariate sets.
• Correction found by the first run (disclosed). My first construction kept the full-corpus matrix M in the estimating equation and only varied lam. That estimator at lam = 0 is not human-only OLS (it uses the corpus's M with the audit's x y), and it came out wider than human-only on all 15 cells (se ratio 0.99 to 1.40, tuned lam 0.3 to 0.86 chasing a control-variate effect). The general PPI++ equation blends the matrix too: M_lam = lam M_all + (1 - lam) M_audit, which makes lam = 0 exactly audit OLS (checked in test_llmci.py). llmci.regression.ppi_ols_lam / ppi_ols_tuned now implement that; lam is chosen per coefficient on a 21-point grid in [0, 1] by smallest estimated se. The predictions were written before either run; the numbers below are the corrected estimator's (the clipped variant is gone, the grid is the clip). Note also that 6g's lam = 1 ppi_ols is the special case with M_all only.

### Outcome (experiments/h6h_ppipp_regression.txt; python3.13 experiments/h6h_ppipp_regression.py, about 2 minutes; same corpus, coders, n = 400 audit, 1,000 draws and seed as 6g)

• (1) held. Coverage of the corpus OLS coefficient is 93.3 to 95.6% in all 15 cells (human-only 93.9 to 95.6%, PPI lam = 1 93.6 to 95.7%); the lowest is 0.933 (nb-in-language pcm, lexicon-en pcm 0.936). Choosing lam on the audit it is then evaluated on costs at most about one point against human-only, inside 2 binomial sd of nominal for 1,000 draws.
• (2) held. Mean se over human-only is 0.991 to 0.999 for lexicon-en and nb-english on every coefficient (mean tuned lam 0.05 to 0.17: the tuner nearly switches the coder off), so tuning removes the 1.08 to 1.41 penalty of 6g.
• (3) held. nb-in-language, the 74%-agreement coder, is narrower than human-only on all five coefficients: ratio 0.967 intercept, 0.935 hau, 0.964 ibo, 0.988 pcm, 0.945 loglen_z (mean tuned lam 0.17 to 0.40). lexicon-en: none under 0.98 (best 0.991).
• Reading: for regression with a weak coder the right default is the tuned PPI++ equation, not lam = 1: it is valid at about the nominal level, never wider than human-only to within a rounding error, and 1 to 7% narrower when the coder reaches roughly 74% agreement. That is a small gain; the width benefit in H5's regression half still waits on a coder far better than these, i.e. milestone 5.
• Limits: se columns are mean estimated se (tuned se is a minimum over 21 lam values and so slightly optimistic; coverage is the honest check); one outcome and covariate set; simple random audit; stand-in coders; truth is the corpus's own OLS; lam grid not refined; sampling variance of lam is not propagated.
• Next: milestone 5 with a real coder; propagate the lam-selection variance (sample splitting) to see whether the one-point coverage loss disappears.

2026-10-25 · Milestone 6g: regression coefficients from LLM codes plus an audit (H5 second half; planned before running)

• Question: when the coded variable is the outcome of a regression (here: negative sentiment on language dummies and log tweet length, NaijaSenti hau/ibo/pcm/yor), does PPI for OLS (Angelopoulos et al. 2023, the linear estimating equation: theta = M^-1 [mean_all x f + mean_audit x (y - f)]) give intervals that cover the corpus's own human-label OLS coefficients, while the naive coder-label regression does not?
• Truth: OLS of the human label on the same covariates over the 70% test corpus (finite population). Audit n = 400 random, 1,000 draws, seed 20261025, the three coders of group_shift.py.
• Predictions (before running): (1) naive coverage of the language-gap coefficients well below 95% for the weak coders, because their error rate differs by language; (2) PPI coverage 93 to 97% on every coefficient; (3) PPI is narrower than human-only OLS for nb-in-language, not for lexicon-en. (3) is the one I expect may fail for the lexicon coder (low agreement).
• Refutes: any coefficient with PPI coverage under 92% over 1,000 draws (binomial sd about 0.7 points), or PPI wider than human-only for the best coder.
• Variance used: design-based, M^-1 Cov(x(y-f)) M^-1 (1 - n/N)/n, since the corpus (all x, f) is the fixed population.
• Outcome (experiments/h6g_regression.txt; python3.13 experiments/h6g_regression.py, numpy): (1) supported and then some: naive coder-label regression covers 0.000 on all 15 coefficient-by-coder cells, with biases up to +0.52 (pcm gap, in-language coder) and sign flips between coders (hau gap -0.11 for lexicon-en, +0.26 for nb-in-language), so no coder is safe to regress on directly. (2) supported: PPI covers 93.6 to 95.7% on all 15 cells (human-only 93.9 to 95.6%). (3) refuted for every coder: PPI with lam = 1 is wider than human-only OLS on all coefficients (se ratio 1.08 to 1.41 for the intercept, hau, ibo, pcm and loglen_z; nb-in-language is the closest, 1.08 to 1.17). The coders agree with the human label only 68 to 74%, so the rectifier carries more variance than the coder removes, as in H1.
• Consequence: the coefficient result mirrors the prevalence result: validity is free, narrower intervals need a coder better than these. A per-coefficient power-tuned lam (PPI++ for regression) would at worst match human-only and is the next step; with a real LLM (milestone 5) it decides whether H5's regression half has a width gain at all.
• Limits: one outcome, one covariate set, simple random audit, 1,000 draws per cell, stand-in coders, finite-population truth (the corpus's own OLS), not a superpopulation coefficient.

2026-10-25 · Milestone 7a: first draft of the paper (paper/paper.md, paper/references.bib); no new experiment

• Why: the programme had six milestones of results and no paper/. A draft written against the results exposes what a reviewer would ask for. Every number in it is copied from a LOG.md entry or an experiments/*.txt file, and the draft states that no real LLM has coded anything.
• What the draft showed is missing (the useful output): (1) the headline needs H3, the real coder, and the abstract currently ends on "not yet tested"; (2) the proposal asked for three corpora and there are two, each carrying a different claim (NaijaSenti for group shift and sizing, GoEmotions for rare categories), so no single corpus shows group shift, rare category and sizing together; (3) Gligorić et al. and Fisch et al. are cited from their abstracts and still need reading in full before the gap is claimed; (4) the floor multiplier 5 is in-sample.
• Citations: cite_check.py on references.bib reached no host (Crossref 403 through the proxy; the script does not parse eprint= lines, so it checked 0 arXiv ids). The arXiv ids are the ones machine-checked in milestone 1a; the three DOIs are unchecked this session.
• Next: milestone 5 as soon as a model is reachable (HF and the Anthropic API were not); otherwise a second corpus with a rare category and group structure.

2026-10-25 · Milestone 6f: the oracle audit size of 6a to 6e had a bug; the "unexplained" 6e gap was it (found while diagnosing; correction, not a new hypothesis)

• Why: 6e's open item, why the sequential rule stops 11 to 13% under the plug-in oracle. Prediction before looking: the oracle's finite-population term or the allocation (the cap n_h <= N_h) differs from what the stopping rule realises. Instead, computing the analytic half-width of the oracle's own Neyman allocation from the population S_h^2 gave 0.925 points at the "1.0-point" oracle n = 7,833 (lexicon-en), not 1.0.
• Cause: neyman_total (experiments/h5_plugin.py) paired s2 (variances) with a a b * b / N, squaring a variance a second time (S^4 for S^2) in the finite-population term sum W_h^2 S_h^2 / N_h. The term is small at a 3-point target and large at 1 point, so 6a to 6c were nearly unaffected and 6d and 6e were not.
• Fix: the rule is now llmci.estimators.neyman_total (n = (sum W S)^2 / (half^2/z^2 + sum W^2 S^2 / N_h)); test_neyman_total_hits_target_half_width_with_fpc checks that Neyman allocation at the returned n has exactly the target half-width from the closed-form variance (at 3, 1 and 0.5 points). Reran 6a, 6b and 6e (same seeds; the 6c and 6d scripts print no oracle and their simulations do not use it).
• Corrected oracle n: 3 points 888, 1,063, 912 (was 902, 1,083, 927); 1 point 6,881, 8,017, 7,045 (was 7,833, 9,240, 8,039).
• 6e H6e-ii now supported, retroactively: mean stopped n at B = 100 is 6,937, 8,069, 7,098 against 6,881, 8,017, 7,045, i.e. +0.8, +0.6, +0.8%. The "11 to 13% under" in 6e was the bug, and the stopped n sits slightly above the oracle, as the 30-per-round overshoot would predict. Nothing about stopping on a noisy estimate needed explaining.
• 6c H6c-ii restated: stopped n at B = 30 (898, 1,074, 919) is 1.1, 1.0 and 0.8% over the corrected oracle (888, 1,063, 912), not 0.4 to 0.9% under; the bar (within 10%) still holds. 6c's remark that it "sits slightly under, as expected" was wrong in sign.
• 6a: with the corrected n the simulated mean half-width is 3.00 points in all three coders (was 2.97, i.e. 1% over-audited); 95th percentile 3.09 to 3.13; coverage 94.6, 95.3, 95.8%; stratified hand-coding needs 851 (was 865). H5's width ratio (0.996 to 1.000) did not use the oracle size and is unchanged.
• 6b: at p = 20 mean half-width 3.06, 3.07, 3.10 (H6b-i holds), share at or under 3.0 points 46.6, 42.6, 44.5% (H6b-ii holds, 30 to 70%), coverage 94.9 to 95.9% in all nine cells (H6b-iii holds), sd of n at p = 10 over p = 40 is 2.0 to 2.2 (H6b-iv holds). Mean n at p = 40 is 883, 1,054, 903 against the oracle (now 888, 1,063, 912).
• 6d: its explanation (stopped n is 14 to 16% under 9x the 3-point oracle because of the finite-population factor) stands; the numbers it compared against were the 3-point ones, which move by 1.5%.
• Limits: this fixes a planning formula and re-reads earlier results; it adds no new data, corpus or real LLM coder. The 6e coverage and sd columns were regenerated with the same seed and are identical to the earlier run.
• Next: milestone 5 with a real coder (no local model or API key was reachable this session: Hugging Face returned 403 and ANTHROPIC_API_KEY is unset); a second corpus; a rarer category.

2026-10-25 · Milestone 6d: does the sequential rule survive a 1-point target and a tiny start? (planned before running)

• Why: 6c's open item. At a 3-point target and p = 20 per stratum the early variance estimates are decent; the stopping rule's weak spot is optional stopping when estimates are poor. Two harder cells, same corpus, split, three stand-in coders and floor (0.02) as 6c: target 1.0 point (about 9x the audit of 6c, since n scales with 1/target^2), start p in {5, 20} per stratum, batch B = 300, cap 15,000, 200 draws per cell (fewer than 6c because each audit is about 10x larger; coverage standard error is therefore about 1.5 points). Code: experiments/h6d_hard.py.
• H6d-i (coverage): coverage is at least 92% in every one of the 6 cells (3 coders x 2 starts); with 200 draws a 95% interval has a binomial sd of 1.5 points, so this bar tolerates noise but not a real optional-stopping loss. Refuted if any cell is below 92%.
• H6d-ii (a tiny start costs coverage): the p = 5 cell is at least 1 point below the p = 20 cell for the same coder in at least 2 of 3 coders. Prior: weak, noise may hide it. Refuted otherwise.
• H6d-iii (target hit): at least 99% of audits stop under the cap at half-width <= 1.0 point. Refuted otherwise (would show the cap binds).
• H6d-iv (no free lunch): mean stopped n is within 10% of 9x the 6a oracle n (8,118, 9,747, 8,343) for every coder and start. Refuted otherwise.
• Not tested: a real LLM coder, a second corpus, an alpha-spending correction.

### Outcome (experiments/h6d_hard.py, about 2 min; experiments/h6d_hard.txt; 200 draws per cell)

• H6d-i supported. Coverage is 95.0 to 97.0% in all 6 cells (lowest: lexicon-en, p = 5, 95.0%); no optional-stopping loss at a 1-point target.
• H6d-ii refuted. The p = 5 cell is 0.5, 0.5 and 0.0 points below p = 20 (bar: 1 point on at least 2 coders). A start of 5 per stratum costs nothing in coverage here; the only visible price is spread in stopped n for lexicon-en (sd 514 against 88, because one early variance estimate from 5 clean items hits the floor).
• H6d-iii supported. 100% of audits stop under the cap at mean half-width 0.98 to 0.99 points.
• H6d-iv refuted, and the reason is the finite-population correction, not the rule. Mean stopped n is 6,989 to 8,172 against 9x the 6a oracle of 8,118 to 9,747: 13.9, 16.2 and 13.7% below (p = 20). At 1 point the audit is 16 to 18% of the 44,485-tweet corpus, and the estimator's variance carries the (1 - n_h/N_h) factor (llmci/estimators.py, stratified_ppi), so n does not scale as 1/target^2. My 9x prediction ignored this. Not a rule defect.
• Reading: the sequential rule of 6c holds on the harder cell, with the audit about 8,000 tweets and coverage 95 to 97%. Limits: 200 draws (coverage sd about 1.5 points, so it cannot rule out a 2-point loss), one corpus, three stand-in coders, no real LLM coder, no alpha-spending comparison.
• Next: milestone 5 with a real coder; a second corpus.

2026-10-25 · Milestone 6c: a sequential audit that stops when the estimated width meets the target (planned before running)

• Why: 6b's open item. Planning from a pilot hits the +/-3 point target about half the time. The alternative is to audit in batches and stop the first time the estimated half-width (Z times the stratified-PPI standard error on the items audited so far) is at most 3.0 points. The width then meets the target by construction; what is unknown is the price in audit size and in coverage, because stopping on a noisy estimate of the width is optional stopping and tends to stop when the sample happens to look precise. Code: experiments/h6c_sequential.py; same corpus, split, three stand-in coders, seed scheme and floor (0.02) as h6b_pilot.py. Start with p = 20 per stratum (the 6b pilot), then add batches of B items split across strata by the Neyman shares from the current variance estimates (largest w_h / (n_h + 1) first, without replacement); B in {30, 90}; hard cap 2,500 items; 1,000 draws per cell.
• H6c-i (coverage survives optional stopping): coverage of the 95% interval at the stopping time is at least 93% in every one of the 6 cells (3 coders, 2 batch sizes). Refuted if any cell is below 93%.
• H6c-ii (price): mean stopped n at B = 30 is within 10% of the oracle n of 6a (902, 1,083, 927) for every coder. Refuted otherwise. My expectation is that it sits slightly under the oracle, since noisy estimates stop some audits early.
• H6c-iii (target hit): at least 99% of audits stop under the cap with half-width at most 3.0 points. Refuted otherwise (would show the cap binds).
• H6c-iv (batch size trade): mean n at B = 90 exceeds mean n at B = 30 for every coder (overshoot of up to a batch), by at most 90 items. Refuted otherwise.
• Reported without a bar: sd of stopped n, 5th and 95th percentiles, and the 6b p = 40 mean n and share under 3.0 for comparison.
• Not tested: a real LLM coder, a second corpus, a corrected (always-valid or alpha-spending) interval.

### Outcome (experiments/h6c_sequential.py, about 2.5 min; experiments/h6c_sequential.txt; 1,000 draws per cell)

• H6c-i supported. Coverage at the stopping time is 94.2 to 95.2% in all 6 cells (lowest: nb-in-language, B = 30, 94.2%). Stopping on the estimated width did not visibly cost coverage on this corpus.
• H6c-ii supported. Mean stopped n at B = 30 is 898, 1,074, 919 against the oracle 902, 1,083, 927 (0.4 to 0.9% under, as expected).
• H6c-iii supported. 100% of audits in every cell stopped under the cap at a half-width of at most 3.0 points (the rule meets the target by construction; the cap never bound).
• H6c-iv supported. B = 90 costs 29 to 30 more items on average than B = 30 (928, 1,103, 948), well inside one batch.
• Against 6b: the sequential rule hits the target in 100% of audits where a p = 20 pilot plan hit it in 44 to 52%, at essentially the oracle mean n. The sd of the audit size is 34 to 48 (B = 30), against 79 to 119 for the p = 40 pilot plan, because the stopping time absorbs the variance-estimation error instead of fixing n in advance. The price is operational: the audit has to be hand-coded in rounds (B = 30 means about 30 rounds) rather than in one pass.
• Limits: one corpus, three stand-in coders, one target (3 points), one start p = 20 and floor 0.02, no tuning of either; coverage is measured against the true share of this one corpus and the nominal normal interval was used with no optional-stopping correction, so the result says it is not needed here, not that it is never needed (a coder with heavier-tailed residuals or a much smaller target could differ); no real LLM coder.
• Next: a harder case for the stopping rule (rarer category or a 1-point target, where early variance estimates are poorer); milestone 5 with a real coder; a second corpus.

2026-10-25 · Milestone 6b: plan the audit size from a small pilot, not from oracle variances (planned before running)

• Why: 6a's size rule used the true per-stratum variances of r = y - f, which nobody has before auditing. The real use is two stage: audit p items per stratum as a pilot, estimate S_h^2 by the sample variance of r (floor 0.02 so a clean pilot cannot give zero), run the Neyman rule for a +/-3 point target to get the total n, top up each stratum to its Neyman share (never below p), and estimate with stratified PPI on all audited items (pilot included: within a stratum the pilot plus top-up is one simple random sample of size n_h). Code: experiments/h6b_pilot.py; same corpus, split, three stand-in coders, target and seed scheme as h5_plugin.py; p in {10, 20, 40} per stratum (pilot 40, 80, 160 items); 1,000 draws per cell.
• H6b-i (mean width near target): at p = 20 the mean simulated half-width is within 10% of 3 points (2.7 to 3.3) for all three coders. Refuted otherwise.
• H6b-ii (a coin flip, not a guarantee): at p = 20 the share of audits whose half-width is at most 3.0 points lies between 30% and 70% for every coder. The estimated variance is noisy and n is a function of it, so I expect no better than even odds of meeting the target; outside the range would mean the pilot estimate is biased.
• H6b-iii (coverage): coverage of the 95% interval is at least 93% in every one of the 9 cells (the plan depends on the pilot, which could hurt it). Refuted if any cell is below 93%.
• H6b-iv (pilot size buys stability, not the mean): the sd of the realised total n across draws at p = 10 is at least 1.5 times that at p = 40, for every coder.
• Reported without a bar: mean total n against the oracle n of 6a (902, 1,083, 927), the 95th percentile of the half-width, and the share of audits within 3.3 points.
• Not tested: a real LLM coder, a second corpus, a sequential rule that keeps auditing until the width is met.

### Outcome (experiments/h6b_pilot.py, about 30 s; experiments/h6b_pilot.txt; 1,000 draws per cell)

• H6b-i supported. At p = 20 the mean half-width is 3.03, 3.04 and 3.10 points (lexicon-en, nb-english, nb-in-language), inside 2.7 to 3.3.
• H6b-ii supported. Share of audits at or under 3.0 points at p = 20: 52.3%, 49.3%, 43.8%, as expected a coin flip. At p = 10 it is 42.7 to 43.7% and the 95th percentile of the half-width is 4.0 to 4.5 points; at p = 40 it is 50.1 to 54.7% with the 95th percentile at 3.2 to 3.4. Meeting a 10% looser bar (3.3 points) takes p = 40 (91.4 to 98.0%), not p = 20 (76.1 to 89.6%).
• H6b-iii supported. Coverage is 94.1 to 95.7% in all nine cells; the planning step does not hurt it.
• H6b-iv supported. The sd of the total n falls from 174, 240, 258 (p = 10) to 79, 109, 119 (p = 40), a ratio of 2.17 to 2.20 (bar 1.5).
• Mean total n at p = 40 (pilot 160 items included) is 892, 1,076, 917 against the oracle 902, 1,083, 927: the pilot costs nothing in expectation because the pilot items are reused in the final estimate, and the mean n is within 1.1% of the oracle at p = 40 and within 4% at p = 10. What the pilot changes is the spread: at p = 10 the audit size ranges widely (sd about 20% to 30% of n).
• Reading: planning from a pilot works on this corpus as a default (valid, unbiased in mean n), but it hits the target about half the time, so a user who needs "at most 3 points" should plan for a tighter target (about 10%, 3.3 vs 3.0) or use a pilot of 40 per stratum; a sequential rule that audits until the width is met is the clean alternative and is not tested here.
• Limits: same corpus, three stand-in coders, one target, one floor (0.02, not tuned: no other value tried); the pilot is a simple random sample per stratum, oracle strata sizes are known; no real LLM coder.
• Next: a sequential stopping rule (audit in batches until the estimated width is met) and its coverage; milestone 5 with a real coder; a second corpus.

2026-10-24 · Milestone 6a outcome: H5 supported for the width of the interval; the audit-size rule works but weak coders need a bigger audit than none

• Ran experiments/h5_plugin.py (1,000 draws per cell; experiments/h5_plugin.txt). Simulated mean half-width over plug-in: 0.996 to 1.000 in all 9 design-by-coder cells (bar 0.90 to 1.10), so H5 holds with room. As predicted this is near-identity (the plug-in is the expectation of the variance estimator); coverage in those cells is 93.6 to 95.8% (lowest: design C with nb-english, 93.6%, about 2 binomial sd under 95%).
• Audit-size rule (Neyman allocation, oracle S_h^2, +/-3 point target): total n = 902 (lexicon-en), 1,083 (nb-english), 927 (nb-in-language). Simulated mean half-width 2.97 points in all three (target met), but the 95th percentile is 3.06 to 3.10 and coverage 94.1, 94.1 and 95.2%, so the rule hits the target on average, not in every audit.
• Against no coder: human-only with the same Neyman rule needs 865 hand-coded items for the same half-width. The three stand-in coders need 4 to 25% more, because their residual y - f has higher variance than y itself in the strata where they are poor (nb-in-language has sensitivity 0.98 and specificity 0.06 in Nigerian Pidgin; nb-english sensitivity 0.39 in the same stratum). Prediction before the run said nothing on this; it matters: a coder only reduces the audit size once its error rate per stratum is below the strata's own prevalence variance, and the formula says when.
• Limits: sens/spec/prevalence are oracle values from the full corpus; planning from a pilot audit (the real use) adds estimation error that is not tested; one corpus, one target category, stand-in coders; the width formula is nearly an identity, so the informative parts are the size rule and the comparison with human-only. Regression-coefficient extension (milestone 6b) not started.
• Next: pilot-based planning (choose n from 10% of the audit and check realised width); milestone 5 with a real coder, which is where PPI's sample-size saving should show.

2026-10-24 · Milestone 6a: H5, the plug-in width formula and the audit-size rule (planned before running)

• Data and coders as milestone 3 (NaijaSenti, four language strata, three stand-in coders). Plug-in half-width = 1.96 sqrt(sum_h W_h^2 (1 - n_h/N_h) S_h^2 / n_h) with S_h^2 the variance of r = y - f in stratum h, written from the stratum's prevalence, sensitivity and specificity (S_h^2 = [fn + fp - (fn - fp)^2] N_h/(N_h - 1), fn = pi(1 - sens), fp = (1 - pi)(1 - spec)). The inputs are the true finite-stratum values (an oracle), so this tests the formula, not the planning from a pilot. Designs: equal quota (B), 70% Hausa (C) and Neyman allocation at n = 400. experiments/h5_plugin.py.
• Prediction (H5): simulated mean half-width is within 10% of the plug-in in all 9 cells (ratio 0.90 to 1.10); in fact the formula is an identity for the mean of the variance estimator up to Jensen's gap, so I expect ratios of 0.98 to 1.00. Audit-size rule: Neyman total n for a 3-point half-width gives a simulated mean half-width within 5% of 3 points and coverage of 93% or more. Refuted if any ratio is outside 0.90 to 1.10.
• What this cannot show: the formula is closed-form and nearly tautological for the width of the variance estimate; it says nothing about planning when sens/spec must be estimated from a pilot, which is the harder claim and is left open.

2026-10-24 · Milestone 4e outcome: the floor rule stays valid on 18 unseen categories but is wider than Wilson in 16 (width prediction refuted)

• Ran experiments/floor_rule_heldout.py (500 draws per cell, ±1.9 points; experiments/floor_rule_heldout.txt). Prevalences 0.99% (desire) to 7.69% (admiration); the rule picked floor 0.054 (desire) to 0.100 (twelve categories).
• Coverage of the normal IPW interval: 93.2 to 97.0% in all 18 (several sit at 93.2%, close to the bar). Prediction (15 of 18 at 93% or more) supported; the refutation bar (4 or more under) not approached.
• Width against uniform Wilson: 0.89 (desire), 1.00 (surprise), then 1.02 to 1.49 for the other 16 (admiration 1.46, approval 1.49, gratitude 1.44). Prediction (at most 1.00 in 12 of 18) refuted: only 2 of 18. Above ~1.5% prevalence the audit design is valid but costs 2 to 49% more width than just sampling uniformly and using Wilson, as 4d warned for the top of the range.
• So the rule is a safe default, not a gain, except below ~1%: the 4d gain (0.48 to 0.56 of Wilson) is confined to the five rarest categories, and at 1 to 2% it breaks even. A rule that falls back to uniform Wilson above a prevalence threshold (qbar of about 1.5%) would dominate it on this corpus; that is a post-hoc idea and would again need its own held-out test.
• Limits: same corpus, split, coder (weak, non-LLM) and calibration as 4c/4d; held-out categories, not a held-out corpus; no category below 0.99% was available as held-out, so the sub-1% regime that carries the claimed gain is still tested only in-sample. 500 draws.
• Next: a second corpus with rare codes for the sub-1% regime; milestone 5 (real LLM coder, needs the owner's key or a local model).

2026-10-24 · Milestone 4e: the floor rule on the 18 GoEmotions categories it was not designed on (planned before running)

• Why: 4d was in-sample (rule chosen after reading 4c on the nine rare categories). No second rare-code corpus is fetched, so the cheapest honest out-of-sample test is the same rule, frozen, on the other 18 non-neutral emotions (admiration ... surprise), none of which were looked at. Same corpus, split, coder, calibration, n = 200, normal IPW interval, floor = clip(5 qbar, 0.02, 0.10); 500 draws per cell. These are mostly 2 to 15% prevalence, so this tests the rule's upper regime, not the sub-1% regime that drove it. experiments/floor_rule_heldout.py.
• Prediction: coverage of 93% or more in at least 15 of 18 categories, and width at most 1.00 of uniform Wilson in at least 12 of 18 (the rule should not cost width where it cannot gain). Refuted if coverage is under 93% in 4 or more categories.

2026-10-24 · Milestone 4d outcome: a label-free floor rule is valid in 8 of 9 categories and narrower than Wilson in 6 (prediction supported, in-sample)

• Ran experiments/floor_rule.py (500 draws per cell, ±1.9 points; results in experiments/floor_rule.txt; 22 s). Floor = clip(5 qbar, 0.02, 0.10) came out 0.020 for the five rarest categories, 0.047 to 0.059 for remorse, fear and disgust, 0.100 for sadness.
• Normal-IPW coverage: 94.8 to 96.0% for the five rarest, 92.4% remorse, 93.0% disgust, 94.4% fear, 95.0% sadness. So 8 of 9 reach 93% (remorse 92.4% is 0.6 under, inside one binomial sd of 1.0), against the refutation bar of three or more under.
• Width against uniform Wilson: 0.48 (grief), 0.54 (pride), 0.56 (relief), 0.65 (nervousness), 0.75 (remorse), 0.80 (embarrassment), 0.95 (fear), 1.00 (disgust), 1.13 (sadness). So six of nine are 20 to 52% narrower, and the prediction (three rarest at 0.90 or less) holds with room. Above ~1% prevalence the audit design buys nothing over uniform Wilson; it is merely valid.
• Limits: the multiplier 5 and the clip bounds were chosen after reading 4c on this same corpus, so this is an in-sample check of a one-parameter rule, not a test of it; one corpus, one weak non-LLM coder, 500 draws. The 0.02 floor leaves 49 to 59% of audits with no positive for the three rarest, where the interval still covers by design but a user would see a degenerate estimate.
• Next: milestone 5 (a real LLM coder; none available offline in this session, and an API-called one needs the owner's key); test the rule on a second corpus (NaijaSenti negative is not rare, so a rare-code corpus is needed) before it goes in the paper.

2026-10-24 · Milestone 4d: choose the pi floor from the coder, no pilot labels (planned before running)

• Why: 4c left two regimes (floor 0.02 best below ~1% prevalence, 0.05 to 0.10 best above), and a rule must pick the floor without knowing the prevalence. The calibrated coder probability q is known for every corpus item with no human labels; its corpus mean qbar tracks prevalence (calibrated on the 30% train part). Rule fixed here, before running: floor = clip(5 * qbar, 0.02, 0.10). Same corpus, split, coder, calibration, n = 200 and normal IPW interval as 4c; 500 draws per cell (+/- 1.9 points). experiments/floor_rule.py.
• Prediction: the rule gives normal-IPW coverage of 93% or more in at least eight of nine categories, and a width at most 0.90 of uniform Wilson in at least the three rarest. Refuted if coverage is under 93% in three or more categories, or if no category is narrower than uniform Wilson by 10%. The rule is a guess made after seeing 4c on this corpus, so even a pass is in-sample for the floor range; I will say so.

2026-10-24 · Milestone 4c outcome: a bigger pi floor helps the common-rare categories and hurts the rarest (prediction refuted)

• Ran experiments/pi_floor.py (500 draws per cell, ±1.9 points; same corpus, split, coder and calibration as milestone 4; results in experiments/pi_floor.txt). Planned before running: raising the floor on the inclusion weight w = max(floor, sqrt(q(1-q))) from 0.02 to 0.05, 0.10, 0.20 and 0.50 (0.50 is uniform sampling) puts audited positives in more audits and so brings the normal and Wilson IPW intervals to 93% or more in all nine categories. Refuted if any category stays under 93% at every floor.
• Refuted for the six rarest categories (0.10 to 0.89%): the floor 0.02 is the best of the five, and coverage falls as the floor rises (grief 96.0% at 0.02, 80.2% at 0.05, 59.4% at uniform; relief 94.8 / 86.2 / 69.2%; nervousness 95.0 / 98.0 / 80.0%). Share of audits holding no positive rises with the floor (grief 59% to 85%). The 0.02 floor concentrates the audit where the coder says "uncertain", which is where the few positives sit; more uniform sampling dilutes it, and a normal interval on an audit with zero positives is too narrow.
• Supported for the three commonest (1.05 to 1.96%): a floor of 0.05 to 0.10 gives normal-IPW coverage 93.0 to 95.6% (disgust 93.0 / 93.4%, fear 94.8 / 95.6%, sadness 93.4 / 95.0%), against 88.8 / 92.8 / 90.2% at 0.02. Width there is 0.98 to 1.13 times the uniform Wilson width, so the gain over the valid baseline is gone. Remorse (0.89%) covers 91.4 to 93.6% at every floor.
• The Kish-size Wilson interval is not a fix at any floor: it covers 55 to 96%, is erratic in width (up to 13 times the uniform Wilson width in some cells, from audits whose variance estimate is near zero), and is below the normal interval in most cells at floors above 0.02.
• So the two goals conflict: below about 1% prevalence only the 0.02 floor is near-valid and it is 44 to 52% narrower than uniform Wilson (grief, pride, relief; 0.48 to 0.56 of the width); from 1% up, a floor of 0.05 to 0.10 is valid but no narrower than uniform Wilson. A single audit rule would need the floor to depend on the category's (unknown) prevalence; the coder-positive rate or a pilot audit is the obvious proxy and is untested.
• Limits: one corpus, one weak non-LLM coder, one calibration rule, 500 draws per cell, floors on a coarse grid, the true prevalence is only known because the corpus is hand-coded. Still not tested: an exact conditional interval, and a real LLM coder (milestone 5).
• Next: milestone 5 with a real (local) LLM coder; choosing the floor from a pilot audit.

2026-10-24 · Milestone 4b outcome: neither small-count interval fixes the weighted audit (refuted)

• Re-ran experiments/rare_audit.py (same corpus, split and seed; the extra bootstrap draws shift the random stream, so the other rows differ from the milestone 4 file by Monte Carlo noise, e.g. Q coverage for disgust 85.1% vs 85.6%). New rows in experiments/rare_audit.txt.
• Kish-size Wilson (ipw_audit_wilson) covers 82.4% (grief) and 84.2% (pride), 89.4 to 91.5% for relief, embarrassment and nervousness, 90.4% for disgust, 93.2 to 95.4% for remorse, fear and sadness. It fixes sadness (95.4% vs 87.5%) but is 1.3 to 1.6 times as wide as the normal IPW interval there, and undercovers where the audit holds almost no positives, because the effective size from a variance estimated on 0 to 3 positives is itself unreliable.
• Percentile bootstrap (ipw_audit_bootstrap) covers 93.8 and 93.9% for grief and pride but 88.6 to 92.6% for the middle categories and 77.5 to 89.8% for disgust, fear and sadness (worse than normal: resampling the audit cannot see the positives it lacks, and the heavy 1/pi terms make the percentile interval skewed).
• Prediction check: refuted (Wilson below 93% in six of nine, bootstrap in seven). The normal IPW interval remains the best of the three at the six rarest categories (93.2 to 96.1% before, 94.0 to 95.9% now) and fails at disgust and sadness; Wilson is the better choice only above ~1.5% prevalence. No interval tried is valid across the whole range; a design-based fix (a larger floor on pi, so every audit holds positives, or an exact conditional interval) is the open piece.
• Limits: one corpus and coder, 1,000 draws per cell (+/- 1.4 points), fixed corpus.
• Next: pi floor sweep and an exact (Clopper-Pearson-type) weighted interval; then milestone 5 with a real LLM coder.

2026-10-24 · Milestone 4b: small-count-safe intervals for the weighted audit (planned before running)

• Why: the Q (uncertainty IPW) audit's normal interval covered 85.6% (disgust, 1.32%) and 89.0% (sadness, 1.96%). Added to llmci: ipw_audit_wilson (Wilson score at the Kish-type effective size p(1-p)/se^2) and ipw_audit_bootstrap (percentile, 300 resamples of the audited items).
• Prediction: both reach 93% or more coverage in all nine categories at n = 200 (same corpus, seeds and draws as milestone 4). Wilson will be wider than normal; the bootstrap, which resamples few rare positives, may undercover in the rarest categories (grief, pride) where the audit holds 0 to 3 positives. Width is compared with uniform Wilson, the valid baseline. Refuted if either is below 93% in more than two categories, or if neither is narrower than uniform Wilson in at least 3.

2026-10-24 · Milestone 4 outcome: H2 and H4 refuted as stated; the useful result is about validity

• Ran experiments/rare_audit.py (results in experiments/rare_audit.txt): GoEmotions parts 1 to 3, 54,619 examples after the filters, corpus 38,234, nine categories from 0.10% (grief) to 1.96% (sadness), n = 200, 1,000 draws per cell. Deviation from the plan: naive Bayes gave precision 0.00 to 0.03, so the stand-in coder became a class-weighted logistic regression on word presence (precision 0.04 to 0.24, recall 0.07 to 0.78); still weak, still not an LLM. Also the stratified and the plain-human stratified rows are identical by construction (the coder is constant inside a coder-label stratum), so "label-stratified PPI" is just post-stratified hand-coding; the coder's codes only define the strata.
• Validity first. The normal-approximation intervals break at low prevalence with n = 200: uniform human-only covers 16% (grief, 0.10%) to 61% (0.42%), 83 to 93% from 0.9%; PPI++ the same or worse; label-stratified 10 to 81%. Wilson on uniform human-only covers 91 to 99% in every category. The coder-uncertainty IPW audit (Q) covers 93.2 to 96.1% for the six categories at or under 0.9% and 94.4% for fear (1.05%), but only 85.6% for disgust (1.32%) and 89.0% for sadness (1.96%).
• H2: refuted. Uniform PPI++ is within 10% of human-only width in eight of nine categories (0.93 to 0.98 of it; remorse 0.87), so the first half holds, but it is irrelevant when both undercover. The stratified audit cut width by 25% or more in 2 of 9 (pride 26%, remorse 47%; fear 22% falls short) and by under 10% in 3 (grief 2%, relief 8%, nervousness 6%), with coverage 10 to 81%, so no valid gain. Refutation condition met (gain under 10% in several categories).
• H4: refuted as stated. Label stratification is not valid in normal approximation here, and it differs from uncertainty sampling by far more than 10%. What survives: against the only valid baseline at these prevalences (Wilson), uncertainty-IPW intervals are 45 to 52% narrower for grief, pride and relief (0.48 to 0.55 of the width), 22 to 34% narrower for embarrassment, nervousness and remorse, 5 to 13% for disgust and fear (disgust at 85.6% coverage), and wider for sadness (1.04), with coverage 93 to 96% except disgust and sadness.
• Limits: one corpus, a weak stand-in coder, truth = majority of 3+ raters (stricter than any single rater: prevalences are small), one uncertainty rule (pi proportional to sqrt(q(1-q)) from train deciles, floor 0.02), a fixed corpus, normal intervals for the IPW design (a small-count-safe interval for IPW audits is the next thing to build; its undercoverage at 1 to 2% is probably the same normal-approximation failure). Not tested: that a better coder moves the gain (milestone 5).
• Next: a small-count-safe interval for weighted and stratified audits (Wilson-type or bootstrap), then milestone 5 with a real LLM coder.

2026-10-24 · Milestone 4: H2 and H4, rare-category and audit-design benchmark (planned before running)

• Data: GoEmotions part 1 (Demszky et al. 2020; Reddit comments, ~58k after the qual-real filters: at least 3 raters, not flagged unclear). Truth for a category = a majority of raters marked that emotion. Nine categories chosen for their prevalence, not their difficulty: grief, pride, relief, embarrassment, nervousness, remorse, disgust, fear, sadness (the true prevalence in the corpus is printed first and decides which fall at or under 2%). Seeded 30/70 split; the 70% test part is the corpus, so truth is known.
• Coder: a multinomial naive-Bayes log-odds score trained per category on the 30% part (a stand-in, not an LLM; milestone 5). Coder label = score above a threshold set on the train part so that coder-positives are about 3x the category's prevalence. Scores are binned into deciles on the train part to give a calibrated probability q for uncertainty sampling.
• Audit designs, n = 200 expected: (U) uniform, with human-only and PPI++; (L) coder-label stratified 60 positive / 140 negative, with stratified PPI and stratified human-only; (Q) uncertainty sampling, Poisson inclusion pi_i proportional to sqrt(q_i(1-q_i)) (floored at 0.02, scaled to 200 expected), ipw_audit. Also human-only with a Wilson interval, as the small-count-safe baseline. 1,000 draws per cell, fixed corpus, seed fixed.
• Prediction. H2: at prevalence <= 2%, PPI++ under U is within 10% of the width of human-only; stratified L cuts width by at least 25% at equal n. H4: L and Q have narrower intervals than U below 5% and are within 10% of each other, with coverage judged against 95% (+/- 1.4 points). I expect the normal-approximation intervals to undercover in the rarest categories (the simulated result says 57% at 1%), which would mean "narrower" is not "valid" there; I will report width only for cells that cover at least 93%, and say so where a gain exists only by undercovering.
• Refuted per the proposal: H2 if uniform PPI++ is 25% or more narrower than human-only at 2% or below, or L gains under 10%. H4 if either design loses to uniform, or they differ by more than 10%.

2026-10-24 · Milestone 3 outcome: H1 supported on real data

• Ran experiments/group_shift.py as planned (results in experiments/group_shift.txt). Added stratified_human to the library (+1 test, 8 pass).
• Under a random audit (A) every estimator covers 94.5 to 95.9%. Under shifted audits the pooled PPI interval undercovers for the coders that read languages unequally: 67.2% (nb-english) and 69.3% (nb-in-language) with a 70%-Hausa audit (C), 84.2% (nb-in-language) with an equal-quota audit (B). Pooled PPI++ is better but not safe (89.1% and 89.3%). Pooled human-only, which uses no coder, also fails under B (86.9%) because the audit over-represents Pidgin, so the failure is the audit's mix, not the coder alone.
• Language-stratified PPI with corpus-known sizes: 93.8 to 95.9% in all nine shifted/random cells. The planned bar was at least 94%: one cell (B, nb-in-language) is 93.8%, 0.2 points under, inside one binomial sd (0.7); I report it as a narrow miss of the stated threshold, not as met. The refutation condition (pooled at 93% or more everywhere) did not hold, so H1 stands for these coders.
• Against the grain of the correction's purpose: stratified PPI is not narrower than stratified human-only here (C: 11.2 to 12.6 vs 11.0 points; A: 9.1 to 9.9 vs 8.8), because these three stand-in coders are weak, as in qual-dialect. Coder codes buy validity under shift, not width. Whether a real LLM changes that is H3 (milestone 5).
• Limits: one corpus, one target category (negative), three audit mixes chosen by me, a fixed corpus (only the audit is random), stand-in coders. The corpus is the test split, so intervals target the finite corpus.
• Next: milestone 4 (H2 and H4, rare-category and audit-design benchmark); a rare category needs small-count-safe intervals first (see the simulated result in the README).

2026-10-24 · Milestone 3: H1, group-shift benchmark on NaijaSenti (planned before running)

• Data: NaijaSenti (hau, ibo, pcm, yor; 63,550 tweets, native-speaker labels; CC BY), the same seeded 30/70 split as qual-dialect; the 44,485-tweet test part is the corpus, so the true corpus prevalence of "negative" (and of each language) is known. Coders: the three stand-ins of qual-dialect (lexicon-en, nb-english, nb-in-language); no LLM yet (milestone 5).
• Audit designs, n = 400 hand-coded: (A) random over the corpus, mix matches; (B) equal quota per language (100 each); (C) 70% Hausa, 10% each of the others. Audits are drawn without replacement inside each language.
• Estimators of the corpus prevalence: pooled human-only, pooled PPI (lambda = 1), pooled PPI++, and language-stratified PPI and human-only with corpus-known language sizes (stratified_ppi; a new stratified_human is added). 1,000 audit draws per cell, fixed corpus, seed fixed.
• Prediction (H1): under B and C the pooled PPI/PPI++ intervals undercover (coverage below 90% for at least one coder) because the audit's group mix differs from the corpus and error differs by language; under A they cover about 95%; stratified PPI covers at least 94% in all designs. H1 is refuted if pooled coverage is 93% or more in every shifted design with every coder. Coverage is judged with a binomial interval on 1,000 draws (+/- 1.4 points at 95%).

2026-10-24 · Milestone 2: estimator library llmci/

• Hypothesis (planned before running): the library's PPI++ and stratified-PPI intervals cover 95% (within 3 binomial sd over 600 simulated studies, n=2000, audit 150, prevalence 25%, coder sens 0.75 / spec 0.90), and are narrower than human-only.
• Built llmci/estimators.py (stdlib only): human_only, naive, ppi (fixed lambda), ppi_pp (clipped optimal lambda), ipw_audit (known inclusion probabilities, Poisson variance), stratified_ppi (finite-population correction), label_shift (binary BBSE, delta-method se). 7 tests in test_llmci.py: lambda endpoints, a perfect coder, closed forms for IPW and stratified variance, exact label-shift recovery, and the Monte Carlo coverage check above.
• Outcome: all 7 pass. The coverage check held. Checked against closed forms and simulation only; no published numerical example is reproduced yet (the PPI papers' datasets are not fetched). Not yet covered: DSL with a fitted nuisance model, cross-PPI, active sampling, regression coefficients.
• Consolidation is partial: the qual-* modules are untouched and still carry their own copies.
• Next: milestone 3 (H1, group-shift benchmark on a real multi-group corpus), which needs a group-aware estimator on top of stratified_ppi.

2026-10-24 · Milestone 1a: citations machine-checked

• Ran cite_check.py on PROPOSAL.md (network worked this time). Two arXiv ids written from memory were wrong: 2309.08574 is an unrelated differential-privacy paper (Cross-prediction-powered inference is 2309.16598) and 2405.20039 is Miao & Lu, Task-agnostic ML-assisted inference (Fisch et al., stratified PPI for hybrid LM evaluation, is 2406.04291; its title was also misremembered). Fixed in the proposal. The tool reports ok whenever an id resolves, so matching titles had to be read by eye.
• 13 of 14 ids/DOIs now resolve to the intended paper; doi:10.1145/3117807 (González et al.) got HTTP 429 and is unchecked.
• Not done in milestone 1: reading Gligorić et al. and Fisch et al. in full, corpus licences, local-LLM throughput.

2026-10-24 · Programme opened: proposal written

• Opened from the table row in research/PROGRAMMES.md that had no proposal. Consolidates the qual-* directories (22) into one question: valid intervals from LLM-coded text under group shift and rare categories.
• Not machine-checked: arXiv and Crossref returned 403 through the egress proxy this session, so the proposal's ids were written from memory and the proposal says so. Milestone 1 starts by running research/_shared/cite_check.py on them and reading the two closest competitors (Gligorić et al. 2024; Fisch et al. 2024) in full. If either already covers the combination in the Gap section, the programme narrows.
• No experiment run. Next: milestone 1.

2026-10-25 · Milestone 6e: the 1-point target against the finite-population-aware oracle (planned before running; run alongside 6d, PR 10307)

• Why: 6c's open item. At a 3-point target the audit is about 900 items and the p = 20 start (80 items) is a tenth of it; at 1 point the audit is about 9x larger, the sampling fraction reaches 20% in some strata (finite-population effects), and the stopping rule is exercised over ~70 rounds. Same corpus, split, coders, floor (0.02) and start (p = 20) as 6c; target 1.0 point; batches B in {100, 300}; cap 20,000; 300 draws per cell (+/- 2.5 points on coverage). Oracle n from the 6a Neyman rule at 1 point. experiments/h6e_tight_target.py.
• H6e-i (coverage): coverage at the stopping time is at least 92% in all 6 cells (300 draws, so the bar allows for binomial noise of about 1.3 points). Refuted if any cell is below 92%.
• H6e-ii (price): mean stopped n at B = 100 is within 10% of the oracle n for every coder. Refuted otherwise.
• H6e-iii (target hit): 100% of audits stop under the cap at half-width at most 1.0 points.
• H6e-iv (relative spread shrinks): the sd of stopped n divided by its mean at B = 100 is below that of 6c at B = 30 (0.038, 0.040, 0.052) for every coder.
• Not tested: a real LLM coder, a second corpus, a rarer category.
• Note: 6d (another run, same day) explained a similar gap by the finite-population correction. My oracle already includes it (neyman_total has the sum W^2 S^2 / N term), so this run asks whether a gap remains once it is included.

### Outcome (experiments/h6e_tight_target.py, about 17 min; experiments/h6e_tight_target.txt; 300 draws per cell)

• H6e-i supported. Coverage at the stopping time is 93.0 to 96.0% in all 6 cells (lowest: lexicon-en, B = 100, 93.0%; bar 92%).
• H6e-ii refuted. Mean stopped n at B = 100 is 6,937, 8,069 and 7,098 against the oracle 7,833, 9,240 and 8,039: 11.4%, 12.7% and 11.7% under, outside the 10% bar. At a 3-point target the gap was under 1%. The rule stops earlier than the plug-in Neyman size, yet coverage holds, so the oracle size is not what the stopped audit needs here. I did not test why; candidate causes are the sampling fraction (up to about a quarter of a stratum is audited, so the plug-in's constant variances and the realised without-replacement variances can differ) and stopping when the estimate happens to look narrow. Neither is checked.
• H6e-iii supported. 100% of audits stopped under the cap at half-width at most 1.0 points (mean 0.99 to 1.00).
• H6e-iv supported. sd/mean of stopped n at B = 100 is 0.011, 0.012, 0.015, against 0.038, 0.040, 0.052 for 6c at B = 30. B = 300 costs 75 to 144 more tweets than B = 100.
• Limits: one corpus, three stand-in coders, 300 draws (coverage +/- 2.5 points), no optional-stopping correction, no real LLM coder; the oracle-gap cause is open. Because this oracle already includes the finite-population correction, 6d's explanation (a 9x oracle that ignores it) does not account for the 11 to 13% gap here; the gap is not explained.
• Next: find why the stopped n sits under the oracle (compare realised half-width at fixed n against the plug-in); a rarer category; milestone 5 with a real coder.