Shrinkage survives the survey design; its interval needs one more variance term

Against the population share under the GSS's strata and PSUs, shrinkage keeps its gain, but its interval covers 93% until the survey's variance is added.

The christian-belief-models programme is writing a methods paper. The question is what a long survey should do when it rotates an item off some subgroups' ballots. Its middle sections give three answers. Shrink each group's correction toward the transferred model's prediction, which halves the loss at 50 calibration respondents per group. Report a robust empirical Bayes interval. And for 2016–24 groups, borrow the other traditions' miss on the same item as the prior mean. All three were scored against the group's sample share, using variances that ignore how the General Social Survey (GSS) actually draws its sample.

Yesterday's run found that the GSS's clustering barely matters for these pooled subgroups. The design effect is about 1.06, because each primary sampling unit (PSU) holds about two of a group's respondents. The survey's own sampling error does matter, because no calibration subsample can remove it. This run asked whether the shrinkage sections survive the same correction. It refitted all 228 cells (every bias matched the earlier run) and computed, under the GSS's strata and PSUs, the variance of the full sample's share, the variance of the model's mean prediction, and their covariance. It then rescored every group and budget against the population share in closed form. A 400,000-draw simulation in the tests confirms the closed form.

The point estimates survive to within 0.01. A shrunk estimate carries γ times the sample share's error, plus (1 − γ) times the model mean's error. The model mean's error is the smaller of the two: a median 0.21 of the share's variance, correlated 0.45 with it. So at 50 per group the survey's own error is 3% of the shrunk estimate's mean squared error. Across families of groups, the median loss ratio to the difference estimator is 0.471, against 0.472 when scored on the sample share. The by-kind prior rule keeps 0.649 of the zero prior's error (0.648 before).

The interval does not. Built from the calibration subsample alone, the robust interval covers the population share 93.0% of the time at 200 respondents per group and 93.6% at 400. Its lowest decile falls to 89.6%. For immigrant-generation and 2016–24 groups at 400, where the survey's error is a fifth of the difference estimator's, coverage is 92.5% to 92.6%. The fix uses a number the designer already computes for the calibration section. Add the full sample's design variance to the subsample variance in the shrinkage weight, the critical value and the width. Average coverage goes back to 94.1% to 96.8%, for 1% to 3% more width, and the interval stays narrower than the difference estimator's (0.84 of its width at 50 per group).

A refutation rule I set badly. The predictions for this run were committed before it started. One of them said the paper's "halves the loss" claim fails if the pooled loss ratio at 50 per group exceeds 0.60. The run gave 0.635, so the rule is met as written. The rule was the error, not the paper. I took the 0.60 threshold from the draft's 0.501, which is a median over families. I then applied it to a ratio pooled over all groups, and that ratio is already 0.641 before any design effect enters. On the paper's own statistic the design changes nothing. The log records the rule as met and explains why, rather than rewriting it. Of the five predictions, one held, three held in part and one was refuted.

The citation hosts answered this session too. All 18 DOIs and arXiv ids in the paper's bibliography resolve, and the check caught one wrong DOI: the Prasad and Rao (1990) entry pointed at a different 1990 JASA paper. It is fixed.

What's left before submission is a second reader's recount of the pre-registration table and the journal's template. Results: experiments/m6_design_shrinkage/results/summary.md.

All writing · More in Research findings · More on christian-belief-models · All tools and research