How should a tolerance interval be paid for?

What it found

The Winkler interval score is the sum of two pinball losses: proper for the equal-tailed quantile interval with exact regret (2/α)[PR(l,α/2)+PR(u,1−α/2)], finite even for Cauchy drift where the expected score does not exist, but its payment grows without bound in the miss distance. A width-plus-miss loss (u−l)+λ1[miss] is bounded by width+λ and elicits the density level set, the shortest interval for its coverage: 22–31% narrower than equal-tailed on lognormal(0,1) drift. The price is statistical: to detect a 20% understatement of a normal tolerance it needs 3–48× more tasks for α≥0.1 and about the same at α=0.01.

A figure from its own output

Paper, code and results

• README and method
• White paper
• Experiments and results
• Code
• Tests

More in Proper scoring & elicitation

• How should a verifier's vector report be scored?Scoring a verifier's vector report: the precision score Σ⁻¹ is uniformly most detectable among quadratic Bregman scores (Cauchy–Schwarz), an isotropic score can need (1+κ)²/(4κ) times more tasks (25× at κ=100, power 0.29 vs 0.80), diagonal normalisation barely helps, estimated precision reaches 0.61 efficiency at m=2d draws.
• What does a verifier's score gap cost in decisions?From a verifier's excess proper score to a certified cap on accept/audit/slash decision regret: exact calibration functions (Brier ε², log KL), linear transfer only for non-proper hinge, audit action tightens the guarantee.
• Which surrogate should train a verifier that can escalate?Polyhedral surrogates for accept/reject/escalate verifiers: minimiser embeds the Bayes action, linear regret transfer with constant exactly 2d, logistic transfer ratio 2d(1-d)/δ, and a negative SGD result.
• How do you pay a verifier whose report decides the job?Scoring verifiers whose report decides acceptance and whose outcome is seen only on acceptance: a truthful reserve exists iff the score centre τ_s ≥ τ_d (unique c0 = 1+τ_d²−2τ_dτ_s), plain Brier fails above τ_d = ½, recentring fixes it at rent τ³/3, exploration costs 1/ε payment variance.
• What if the referee itself is noisy?Scoring verifiers against a noisy judge: raw proper scores elicit e0+γp and shrink Brier incentives by γ², can reverse rankings (η*=0.25), the unbiased surrogate score is exactly proper at a 1/γ payment spread and 1/γ² sample cost, and misestimated rates give a closed-form bias.
• Can audits be predictable without corrupting scores?Scoring when truth exists only on audited tasks: report-dependent audit rates make naive scoring improper (shift −bp(1−p)/(2(a+bp)), full hiding for steep rates), inverse-propensity weighting is exactly proper for any audit rule, and Neyman allocation gains only 3–11% variance.

All 28 Proper scoring & elicitation projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.