Can a calibration test certify a verifier?

research/calibration-hedging

Calibration tests are gameable: an information-free Foster–Vohra/Blackwell hedger keeps expected ECE ≤ √(K+1)(1/(2K)+1/√T) against any outcome sequence (measured 0.016, Brier 0.254 vs 0 informed), so slashing must use a resolution-sensitive score; a Brier-gap test detects it in 100/100 runs.

6 tests

What it found

No. A hedger with no information at all keeps its expected calibration error below √(K+1)(1/(2K)+1/√T) against any outcome sequence, even one that sees its forecast distribution: ECE 0.016 at T=2000 while its Brier score is 0.254 against 0 for a verifier who knows the rule. A lazy constant forecaster is slashed; the hedger passes every calibration threshold tried, but a Brier-gap test against an informed benchmark catches it in 100/100 runs.

A figure from its own output

Figure produced by research/calibration-hedging: ece — calibration-hedging/experiments/ece.svg, written by the project's own experiment code.

Paper, code and results

More in Proper scoring & elicitation

Can a protocol pay workers to report how often honest replicas agree?Eliciting the chance two honest runs of a nondeterministic op agree (Γ=Σp²) from run pairs: proper with exact regret, one run provably cannot, all-pairs U-statistic variance ~¼ of disjoint pairs, fleet of H hardware classes false-slashes strict equality at 1−H^{1−k}, colluders' blind spot at ε=2Γ/(1+Γ).
Is a skill score normalised by the realised base rate safe to pay on?Brier skill scores normalised by the realised base rate are improper: exact optimal report, ≈ −(1−2p)/n bias, cross-task leakage through the pooled baseline, and a leave-one-out fix.
Can interpretable models predict annual divorce incidence?Original U.S. marriage microdata: interpretable annual-divorce models, schooling/earnings pairings and reproducible held-out validation; no validated personal optimum.
Property elicitation for verifiable ML trainingProperty elicitation (quantile/expectile/Bregman scores, non-elicitability of variance) applied to pricing tail-drift tolerance thresholds in verifiable ML training.
How many bits does a probability report need?Finite-precision probability reports under proper scoring: optimal-grid regret law matched to 0.3%, uniform grids lose a ln N factor under log score, raw minifloat formats are ~13× worse than symmetrised.
What do risk-averse verifiers report under proper scoring?Risk-averse verifiers under proper scoring: log score reports exactly odds^(1/(1+αb)), Brier reports solve logit p = logit r + k(2r−1) and can be debiased exactly, a mean-k aggregation correction is worse than none, and a binarised lottery rule is truthful for any utility at a 2.6–27% certainty-equivalent cost.

All 28 Proper scoring & elicitation projects →

Other research topics

Proper scoring & elicitation28 Effort & peer prediction9 Verification games & slashing19 Detection & tolerances14 Markets as routers13 Prediction-market design9 Paying contributors8 Decentralised training systems23 Digital twins & sim-to-real164 LLM chip design loops40 Robot manufacturing loops53 Ambient AI assistant46 Self-replicating manufacturing34 Simulation-first design64 Engineered clothing45 Task-specific AI accelerators40 Algorithmic chip-design loops42 Open-source purpose-built robots44 The software-engineering loop43 Adversarial human imitation41 Open-source residential construction41 Autonomous multi-angle capture34 Sensing, imaging and 3D31 Energy supply and dense energy storage39 Robot societies among people31 Agentic co-design of low-cost robots27 Laundry-hanging robots25 Machine learning on qualitative data23 Robots on the building site27 Civilizations & group psychology1 History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.