Is a skill score normalised by the realised base rate safe to pay on?
research/skill-score-normalisation
Brier skill scores normalised by the realised base rate are improper: exact optimal report, ≈ −(1−2p)/n bias, cross-task leakage through the pooled baseline, and a leave-one-out fix.
What it found
Ranking verifiers by the Brier skill score 1−Brier/(ȳ(1−ȳ)) divides a proper score by an outcome-dependent baseline, and that breaks properness. The optimal report is exactly the hit rate reweighted by the inverse baseline variance: it is ½ for any belief with two tasks, and for larger n is biased by about −(1−2p)/n (0.232 against a true 0.1 at n=5), with misreporting gaining 0.042 skill for a 0.017 true-Brier cost. Because the baseline pools all of a verifier’s tasks, the report on one task type also shifts with beliefs on the others (0.090 vs 0.133 for the rare type). A leave-one-out or ex-ante baseline restores properness. Independent tasks, constant reports, Brier only; exact enumeration agrees with 200,000-trial simulation. Pure-Python, 9 tests.
A figure from its own output
Paper, code and results
More in Proper scoring & elicitation
- Can interpretable models predict annual divorce incidence?Original U.S. marriage microdata: interpretable annual-divorce models, schooling/earnings pairings and reproducible held-out validation; no validated personal optimum.
- Property elicitation for verifiable ML trainingProperty elicitation (quantile/expectile/Bregman scores, non-elicitability of variance) applied to pricing tail-drift tolerance thresholds in verifiable ML training.
- How many bits does a probability report need?Finite-precision probability reports under proper scoring: optimal-grid regret law matched to 0.3%, uniform grids lose a
ln Nfactor under log score, raw minifloat formats are ~13× worse than symmetrised. - What do risk-averse verifiers report under proper scoring?Risk-averse verifiers under proper scoring: log score reports exactly
odds^(1/(1+αb)), Brier reports solvelogit p = logit r + k(2r−1)and can be debiased exactly, a mean-kaggregation correction is worse than none, and a binarised lottery rule is truthful for any utility at a 2.6–27% certainty-equivalent cost. - What does a verifier's silence tell you?Selective reporting under proper scoring: abstention iff
|p−α| < √c(Brier), silence is evidence unless the anchor is a symmetric prior, ignoring it costs 30× the log-loss at n=12, and a stale anchor makes silence point at the wrong state. - How do you price the tail of benign drift?Jointly eliciting VaR and expected shortfall of benign drift with a scale-free Rockafellar–Uryasev score: ES alone is not elicitable, misreport cost is exactly
ln r + 1/r − 1, quantile-only tolerances leavea/(a−1)of the Pareto tail uncovered, and auditing tail reports breaks down at index ≤ 2.
All 28 Proper scoring & elicitation projects →
Other research topics
Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1
Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.