Can the tail of benign drift be elicited with a proper score?

What it found

Verifiers forecast a tolerance threshold (VaR) and the mean drift beyond it (expected shortfall, ES). ES alone is not elicitable (a 50/50 mixture of two ES = 10 laws has ES 12.5), but the 0-homogeneous Fissler–Ziegel score ln e − 1 + (v + (y−v)₊/(1−τ))/e is minimised exactly at (VaR, ES), with exact closed-form excess scores matched to 10⁻¹⁰. Pinball loss scores a forecaster with ES off by 30% identically to the truth; the joint score separates them with about 0.6–1.2k samples at τ = 0.9–0.95. Stylised. Follows up property-elicitation-verification.

A figure from its own output

Paper, code and results

• README and method
• White paper
• Experiments and results
• Code
• Tests

More in Proper scoring & elicitation

• Can we elicit the mode of a mixture of honest and cheating drift?Eliciting the modal drift with the window loss 1[|y−r|>α]: exact window-mass regret, best width α=σ (normal) / γ/√3 (Cauchy), closed-form critical width α* above which bimodal drift has one α-mode, and 0 bias against contamination where the mean moves 5ε.
• Can a verifier be paid for a tolerance interval, not a point?Companion to interval-elicitation: Winkler interval score for drift-tolerance intervals: strictly proper for the α/2 and 1−α/2 quantiles, closed-form regret, optimal expected score 4sφ(z)/α, width-misreport costs whose asymmetry flips with the level, and Cauchy drift where scores have no mean but paired differences are bounded.
• How should a tolerance interval be paid for?Paying for a drift tolerance interval: Winkler interval score is proper for the equal-tailed pair with exact regret (finite for Cauchy) but unbounded payment; the bounded width-plus-miss loss elicits the shortest interval (22–31% narrower on lognormal drift) at a 3–48× larger detection sample for α≥0.1.
• How should a verifier's vector report be scored?Scoring a verifier's vector report: the precision score Σ⁻¹ is uniformly most detectable among quadratic Bregman scores (Cauchy–Schwarz), an isotropic score can need (1+κ)²/(4κ) times more tasks (25× at κ=100, power 0.29 vs 0.80), diagonal normalisation barely helps, estimated precision reaches 0.61 efficiency at m=2d draws.
• What does a verifier's score gap cost in decisions?From a verifier's excess proper score to a certified cap on accept/audit/slash decision regret: exact calibration functions (Brier ε², log KL), linear transfer only for non-proper hinge, audit action tightens the guarantee.
• Which surrogate should train a verifier that can escalate?Polyhedral surrogates for accept/reject/escalate verifiers: minimiser embeds the Bayes action, linear regret transfer with constant exactly 2d, logistic transfer ratio 2d(1-d)/δ, and a negative SGD result.

All 28 Proper scoring & elicitation projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.