How do you price the tail of benign drift?

What it found

A drift tolerance is a quantile, but compensation depends on expected shortfall, which is not elicitable alone (a mixture moves ES 1.6→2.24). The scale-free score ln e + (v + (x−v)₊/(1−α))/e − 1 elicits (VaR, ES) jointly with exact excess law; misreporting ES by a factor r costs ln r + 1/r − 1 (0.307 at ½, 0.193 at 2), matched to quadrature within 2·10⁻⁵. A quantile-only tolerance leaves ES/VaR = a/(a−1) uncovered under Pareto drift (6× at a=1.2); auditing needs ~2,000 samples for a 20% understatement at index 4 and the CLT plan fails for index ≤ 2. Stylised.

A figure from its own output

Paper, code and results

• README and method
• White paper
• Experiments and results
• Code
• Tests

More in Proper scoring & elicitation

• Should you pay verifiers for their calibration?Paying verifiers for resolution, not calibration: raw Brier/log excess is exactly E[(r-p)²]/E[KL(p‖r)], Brier is bounded but log is not, log-score selection reverses for overconfident high-resolution verifiers (a ≥ 2.49), and n-sample recalibration costs tr(HI⁻¹)/2n (0.264/n).
• Should a verifier's continuous drift report be scored by CRPS or log score?CRPS vs log score for continuous drift reports: exact Gaussian closed forms, overconfidence capped at 0.234σ under CRPS (unbounded under log), 1-Lipschitz outcome sensitivity, plug-in cost 5σ/(8√π n), and finite score variance for tails ν>2 vs ν>4.
• Can a verifier be paid for reporting drift variance?Eliciting drift variance from two independent re-executions: D=(y1−y2)²/2 unbiased with Var=σ⁴(κ+1)/2, exact Brier and scale-free misreport costs (1/λ+ln λ−1), detection needs ≈2z²(κ+1)/(λ−1)² tasks, disjoint pairs 1.8× worse than the sample variance, heavy tails break m^{−1/2} concentration.
• What can be elicited from several independent draws of benign drift?Eliciting drift *variance* from pairs of independent draws: exact pair score (excess loss (r−σ²)²), degree table (mean² 2, third moment 3, σ⁴ 4), audit cost ((z_α+z_β)/ρ)²(κ+1)/2 pairs, robust Gini score beats it for heavy tails, all-pairs pooling saves (κ+1)/(κ−1), correlated draws bias the target to σ²(1−ρ).
• Can the log score be made payable?Bounded-payment log score by quadratic tangent extension: strictly proper, log-identical on [ε,1−ε], worst-case payment exactly ln((1−ε)/ε)+1/(1−ε), 63× Brier's curvature per range at p=10⁻³, crossover p(1−p)=1/(2R), best cutoff ε=p.
• Which proper score should verifiers face with K outcomes?K-ary proper scores: exact worst-case regrets (1−2p_min+‖p‖² Brier, ‖p‖−p_min spherical), spherical beats Brier by exactly √K per payment range at a uniform truth, and both bounded scores nearly ignore a dropped rare class (1e-4 vs 0.15 for log).

All 28 Proper scoring & elicitation projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.