What can be elicited from several independent draws of benign drift?

What it found

The pair score (r−(y₁−y₂)²/2)² has excess loss exactly (r−σ²)², and kernels for mean², the third central moment and σ⁴ need 2, 3 and 4 draws. Detecting a variance under-report takes ((z_α+z_β)/ρ)²(κ+1)/2 pairs (Monte Carlo power 0.70–0.79 against a nominal 0.8); scoring |y₁−y₂| beats the variance score for Laplace and t₅; all-pairs pooling cuts payment variance by (κ+1)/(κ−1); correlated draws make the score elicit σ²(1−ρ). Stylised distributions, no measured drift.

A figure from its own output

Paper, code and results

• README and method
• White paper
• Experiments and results
• Code
• Tests

More in Proper scoring & elicitation

• Can the log score be made payable?Bounded-payment log score by quadratic tangent extension: strictly proper, log-identical on [ε,1−ε], worst-case payment exactly ln((1−ε)/ε)+1/(1−ε), 63× Brier's curvature per range at p=10⁻³, crossover p(1−p)=1/(2R), best cutoff ε=p.
• Which proper score should verifiers face with K outcomes?K-ary proper scores: exact worst-case regrets (1−2p_min+‖p‖² Brier, ‖p‖−p_min spherical), spherical beats Brier by exactly √K per payment range at a uniform truth, and both bounded scores nearly ignore a dropped rare class (1e-4 vs 0.15 for log).
• Can the multiclass log score be made payable?Bounded multiclass log score by quadratic tangent extension of x ln x: log-identical when all r_j≥ε, payment range exactly 1−ln ε for any K, 63× Brier's rare-class incentive per range at ρ=10⁻³ with best cutoff ε=p_min, and a closed-form bounded cost for dropping a rare class.
• What does it cost to pay for a ranking instead of probabilities?Eliciting class rankings with linear rank-weight scores: strictly proper, adjacent-swap regret exactly (p_i−p_j)(w_r−w_{r+1}), beats Brier-derived rankings per unit range below gap 1/(K−1) (250× at gap 10⁻³), equal spacing is optimal, top-k set costs the same as the mode, detection sample size unchanged.
• Should severity reports be scored with the order in mind?Ranked probability score for ordered severities: exact regret Σ(R_k−P_k)², shift over distance d costs dε² (Brier 2ε²), pooling adjacent classes costs half of Brier's, threshold errors ≤ √regret, and per range RPS never beats Brier's incentive.
• Can the tail of benign drift be elicited with a proper score?Jointly eliciting VaR and expected shortfall of benign drift: 0-homogeneous Fissler–Ziegel score minimised at (VaR, ES) with profile ln ES, ES alone non-elicitable (mixture 12.5 vs 10), exact regret formulas, ES-understatement of 30% detectable with ~1k samples at τ=0.95 while pinball is blind.

All 28 Proper scoring & elicitation projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.