Proper scoring & elicitation

Quantile, expectile and Bregman scoring rules, a check that variance is not elicitable alone but is jointly, and a study of pricing benign-drift thresholds for replica verification.

Regret of reporting from a finite grid under Brier and log scores: a high-resolution law for the optimal grid (matched to 0.3%), a ln N loss for uniform grids under log score, and a 13× penalty for raw minifloat formats versus symmetrised ones. Stylised.

Proper scoring rules are truthful only for risk-neutral agents. For a CARA verifier the log score yields reported odds exactly equal to true odds^(1/(1+k)), k = αb; the Brier score yields logit p = logit r + k(2r−1), which the principal can invert exactly. Summing reported logits under-extremises (log-loss 0.332 vs 0.214 at k=2); correcting with the mean k under heterogeneous verifiers is worse than not correcting; a log-score report needs true belief 0.964 to clear a 0.9 accept threshold at k=0.5. A binarised lottery rule is truthful for every increasing utility at a 2.6–27% certainty-equivalent cost. Stylised.

When reporting costs c, a truthful verifier paid the score gain over an anchor α abstains exactly when |p−α| < √c (Brier) or KL(p‖α) < c (log), so silence is an interval of signals and carries evidence unless the anchor is a symmetric prior. At prior 0.1 the outcome rate among reporters is 0.57; an aggregator that drops silent verifiers has log-loss 0.62 against 0.02 for one that uses silence at n=12 (it worsens as verifiers are added); a stale anchor makes silence point at the wrong state. Quadrature and seeded Monte Carlo. Stylised.

A drift tolerance is a quantile, but compensation depends on expected shortfall, which is not elicitable alone (a mixture moves ES 1.6→2.24). The scale-free score ln e + (v + (x−v)₊/(1−α))/e − 1 elicits (VaR, ES) jointly with exact excess law; misreporting ES by a factor r costs ln r + 1/r − 1 (0.307 at ½, 0.193 at 2), matched to quadrature within 2·10⁻⁵. A quantile-only tolerance leaves ES/VaR = a/(a−1) uncovered under Pareto drift (6× at a=1.2); auditing needs ~2,000 samples for a 20% understatement at index 4 and the CLT plan fails for index ≤ 2. Stylised.

A verifier reporting expit(aL+b) for the true posterior logit L is overpaid-for-error by exactly E[(r−p)²] under Brier and E[KL(p‖r)] under log score. Brier is bounded (a hard 0/1 report still beats the prior at μ=1) but log is not: raw log score ranks a higher-resolution verifier first only for slope a in (0.264, 2.49), so at a=4 a weaker calibrated verifier wins, while recalibrated scores never reverse. Recalibrating on n outcomes costs about tr(HI⁻¹)/2n excess Brier (0.264/n, simulation within ~10% for n≥100), beating raw pay beyond n≈20–80. Quadrature and seeded simulation. Stylised.

For Gaussian reports N(m,s²) both rules have exact expected scores. Overconfidence costs at most σ(√2−1)/√π = 0.234σ under CRPS but is unbounded under log (4995 nats at s=0.01σ); mean errors cost linearly versus quadratically; CRPS is 1-Lipschitz in the realised outcome (log slope 75/σ in our example); a plug-in fit from n samples costs 5σ/(8√πn) versus 1/n; and per-round score variance is finite for Student-t tails ν>2 (CRPS) versus ν>4 (log), which matters for sequential tests. Gaussian family, one verifier. Stylised.

Not from one re-execution (variance’s level sets are not convex), but two independent ones suffice: D=(y₁−y₂)²/2 is unbiased with variance σ⁴(κ+1)/2. Brier excess is exactly (r−σ²)²; the scale-free loss D/r+ln r has excess 1/λ+ln λ−1, so halving the report costs 0.307 and doubling 0.193. Catching a 10% misreport at 95% takes about 2–4 thousand tasks (2z²(κ+1)/(λ−1)², simulated 0.93–0.97); disjoint pairs are 1.8× worse than the sample variance at n=10; with Pareto tail index 3 the error decays like m^−1/3, not m^−1/2. Stylised.

The pair score (r−(y₁−y₂)²/2)² has excess loss exactly (r−σ²)², and kernels for mean², the third central moment and σ⁴ need 2, 3 and 4 draws. Detecting a variance under-report takes ((z_α+z_β)/ρ)²(κ+1)/2 pairs (Monte Carlo power 0.70–0.79 against a nominal 0.8); scoring |y₁−y₂| beats the variance score for Laplace and t₅; all-pairs pooling cuts payment variance by (κ+1)/(κ−1); correlated draws make the score elicit σ²(1−ρ). Stylised distributions, no measured drift.

Log score is ideal for rare faults but a confident wrong report costs unbounded stake. Extending the log generator past [ε,1−ε] by its second-order Taylor polynomial gives a strictly proper score, identical to log up to a constant inside, with worst-case payment exactly ln((1−ε)/ε)+1/(1−ε). Per unit of payment range it has 63× Brier’s curvature at p=10⁻³ but loses to Brier above a closed-form crossover (≈0.07), and ε=p is the best cutoff for truth p.

For K-ary reports the Brier, spherical and log scores have exact regrets and exact worst cases (1−2p_min+‖p‖² for Brier, ‖p‖−p_min for spherical). Per unit of payment range the spherical score gives exactly √K times Brier’s incentive at a uniform truth, but both bounded scores nearly ignore a dropped rare class: at a 1% class the loss is 1.3×10⁻⁴ (Brier) and 1.0×10⁻⁴ (spherical) against 0.15 for log. Multiclass follow-up to tangent-log.

Extending x ln x below a cutoff ε by its second-order Taylor polynomial gives a strictly proper K-class score that equals −ln r_y whenever every r_j≥ε and has payment range exactly 1−ln ε, the same for any K. Per unit of range it has 3× Brier’s rare-class incentive at a 5% class and 63× at 0.1%, the best cutoff is the rarest class’s mass, and a verifier that zeroes out a 1% class loses a closed-form 0.005 (Brier 1.3×10⁻⁴, log unbounded). Closes the gap left open in categorical-scores.

A verifier who reports only the order of K classes can be paid with a score linear in the outcome indicator. It is strictly proper, and swapping two classes at adjacent ranks costs exactly (p_i−p_j)(w_r−w_r+1), linear in the gap. Per unit of payment range that beats a Brier-derived ranking whenever the gap is below 1/(K−1) (2.5× at gap 0.1, 250× at 0.001, K=5). Equal spacing maximises the smallest incentive, ranking everything costs K−1 times the mode, a top-k set costs the same as the mode, and the sample size to detect a swap is identical to Brier’s.

For ordered fault severities the ranked probability score is Brier on the cumulative distribution: strictly proper with exact regret Σ(R_k−P_k)², so moving mass ε a distance d costs exactly dε² where Brier charges 2ε² for every d. Pooling adjacent classes costs exactly half of Brier’s, and every threshold probability is within √regret of the truth. The price: per unit of payment range RPS gives Brier’s incentive only for a maximal-distance shift (1/(K−1) for adjacent ones). Extends categorical-scores.

Verifiers forecast a tolerance threshold (VaR) and the mean drift beyond it (expected shortfall, ES). ES alone is not elicitable (a 50/50 mixture of two ES = 10 laws has ES 12.5), but the 0-homogeneous Fissler–Ziegel score ln e − 1 + (v + (y−v)₊/(1−τ))/e is minimised exactly at (VaR, ES), with exact closed-form excess scores matched to 10⁻¹⁰. Pinball loss scores a forecaster with ES off by 30% identically to the truth; the joint score separates them with about 0.6–1.2k samples at τ = 0.9–0.95. Stylised. Follows up property-elicitation-verification.

When some workers cheat, drift is a mixture and the mean, which squared loss elicits, lands where almost no task lies (at a 30% cheater 6σ away, the mean’s window holds 0.15 of the mass versus 0.48 at the mode). The mode is not elicitable, but the centre of the width-2α window with the most mass is, with the bounded loss 1[|y−r|>α]. Regret is an exact window-mass difference; the best window is α=σ for normal drift and γ/√3 for Cauchy (no mean exists); an equal bimodal mixture has one α-mode iff α exceeds a closed-form root α*; and under 10–40% contamination the sample α-mode has ~0 bias while the mean is off by 5ε, at a 3.7× RMSE cost when clean.

The Winkler interval score is strictly proper for the pair of quantiles, with closed-form regret and an optimal expected score of exactly 4sφ(z)/α. Misreporting the width by the same factor costs more when narrow at 95% coverage but more when wide at 80%, so there is no universal rule that overconfidence is worse. Under Cauchy drift the scores have no mean, yet paired differences are bounded, so evaluate verifiers against a benchmark report.

The Winkler interval score is the sum of two pinball losses: proper for the equal-tailed quantile interval with exact regret (2/α)[PR(l,α/2)+PR(u,1−α/2)], finite even for Cauchy drift where the expected score does not exist, but its payment grows without bound in the miss distance. A width-plus-miss loss (u−l)+λ1[miss] is bounded by width+λ and elicits the density level set, the shortest interval for its coverage: 22–31% narrower than equal-tailed on lognormal(0,1) drift. The price is statistical: to detect a 20% understatement of a normal tolerance it needs 3–48× more tasks for α≥0.1 and about the same at α=0.01.

For a report of an expected drift r ∈ ℝ^d with noise covariance Σ, every quadratic Bregman score −(y−r)^TA(y−r) is proper and charges a misreport D exactly D^TAD. Against a trusted reference the per-task gap has exact mean D^TAD and variance 4D^TAΣAD for any noise law, and Cauchy–Schwarz shows A=Σ⁻¹ has the best signal to noise for every deviation at once, needing 4z²/(D^TΣ⁻¹D) tasks. An isotropic score has efficiency at least 4κ/(1+κ)² (Kantorovich), attained by a deviation split between the extreme eigendirections: 25× more tasks at condition number 100, and power 0.29 against 0.80 at 48 tasks in simulation. Per-layer normalisation alone barely helps; with Σ estimated from m reference draws, efficiency is 0.61 at m=2d and 0.95 at m=100. Quadratic scores, trusted reference, finite covariance.

Exact calibration functions turning a proper-score gap into a certified cap on accept/audit/slash regret: Brier transfers as ε², log as a KL, polyhedral hinge linearly but without proper elicitation, and an audit action tightens the guarantee; the certificate is never violated in simulation and is attained by a two-point verifier. Stylised.

Reject-option surrogates for accept/reject/escalate decisions: the generalised-hinge conditional risk embeds the Bayes action at its kinks, regret transfers linearly with constant exactly 2d (grid-verified), logistic transfer degrades as 2d(1−d)/δ, and a negative result shows logistic-plus-thresholds still learns faster in scalar SGD. Stylised.

If a report of “fault” gets the job rejected and the outcome is seen only on acceptance, scoring observed jobs is improper: verifiers shade under the threshold to get scored, and every job is accepted (1.1–2.8× optimal decision loss at thresholds 0.7–0.2). A reserve payment for rejection makes truth optimal iff the score’s entropy minimum τ_s is at least the threshold τ_d, and then it is unique, 1+τ_d²−2τ_dτ_s. Plain Brier (τ_s=½) therefore fails above τ_d=½; recentring the score by a term linear in the outcome fixes every threshold, cheapest at τ_s=τ_d with rent τ³/3. Exploration with inverse-propensity payment is also proper but multiplies payment variance by about 1/ε. Stylised, single verifier.

When verifiers are paid against a judge whose label flips with rates e₀, e₁, any proper score elicits the noisy probability e₀+γp (γ=1−e₀−e₁), the Brier incentive shrinks by γ², and rankings of truthful verifiers reverse at η*=0.25. The backward-corrected surrogate score is exactly unbiased for the true loss, at a 1/γ payment spread and (balanced case) 1/γ² more evidence; misestimated rates produce a closed-form affine bias whose cost falls as 1/m in gold checks. Stylised.

If truth arrives only on audited tasks and the audit rate depends on the report (say g(r)=a+br), “score when audited” is no longer proper: the verifier shades by about −bp(1−p)/(2(a+bp)) and, for steep b, hides at r=0 (Brier excess 0.25 at p=0.5). Weighting the audited loss by 1/g(r) is exactly proper for any audit rule, at variance E[B²]/g−L² and payout cap 1/g_min; the Neyman audit allocation minimises variance but gains only 3–11% over uniform. Stylised.

No. A hedger with no information at all keeps its expected calibration error below √(K+1)(1/(2K)+1/√T) against any outcome sequence, even one that sees its forecast distribution: ECE 0.016 at T=2000 while its Brier score is 0.254 against 0 for a verifier who knows the rule. A lazy constant forecaster is slashed; the hedger passes every calibration threshold tried, but a Brier-gap test against an informed benchmark catches it in 100/100 runs.

Yes, from two runs and no more: the squared error against the agreement indicator is proper for the collision probability Γ=Σp_i² with exact regret (r−Γ)², and one run provably cannot (the level sets are not convex). The all-pairs U-statistic has exact variance and needs about a quarter of the runs of disjoint pairs (variance ratio 0.27 at n=100). A fleet of H equal hardware classes has Γ=1/H, so strict equality over k honest replicas false-slashes with probability 1−H^1−k. Colluders returning one common wrong value leave agreement unchanged at ε=2Γ/(1+Γ), so Γ cannot be used alone. Runs are i.i.d. and float32 fleets are simulated summation orders.

Ranking verifiers by the Brier skill score 1−Brier/(ȳ(1−ȳ)) divides a proper score by an outcome-dependent baseline, and that breaks properness. The optimal report is exactly the hit rate reweighted by the inverse baseline variance: it is ½ for any belief with two tasks, and for larger n is biased by about −(1−2p)/n (0.232 against a true 0.1 at n=5), with misreporting gaining 0.042 skill for a 0.017 true-Brier cost. Because the baseline pools all of a verifier’s tasks, the report on one task type also shifts with beliefs on the others (0.090 vs 0.133 for the rare type). A leave-one-out or ex-ante baseline restores properness. Independent tasks, constant reports, Brier only; exact enumeration agrees with 200,000-trial simulation. Pure-Python, 9 tests.

Original ACS, NSFG and NLSY79 microdata analyses separate annual incidence from prevalence. A respondent-held-out enriched additive couple model reduces Poisson loss by about 3.4%; more flexible trees do not improve on it. Includes schooling/earnings pairings, direct contrasts, duration and first-birth checks, trained models and a portable report. A browser app runs the five models on two profiles to give year-by-year divorce odds with explanations, model disagreement and out-of-range flags. Older cohort; observational, not a personal or causal forecast.