Can audits be predictable without corrupting scores?

What it found

If truth arrives only on audited tasks and the audit rate depends on the report (say g(r)=a+br), “score when audited” is no longer proper: the verifier shades by about −bp(1−p)/(2(a+bp)) and, for steep b, hides at r=0 (Brier excess 0.25 at p=0.5). Weighting the audited loss by 1/g(r) is exactly proper for any audit rule, at variance E[B²]/g−L² and payout cap 1/g_min; the Neyman audit allocation minimises variance but gains only 3–11% over uniform. Stylised.

A figure from its own output

Paper, code and results

• README and method
• White paper
• Experiments and results
• Code
• Tests

More in Proper scoring & elicitation

• Can a calibration test certify a verifier?Calibration tests are gameable: an information-free Foster–Vohra/Blackwell hedger keeps expected ECE ≤ √(K+1)(1/(2K)+1/√T) against any outcome sequence (measured 0.016, Brier 0.254 vs 0 informed), so slashing must use a resolution-sensitive score; a Brier-gap test detects it in 100/100 runs.
• Can a protocol pay workers to report how often honest replicas agree?Eliciting the chance two honest runs of a nondeterministic op agree (Γ=Σp²) from run pairs: proper with exact regret, one run provably cannot, all-pairs U-statistic variance ~¼ of disjoint pairs, fleet of H hardware classes false-slashes strict equality at 1−H^{1−k}, colluders' blind spot at ε=2Γ/(1+Γ).
• Is a skill score normalised by the realised base rate safe to pay on?Brier skill scores normalised by the realised base rate are improper: exact optimal report, ≈ −(1−2p)/n bias, cross-task leakage through the pooled baseline, and a leave-one-out fix.
• Can interpretable models predict annual divorce incidence?Original U.S. marriage microdata: interpretable annual-divorce models, schooling/earnings pairings and reproducible held-out validation; no validated personal optimum.
• Property elicitation for verifiable ML trainingProperty elicitation (quantile/expectile/Bregman scores, non-elicitability of variance) applied to pricing tail-drift tolerance thresholds in verifiable ML training.
• How many bits does a probability report need?Finite-precision probability reports under proper scoring: optimal-grid regret law matched to 0.3%, uniform grids lose a ln N factor under log score, raw minifloat formats are ~13× worse than symmetrised.

All 28 Proper scoring & elicitation projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.