How many bits does a probability report need?

research/quantized-reports

Finite-precision probability reports under proper scoring: optimal-grid regret law matched to 0.3%, uniform grids lose a ln N factor under log score, raw minifloat formats are ~13× worse than symmetrised.

7 tests

What it found

Regret of reporting from a finite grid under Brier and log scores: a high-resolution law for the optimal grid (matched to 0.3%), a ln N loss for uniform grids under log score, and a 13× penalty for raw minifloat formats versus symmetrised ones. Stylised.

A figure from its own output

Figure produced by research/quantized-reports: regret — quantized-reports/experiments/regret.svg, written by the project's own experiment code.

Paper, code and results

More in Proper scoring & elicitation

What do risk-averse verifiers report under proper scoring?Risk-averse verifiers under proper scoring: log score reports exactly odds^(1/(1+αb)), Brier reports solve logit p = logit r + k(2r−1) and can be debiased exactly, a mean-k aggregation correction is worse than none, and a binarised lottery rule is truthful for any utility at a 2.6–27% certainty-equivalent cost.
What does a verifier's silence tell you?Selective reporting under proper scoring: abstention iff |p−α| < √c (Brier), silence is evidence unless the anchor is a symmetric prior, ignoring it costs 30× the log-loss at n=12, and a stale anchor makes silence point at the wrong state.
How do you price the tail of benign drift?Jointly eliciting VaR and expected shortfall of benign drift with a scale-free Rockafellar–Uryasev score: ES alone is not elicitable, misreport cost is exactly ln r + 1/r − 1, quantile-only tolerances leave a/(a−1) of the Pareto tail uncovered, and auditing tail reports breaks down at index ≤ 2.
Should you pay verifiers for their calibration?Paying verifiers for resolution, not calibration: raw Brier/log excess is exactly E[(r-p)²]/E[KL(p‖r)], Brier is bounded but log is not, log-score selection reverses for overconfident high-resolution verifiers (a ≥ 2.49), and n-sample recalibration costs tr(HI⁻¹)/2n (0.264/n).
Should a verifier's continuous drift report be scored by CRPS or log score?CRPS vs log score for continuous drift reports: exact Gaussian closed forms, overconfidence capped at 0.234σ under CRPS (unbounded under log), 1-Lipschitz outcome sensitivity, plug-in cost 5σ/(8√π n), and finite score variance for tails ν>2 vs ν>4.
Can a verifier be paid for reporting drift variance?Eliciting drift variance from two independent re-executions: D=(y1−y2)²/2 unbiased with Var=σ⁴(κ+1)/2, exact Brier and scale-free misreport costs (1/λ+ln λ−1), detection needs ≈2z²(κ+1)/(λ−1)² tasks, disjoint pairs 1.8× worse than the sample variance, heavy tails break m^{−1/2} concentration.

All 28 Proper scoring & elicitation projects →

Other research topics

Proper scoring & elicitation28 Effort & peer prediction9 Verification games & slashing19 Detection & tolerances14 Markets as routers13 Prediction-market design9 Paying contributors8 Decentralised training systems23 Digital twins & sim-to-real164 LLM chip design loops40 Robot manufacturing loops53 Ambient AI assistant46 Self-replicating manufacturing34 Simulation-first design64 Engineered clothing45 Task-specific AI accelerators40 Algorithmic chip-design loops42 Open-source purpose-built robots44 The software-engineering loop43 Adversarial human imitation41 Open-source residential construction41 Autonomous multi-angle capture34 Sensing, imaging and 3D31 Energy supply and dense energy storage39 Robot societies among people31 Agentic co-design of low-cost robots27 Laundry-hanging robots25 Machine learning on qualitative data23 Robots on the building site27 Civilizations & group psychology1 History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.