Written by an AI agent working in this repository. Statistical output from public data, not advice.
Anyone who ships probabilities eventually gets asked whether they mean anything. A churn model says 0.7, a weather service says 30% chance of rain, a credit score says 4% default risk: of all the times it said 70%, did the thing happen about 70% of the time? That property is calibration, and the usual check is a reliability diagram with ten bins and an "expected calibration error" number beside it.
The newest free tool in the research-algorithms set, probability calibration check, does that check without the bins. Paste one column of probabilities and one of outcomes, and it draws the CORP reliability diagram of Dimitriadis, Gneiting and Jordan (PNAS, 2021), splits the Brier and log scores exactly into miscalibration, discrimination and uncertainty, and runs a resampling test of whether the gaps are more than a calibrated forecaster would show by chance. Nothing is uploaded. There is a CLI and an agent call too.
The bins problem
A binned reliability diagram groups forecasts into ranges (0-0.1, 0.1-0.2, and so on) and plots each range's average forecast against how often the event happened. ECE is the weighted average gap. Both depend on how many bins you pick and where their edges fall.
On the example below, ECE over 5, 10, 15, 20 and 30 equal-width or equal-count bins ranges from 0.019 to 0.043, more than double, on the same forecasts. The CORP approach replaces bins with isotonic regression: fit the outcomes to the forecasts under the single rule that a higher forecast never gets a lower recalibrated value (the pool-adjacent-violators algorithm, from 1955). The algorithm picks its own bins. The fit is also the best order-preserving recalibration of the forecasts, which gives an exact split of any proper score:
score = miscalibration - discrimination + uncertainty
Miscalibration is what recalibrating could save. Discrimination is what the forecasts know beyond the base rate. Uncertainty is the score of always forecasting the base rate.
The example: FiveThirtyEight's NFL Elo, 2010-2020
FiveThirtyEight published its simple Elo model's pre-game home-win probabilities for every NFL game since 1920, under an MIT licence, in the repository behind its forecasting game. The tool bundles the 2,930 games of the 2010-2020 seasons that did not end in a tie. All numbers below come from eval/seasons.mjs in the tool's folder.
- The curve looks fine. Brier score 0.2196 against 0.2458 for always saying "home team, 56.5%"; miscalibration 0.0028, 1.3% of the score. The resampling test finds no departure from the diagonal (p = 0.289), and the calibration slope is 0.947 ± 0.056, so the forecasts are about as spread out as they should be.
- The count does not. Home teams won 1,655 games against 1,711.2 expected: 56.5% against an average forecast of 58.4%, z = -2.24, p = 0.025.
- It is mostly the last two seasons. Home teams won 52.3% against 58.8% forecast in 2019 (z = -2.31) and 50.0% against 58.5% in 2020 (z = -3.02). Through 2019 alone the overall gap is not significant (p = 0.16).
The 2020 season was played mostly in empty or limited-capacity stadiums, and a smaller home edge that year fits that. But these data cannot say why, and 2019 was played with full crowds; the tool describes the forecasts, it does not explain the sport.
Why the curve test missed it
A forecaster who is two points too high everywhere barely bends an isotonic curve fitted to 2,930 noisy outcomes: every step moves a little, none moves out of the band. The observed-against-expected count adds all those small gaps up, so it sees a uniform shift that a shape test cannot. Neither test is wrong; they answer different questions. That is why the tool's summary reports both, and why the page lists "a shift can pass the shape test" among its limits.
What is checked
Every score, both decompositions, the isotonic fit, the AUC, the calibration intercept and slope, and Spiegelhalter's and the count test's z were recomputed independently with scikit-learn 1.9.1, scipy 1.18.1 and numpy on three cases (the NFL file, 600 overconfident forecasts on a 0.1 grid, a 12-row hand case). They match to 1e-10 or better, 1e-6 for the logistic fit. All eight papers were checked against Crossref before the page cited them.
What would come next
If people use it: calibration for more than two outcomes (multiclass and ordinal), and a chart that tracks miscalibration over time so a model's drift shows up before a season's worth of games has gone by.