Your data
Rows with a missing forecast or outcome are skipped and counted.
Result
Reliability diagram (CORP)
recalibrated forecast (isotonic fit) where a calibrated forecaster's curve would fall, by resampling the diagonal; bars at the bottom: how many forecasts at each value
The score, split exactly
Score = miscalibration − discrimination + uncertainty. Lower scores are better; miscalibration is what recalibrating could save, discrimination what the forecasts know beyond the base rate.
| Score | Score | Miscalibration | Discrimination | Uncertainty | After recalibration |
|---|
Tests
| Test | Statistic | p-value | Reads as |
|---|
Binned view
| Forecast range | Forecasts | Mean forecast | Happened | Gap |
|---|
Recalibrate a new forecast
Isotonic recalibration follows the curve above (linear between its steps); Platt scaling uses the fitted calibration intercept and slope. Both are fitted on these data, so check them on forecasts they were not fitted to.
Method and citation
The reliability curve. The forecasts are sorted and the outcomes regressed on them under one constraint, that a higher forecast never gets a lower recalibrated value: isotonic regression, computed by the pool-adjacent-violators algorithm (Ayer et al. 1955), with equal forecasts pooled first. The resulting step function is the CORP reliability diagram of Dimitriadis, Gneiting & Jordan (2021): consistent (a calibrated forecaster's curve tends to the diagonal), optimally binned (the algorithm picks the bins), reproducible and based on pool-adjacent-violators. Unlike a binned chart it has no bin count to choose.
The decomposition. With S the mean score, x the forecasts, c their isotonic recalibration and r the base rate, S(x) = MCB − DSC + UNC, where MCB = S(x) − S(c), DSC = S(r) − S(c) and UNC = S(r). It holds exactly for the Brier score (Brier 1950) and for the log score, because the isotonic fit minimises every proper score at once. The binned Murphy (1973) split, reliability − resolution + uncertainty, is shown for comparison; with forecasts that vary inside a bin it leaves a remainder, reported as the within-bin term.
Is it more than chance? Outcomes are redrawn as if the forecasts were true (each a coin with that probability), the isotonic fit is redone, and the band shows where the middle 90% of those curves fall. The same draws give a p-value: the share of calibrated worlds with at least the observed miscalibration (this tool's addition, using CORP's MCB as the statistic). Alongside: Spiegelhalter's (1986) z test, the observed against expected count of events (calibration in the large), Cox's (1958) calibration intercept and slope from a logistic regression on the log-odds of the forecast, the binned expected and maximum calibration error (Naeini, Cooper & Hauskrecht 2015) and the AUC (Hanley & McNeil 1982), which measures ranking only.
Verified by tests in the source against an independent computation (eval/reference.py): on the example and two further cases, the Brier and log scores and both CORP decompositions match scikit-learn 1.9.1's isotonic regression, brier_score_loss and log_loss to 10⁻¹⁰; the isotonic fit matches value for value; the AUC matches roc_auc_score; the calibration intercept and slope match its unpenalised logistic regression to 10⁻⁶; and Spiegelhalter's and the observed/expected z match numpy with p-values from scipy 1.18.1.
BibTeX
@article{dimitriadis2021stable,
author = {Dimitriadis, Timo and Gneiting, Tilmann and Jordan, Alexander I.},
title = {Stable reliability diagrams for probabilistic classifiers},
journal = {Proceedings of the National Academy of Sciences},
volume = {118}, number = {8}, pages = {e2016191118}, year = {2021},
doi = {10.1073/pnas.2016191118}
}
@article{ayer1955empirical,
author = {Ayer, Miriam and Brunk, H. D. and Ewing, G. M. and Reid, W. T. and Silverman, Edward},
title = {An empirical distribution function for sampling with incomplete information},
journal = {The Annals of Mathematical Statistics},
volume = {26}, number = {4}, pages = {641--647}, year = {1955},
doi = {10.1214/aoms/1177728423}
}
@article{brier1950verification,
author = {Brier, Glenn W.},
title = {Verification of forecasts expressed in terms of probability},
journal = {Monthly Weather Review},
volume = {78}, number = {1}, pages = {1--3}, year = {1950},
doi = {10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2}
}
@article{murphy1973new,
author = {Murphy, Allan H.},
title = {A new vector partition of the probability score},
journal = {Journal of Applied Meteorology},
volume = {12}, number = {4}, pages = {595--600}, year = {1973},
doi = {10.1175/1520-0450(1973)012<0595:ANVPOT>2.0.CO;2}
}
@article{spiegelhalter1986probabilistic,
author = {Spiegelhalter, David J.},
title = {Probabilistic prediction in patient management and clinical trials},
journal = {Statistics in Medicine},
volume = {5}, number = {5}, pages = {421--433}, year = {1986},
doi = {10.1002/sim.4780050506}
}
@article{cox1958two,
author = {Cox, D. R.},
title = {Two further applications of a model for binary regression},
journal = {Biometrika},
volume = {45}, number = {3/4}, pages = {562--565}, year = {1958},
doi = {10.2307/2333203}
}
@inproceedings{naeini2015obtaining,
author = {Naeini, Mahdi Pakdaman and Cooper, Gregory F. and Hauskrecht, Milos},
title = {Obtaining well calibrated probabilities using {B}ayesian binning},
booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence},
volume = {29}, number = {1}, pages = {2901--2907}, year = {2015},
doi = {10.1609/aaai.v29i1.9602}
}
@article{hanley1982meaning,
author = {Hanley, James A. and McNeil, Barbara J.},
title = {The meaning and use of the area under a receiver operating characteristic ({ROC}) curve},
journal = {Radiology},
volume = {143}, number = {1}, pages = {29--36}, year = {1982},
doi = {10.1148/radiology.143.1.7063747}
}
Limits
- Calibration is not skill. Forecasting the base rate every time is perfectly calibrated and useless. Read miscalibration and discrimination together; the AUC and the discrimination term say how much the forecasts know.
- In-sample recalibration flatters. The isotonic and Platt maps are fitted to these outcomes, so the "after recalibration" score is optimistic. Fit them on one period and check them on the next.
- Independence is assumed. The tests and band treat outcomes as independent given the forecasts. Forecasts of the same storm, the same borrower or the same team over a season are often correlated, which makes the p-values too small.
- A shift can pass the shape test. A forecaster who is a few points too high everywhere can show little miscalibration on the curve while the observed/expected count says otherwise; on the example, the resampling test finds nothing (p = 0.289 with the default 500 resamples) and the count test does (p = 0.025). Read both.
- Size. Up to 200,000 forecasts; resampling 500 times takes a few seconds per 100,000.
- Not modelled: forecasts of more than two outcomes (multiclass or ordinal calibration), forecasts of a number (use conformal prediction intervals to check interval coverage) and calibration that drifts over time, beyond splitting the file by period yourself.
Questions
What does it mean for forecasts to be calibrated?
Of all the times the forecast said 70%, the event happened about 70% of the time, and the same for every other probability. Calibration is about honesty, not skill: always forecasting the base rate is calibrated but useless.
What is a CORP reliability diagram?
A reliability diagram whose curve comes from isotonic regression of the outcomes on the forecasts instead of from bins. Dimitriadis, Gneiting and Jordan (2021) showed it does not depend on a bin count, uses every forecast, and gives an exact split of the score into miscalibration, discrimination and uncertainty.
Why not just use expected calibration error (ECE)?
ECE depends on how many bins you choose and where their edges fall. On this page's NFL example it ranges from 0.019 to 0.043 depending on the bins, more than double, while the CORP miscalibration is one number. The tool reports ECE too, for comparison with papers that use it.
What do miscalibration, discrimination and uncertainty mean?
Uncertainty is the score of always forecasting the base rate. Discrimination is how much the recalibrated forecasts improve on that by telling events from non-events. Miscalibration is how much the forecasts lose by not being calibrated: the most an order-preserving recalibration could take off the score. Score equals miscalibration minus discrimination plus uncertainty, exactly.
What does the calibration slope say?
It is the slope of a logistic regression of the outcome on the log-odds of the forecast. A slope of 1 means the spread is right; below 1 the forecasts are too extreme (overconfident), above 1 too timid. The intercept shows whether they run high or low overall.
How many forecasts do I need?
At least 10, but a few hundred before the curve means much. The consistency band shows how far a truly calibrated forecaster's curve would wander with your sample size, so a wide band is the tool telling you the sample is small.
Is my data uploaded anywhere?
No. The file is read and analysed by JavaScript in this browser tab. There are no accounts, cookies or analytics vendors.
Example data: FiveThirtyEight's NFL Elo pre-game probabilities that the home team wins, with the results, for the 2,930 games of the 2010 to 2020 seasons that did not end in a tie, from the nfl_games.csv file of the fivethirtyeight/nfl-elo-game repository (MIT licence, copyright 2021 ABC News Internet Ventures), retrieved on 2026-11-16. In those seasons home teams won 56.5% of games against an average forecast of 58.4%; in 2020, a season played mostly in empty or limited-capacity stadiums, they won 50.0% against 58.5%.