Your data
Rows with a missing value in a column you use are dropped and counted. Non-numeric columns are left out of the features.
The rows are shuffled with the seed, then split three ways: training (fits the model), calibration (sets the interval width) and test (checks coverage honestly). The rest of the rows after test and calibration are training rows.
Result
| Method | Test coverage | Covered | Mean width | Median width | Calibrated quantile | Rank k of n | Avg coverage over splits | Avg width over splits |
|---|
The calibrated quantile is in the units of each method's score: target units for split conformal, multiples of the fitted error spread for locally weighted, and an outward shift of the fitted quantile band (target units) for CQR.
Test rows, sorted by prediction
prediction intervalpredictionactual, insideactual, outside
Intervals for new rows
| Row | Method | Lower | Prediction | Upper |
|---|
New rows use the same fitted and calibrated predictor that was evaluated above (same seed and split), so the guarantee carries over if the new rows come from the same population.
Method and citation
Each method fits a model on the training rows, computes a conformity score on each of the n calibration rows, and takes the k-th smallest score with k = ceil((n+1)(1-alpha)), where 1−alpha is the coverage. That rank, and not the plain 90th percentile, is what makes the guarantee hold for small n: under exchangeability a new score is at most the k-th smallest with probability exactly k/(n+1) when scores have no ties. If k > n the interval is infinite and the tool says so.
- Split conformal (Lei, G'Sell, Rinaldo, Tibshirani & Wasserman 2018, Algorithm 2). Model: ridge regression on standardised features, penalty chosen by 5-fold cross-validation on the training rows. Score |y − μ(x)|. Interval μ(x) ± Q: the same width for every row.
- Locally weighted split conformal (Lei et al. 2018, Section 5.2). A second ridge regression ρ(x) predicts the size of the error from out-of-fold training residuals (floored at 10% of their mean so it stays positive). Score |y − μ(x)| / ρ(x). Interval μ(x) ± Q·ρ(x): wider where errors are larger.
- Conformalized quantile regression, CQR (Romano, Patterson & Candès 2019). Linear quantile regressions at alpha/2 and 1−alpha/2 (Koenker & Bassett 1978), solved exactly as a linear program with a bounded-variable simplex. Score max(qlo(x) − y, y − qhi(x)). Interval [qlo(x) − Q, qhi(x) + Q]. Crossed quantiles are swapped for that row. Training sets above 4,000 rows are subsampled (seeded) for the quantile fits.
The evaluation uses the held-out test rows, which played no part in fitting or calibration, then repeats the whole procedure on 20 further random splits (seeded) and reports the averages. Over random splits of the same data the expected coverage is at least the target, so the average is the fair check; a single split moves by several points by chance.
Verified by tests in the source: the rank k against exact integer arithmetic for every n up to 400; coverage averaged over 200 seeded simulations with normal, skewed and two kinds of heteroskedastic noise falls within [1−alpha, 1−alpha+1/(n+1)] to within 0.01 for every method; ridge against the closed-form normal equations; quantile regression against a brute-force search over every exact-fit line and plane, and against random perturbations on the diabetes data.
BibTeX
@article{lei2018distribution,
author = {Lei, Jing and G'Sell, Max and Rinaldo, Alessandro and Tibshirani, Ryan J. and Wasserman, Larry},
title = {Distribution-Free Predictive Inference for Regression},
journal = {Journal of the American Statistical Association},
volume = {113}, number = {523}, pages = {1094--1111}, year = {2018},
doi = {10.1080/01621459.2017.1307116}
}
@inproceedings{romano2019conformalized,
author = {Romano, Yaniv and Patterson, Evan and Cand{\`e}s, Emmanuel J.},
title = {Conformalized Quantile Regression},
booktitle = {Advances in Neural Information Processing Systems 32},
year = {2019},
eprint = {1905.03222}, archivePrefix = {arXiv}
}
@book{vovk2005algorithmic,
author = {Vovk, Vladimir and Gammerman, Alex and Shafer, Glenn},
title = {Algorithmic Learning in a Random World},
publisher = {Springer}, year = {2005},
doi = {10.1007/b106715}
}
@article{koenker1978regression,
author = {Koenker, Roger and Bassett, Gilbert, Jr.},
title = {Regression Quantiles},
journal = {Econometrica},
volume = {46}, number = {1}, pages = {33--50}, year = {1978},
doi = {10.2307/1913643}
}
@article{hoerl1970ridge,
author = {Hoerl, Arthur E. and Kennard, Robert W.},
title = {Ridge Regression: Biased Estimation for Nonorthogonal Problems},
journal = {Technometrics},
volume = {12}, number = {1}, pages = {55--67}, year = {1970},
doi = {10.1080/00401706.1970.10488634}
}
@article{efron2004least,
author = {Efron, Bradley and Hastie, Trevor and Johnstone, Iain and Tibshirani, Robert},
title = {Least Angle Regression},
journal = {The Annals of Statistics},
volume = {32}, number = {2}, pages = {407--499}, year = {2004},
doi = {10.1214/009053604000000067}
}
This is an independent implementation from the papers' mathematics; no code from the authors or from other libraries was copied.
Example data
"Load example" uses the diabetes data from Efron, Hastie, Johnstone & Tibshirani (2004), "Least Angle Regression", The Annals of Statistics 32(2): 442 patients, ten baseline measurements (age, sex, body mass index, blood pressure and six blood serum measurements S1–S6), and Y, a quantitative measure of disease progression one year later. File from the authors' companion page (www4.stat.ncsu.edu/~boos/var.select/diabetes.tab.txt), retrieved 2026-10-20; distributed publicly by the authors for research and teaching and redistributed in scikit-learn (BSD-3-Clause). It is bundled with this page, so loading it makes no network request.
Limits
- Exchangeability. The guarantee needs calibration rows and new rows to be exchangeable, for example independent draws from the same population. It does not hold for time series with trend or drift, for data sorted in a meaningful order before a split, or when new rows come from a different population.
- Marginal, not conditional. 90% is an average over rows. A subgroup can be covered 70% of the time while another is covered 99%. Nothing here guarantees coverage for one particular person or row.
- Linear base models. The point predictor, the error-spread model and the quantile models are all linear in the features you give. Validity does not depend on that, but width does: if the error size depends on |x| or x², add that column yourself, or the adaptive methods cannot follow it.
- Split conformal spends data. Rows used for calibration and testing are not used for fitting. Full conformal and cross-conformal methods, which reuse data, are not implemented.
- Random split. Results depend on the seed. Change it, or read the average over repeated splits, before drawing conclusions from small data.
- Not modelled: classification, categorical features (recode them as numbers yourself), weighted or covariate-shift conformal, and grouped or clustered rows.
Questions
What is a conformal prediction interval?
It is a range around a model's prediction that is built so that, if your rows are exchangeable (for example a random sample), a new row's true value falls inside it at least as often as the coverage level you ask for, such as 90%. The guarantee holds for any data distribution and any underlying model, and in finite samples. The width is set by how large the model's errors were on calibration rows it did not train on.
How is this different from a regression prediction interval?
A textbook regression prediction interval assumes normal errors with constant variance and a correct linear model. If those fail, its coverage can be well below what it claims. A conformal interval assumes only exchangeable rows; a poor model gives wider intervals, not wrong coverage.
Which method should I use: split conformal, locally weighted or CQR?
All three have the same guarantee, so pick the narrowest on your held-out rows. Split conformal gives the same width everywhere. Locally weighted conformal and CQR widen the interval where errors are larger, which helps when noise grows with a feature. With constant noise, plain split conformal is often as narrow and simpler.
Why is the held-out coverage not exactly 90%?
The guarantee is on average over random splits and new rows. A single test set of 100 rows varies by several points by chance, which is why the tool also averages over 20 random splits. With continuous scores the long-run average lies between the target and the target plus 1/(n+1), where n is the number of calibration rows.
Why did I get an infinite interval?
When the calibration set is so small that k = ceil((n+1)(1-alpha)) exceeds n, no finite width can be guaranteed. At 90% coverage you need at least 9 calibration rows; at 95%, 19; at 99%, 99. Add rows or raise the calibration share.
Does the 90% hold for every subgroup or every row?
No. It is marginal: averaged over all rows. Locally weighted conformal and CQR usually make coverage more even, but neither guarantees conditional coverage, and nothing holds if new data drift away from the calibration data.
Is my data uploaded anywhere?
No. The CSV is read and analysed by JavaScript in this browser tab. There are no accounts, cookies or analytics vendors.