Your tests
Rows with a missing value are left out and counted.
Result
| Method | Controls | Valid under | Kept |
|---|
Tests, smallest p first
Method and citation
Sort the m p-values so that p(1) ≤ … ≤ p(m). Each method's adjusted p-value is the smallest error rate at which it would keep that test, so a test is kept when its adjusted value is at or below your level.
Benjamini-Hochberg (1995): keep the k smallest, where k is the largest i with p_(i) <= i q / m; adjusted value min over j >= i of m p_(j) / j. It controls the false discovery rate at q for independent tests and, as Benjamini & Yekutieli (2001) showed, for positively dependent ones (PRDS). Benjamini-Yekutieli divides q by 1 + 1/2 + ... + 1/m and holds under any dependence. The two-stage BKY procedure (Benjamini, Krieger & Yekutieli 2006) runs BH at q / (1 + q), estimates the number of true nulls as m minus its rejections, and runs BH again at the correspondingly higher level; it is shown as rejections only.
Storey's q-value (Storey 2002; Storey & Tibshirani 2003) is the BH adjustment multiplied by pi0 = #{p > lambda} / (m (1 - lambda)), the estimated share of true nulls, capped at 1. Below the histogram, the dashed line is that estimate: if the right-hand bars sit near it, the nulls look uniform as they should.
Bonferroni multiplies by m; Šidák (1967) uses 1 - (1 - p)^m; Holm (1979) steps down, max over j <= i of (m - j + 1) p_(j), and like Bonferroni holds under any dependence; Hochberg (1988) steps up with the same factors and needs independent or positively dependent tests. All four control the family-wise error rate.
e-BH (Wang & Ramdas 2022): with e-values sorted largest first, keep the k largest where k is the largest i with e_[i] >= m / (alpha i). An e-value (Vovk & Wang 2021) is a non-negative statistic with expectation at most 1 under the null.
Online FDR: tests that arrive over time
Batch methods need the whole list before deciding anything. Online rules decide each test when it arrives, with a level alphat that depends only on earlier tests, so no decision is ever revised. Each spends from a budget ("alpha-wealth") along a sequence gamma that sums to at most 1, and earns budget back with each discovery.
LOND (Javanmard & Montanari 2018): alpha_t = alpha gamma_t (D + 1), with D the discoveries so far. LORD++ (Ramdas, Yang, Wainwright & Jordan 2017): alpha_t = W0 gamma_t + (alpha - W0) gamma_(t - tau_1) + alpha sum_(j>=2) gamma_(t - tau_j), with tauj the time of the j-th discovery and initial wealth W0. SAFFRON (Ramdas, Zrnic, Wainwright & Jordan 2018) is the online analogue of Storey's adaptive procedure: p-values at or below lambda are "candidates", and the clock in each gamma only advances on non-candidates, so a stream rich in signal keeps its levels high; its levels are capped at lambda. Alpha-spending (alpha_t = alpha gamma_t) is the online Bonferroni and bounds the family-wise error under any dependence. LOND, LORD++ and SAFFRON use gamma_j = c log(max(j,2)) / (j e^sqrt(log j)) (LOND, LORD++, spending, with c = 0.07720838) and gamma_j = j^-1.6 / zeta(1.6) (SAFFRON). That c, the usual default, makes the sequence sum to 0.976, not 1, so the rules spend about 97.6% of the budget: valid, slightly conservative.
The online rules are checked to 1e-12 against a second implementation written separately in Python (eval/online_reference.py: every level and decision, 4 rules, 6 streams, 2 settings), and in a seeded simulation (eval/simulate_online.mjs, 1,000 independent one-sided z-tests, 400 runs): every rule kept its false discovery rate under 5%. With 10% true effects in random order, power was LORD++ 0.32, SAFFRON 0.45, LOND 0.18 and alpha-spending 0.09, against 0.60 for batch BH with the whole list in hand; with 50% true effects SAFFRON (0.88) beat batch BH (0.83), because BH's false discovery rate sits at pi0 × alpha (0.025) while SAFFRON's adapts up to 0.047.
Verified by tests in the source against statsmodels 0.15.0: every adjusted p-value for Bonferroni, Šidák, Holm, Hochberg, BH and BY to 1e-12 on six lists (the example, all-null and all-significant edge cases, 200 tests with ties, 1,000 tests, and 120 with p-values down to 1e-14 and a block of exact 1s), the two-stage BKY rejections at four levels, Storey's q-values at three lambdas, and e-BH against a direct implementation of the paper's definition. On the example the tool gives the counts that statsmodels' own tests pin to the Benjamini, Krieger & Yekutieli (2006) paper: one-stage BH keeps 4 and the two-stage procedure 8, with second-stage level 0.0649 (Bonferroni keeps 3).
BibTeX
@article{benjamini1995controlling,
author = {Benjamini, Yoav and Hochberg, Yosef},
title = {Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing},
journal = {Journal of the Royal Statistical Society, Series B},
volume = {57}, number = {1}, pages = {289--300}, year = {1995},
doi = {10.1111/j.2517-6161.1995.tb02031.x}
}
@article{benjamini2001control,
author = {Benjamini, Yoav and Yekutieli, Daniel},
title = {The Control of the False Discovery Rate in Multiple Testing under Dependency},
journal = {Annals of Statistics},
volume = {29}, number = {4}, pages = {1165--1188}, year = {2001},
doi = {10.1214/aos/1013699998}
}
@article{benjamini2006adaptive,
author = {Benjamini, Yoav and Krieger, Abba M. and Yekutieli, Daniel},
title = {Adaptive Linear Step-up Procedures That Control the False Discovery Rate},
journal = {Biometrika},
volume = {93}, number = {3}, pages = {491--507}, year = {2006},
doi = {10.1093/biomet/93.3.491}
}
@article{storey2002direct,
author = {Storey, John D.},
title = {A Direct Approach to False Discovery Rates},
journal = {Journal of the Royal Statistical Society, Series B},
volume = {64}, number = {3}, pages = {479--498}, year = {2002},
doi = {10.1111/1467-9868.00346}
}
@article{storey2003statistical,
author = {Storey, John D. and Tibshirani, Robert},
title = {Statistical Significance for Genomewide Studies},
journal = {Proceedings of the National Academy of Sciences},
volume = {100}, number = {16}, pages = {9440--9445}, year = {2003},
doi = {10.1073/pnas.1530509100}
}
@article{holm1979simple,
author = {Holm, Sture},
title = {A Simple Sequentially Rejective Multiple Test Procedure},
journal = {Scandinavian Journal of Statistics},
volume = {6}, number = {2}, pages = {65--70}, year = {1979}
}
@article{hochberg1988sharper,
author = {Hochberg, Yosef},
title = {A Sharper Bonferroni Procedure for Multiple Tests of Significance},
journal = {Biometrika},
volume = {75}, number = {4}, pages = {800--802}, year = {1988},
doi = {10.1093/biomet/75.4.800}
}
@article{sidak1967rectangular,
author = {{\v{S}}id{\'a}k, Zbyn{\v{e}}k},
title = {Rectangular Confidence Regions for the Means of Multivariate Normal Distributions},
journal = {Journal of the American Statistical Association},
volume = {62}, number = {318}, pages = {626--633}, year = {1967},
doi = {10.1080/01621459.1967.10482935}
}
@article{wang2022false,
author = {Wang, Ruodu and Ramdas, Aaditya},
title = {False Discovery Rate Control with E-values},
journal = {Journal of the Royal Statistical Society, Series B},
volume = {84}, number = {3}, pages = {822--852}, year = {2022},
doi = {10.1111/rssb.12489}
}
@article{vovk2021evalues,
author = {Vovk, Vladimir and Wang, Ruodu},
title = {E-values: Calibration, Combination and Applications},
journal = {Annals of Statistics},
volume = {49}, number = {3}, pages = {1736--1754}, year = {2021},
doi = {10.1214/20-AOS2020}
}
@article{javanmard2018online,
author = {Javanmard, Adel and Montanari, Andrea},
title = {Online Rules for Control of False Discovery Rate and False Discovery Exceedance},
journal = {Annals of Statistics},
volume = {46}, number = {2}, pages = {526--554}, year = {2018},
doi = {10.1214/17-AOS1559}
}
@inproceedings{ramdas2017online,
author = {Ramdas, Aaditya and Yang, Fanny and Wainwright, Martin J. and Jordan, Michael I.},
title = {Online control of the false discovery rate with decaying memory},
booktitle = {Advances in Neural Information Processing Systems 30},
pages = {5650--5659}, year = {2017}
}
@inproceedings{ramdas2018saffron,
author = {Ramdas, Aaditya and Zrnic, Tijana and Wainwright, Martin J. and Jordan, Michael I.},
title = {{SAFFRON}: an adaptive algorithm for online control of the false discovery rate},
booktitle = {Proceedings of the 35th International Conference on Machine Learning},
series = {PMLR}, volume = {80}, pages = {4286--4294}, year = {2018}
}
This is an independent implementation from the papers' definitions; no code from the authors or from other libraries was copied. statsmodels was used only to produce the reference numbers the tests check against.
"Load online example" is simulated, not real data: 400 one-sided z-tests in arrival order, each a true effect (mean shift 3) with probability 0.15 (eval/make_online_example.mjs, seeded). Its truth column exists only because it is simulated.
Example data
"Load example" uses the 15 p-values of the worked example in Benjamini & Hochberg (1995), which Benjamini, Krieger & Yekutieli (2006) reuse for their two-stage procedure. They are labelled H1 to H15 in sorted order; the original hypotheses are not named here. The list is bundled with this page, so loading it makes no network request.
Limits
- The p-values must be valid. Every correction assumes each p-value is uniform (or larger) when its null is true. p-hacked, data-dredged or already adjusted p-values give corrections that mean nothing.
- Count every test you ran. m is the number of rows used. Leaving out tests you looked at, or pre-filtering to the promising ones, makes every method too lenient.
- Dependence. Benjamini-Hochberg, Hochberg, Šidák and the two-stage procedure need independent or positively dependent tests. Bonferroni, Holm, Benjamini-Yekutieli and e-BH hold under any dependence.
- Storey's pi0 needs many tests. With fewer than about 100 it is noisy, and q-values can be optimistic. It is also biased upward when null p-values pile up near 1 (discrete or conservative tests), and it breaks under strong correlation: in this tool's seeded simulation (1,000 one-sided z-tests, 100 real effects, every pair of statistics correlated 0.5, 400 runs) q-values at 0.05 had an actual false discovery rate of 0.109, while Benjamini-Hochberg's was 0.029 (
eval/simulate.mjs). - Online: the order is the design. Rows are read top to bottom as the order the tests arrived. Sorting the list, or picking the order after seeing results, voids the online guarantee. LOND, LORD++ and SAFFRON are proved for independent p-values; alpha-spending holds under any dependence.
- FDR is an average. A 5% false discovery rate bounds the expected share of false results over repeated studies, not the share in your list.
- Not modelled: local false discovery rates, weighted or grouped (hierarchical) procedures, online FDR under dependence, ADDIS for conservative nulls (Tian & Ramdas 2019), decaying-memory and false discovery exceedance rules, resampling methods such as Westfall-Young, Hommel's procedure, closed testing and gatekeeping for clinical endpoints, and boosted e-BH.
Questions
Why correct p-values at all?
Each test at the 5% level has a 5% chance of a false positive when nothing is going on. Run 20 such tests and you expect one false positive; run 20,000 and you expect a thousand. A correction decides which results still stand once you account for how many you ran.
What is the difference between FWER and FDR?
Family-wise error rate (FWER) methods such as Bonferroni and Holm bound the chance of even one false positive among everything you call significant. False discovery rate (FDR) methods such as Benjamini-Hochberg bound the expected share of false positives in that list. FDR keeps more results and suits screening; FWER suits a few confirmatory claims where any error is costly.
What is an adjusted p-value?
The smallest error rate at which that test would be kept by the method. Compare it with your chosen level: a Benjamini-Hochberg adjusted p-value of 0.03 means the test is kept at any false discovery rate of 3% or more.
What is a q-value, and how is it different from a BH-adjusted p-value?
Storey's q-value is the Benjamini-Hochberg adjusted p-value multiplied by pi0, an estimate of the share of tests where nothing is going on. When many tests are real effects pi0 is below 1 and q-values keep more results. The estimate needs many tests, ideally hundreds; with few, use Benjamini-Hochberg.
Which method should I use?
Decide before you look. For a screen (genes, features, many A/B metrics) use Benjamini-Hochberg, or Benjamini-Yekutieli if the tests could be dependent in ways other than positively. For a handful of primary claims use Holm, which is never worse than Bonferroni and valid under any dependence.
What are e-values and e-BH?
An e-value is a non-negative test statistic whose average is at most 1 when the null hypothesis is true, such as a likelihood ratio or the final value of a betting martingale in a sequential test. e-BH (Wang and Ramdas 2022) controls the false discovery rate for e-values under any dependence between tests.
How do I control the false discovery rate when tests arrive one at a time?
Use an online rule such as LORD++, SAFFRON or LOND. Each test is given a level when it arrives, based only on the tests before it, and kept or dropped on the spot; a discovery earns back budget for later tests. Batch Benjamini-Hochberg cannot be used this way because adding a test can change earlier decisions. Choose the order before seeing results: online rules are only valid in the order the tests really arrived.
Is my data uploaded anywhere?
No. The list is read and corrected by JavaScript in this browser tab. There are no accounts, cookies or analytics vendors.