A flat coverage floor — --cov-fail-under=80
in pytest-cov, or an equivalent minimum in any coverage tool — is the
default method, because most CI setups already have one: fail the build if
the overall percentage is below the line. It catches a project that never
had good coverage to begin with. It cannot catch a project walking
down from good coverage, one merge at a time, as long as every
merge stays above the floor — and it cannot catch one file's tests
getting gutted inside a single PR if another file's coverage happens to
improve enough to keep the total above the line.
| File | Baseline | This PR | Verdict |
|---|---|---|---|
auth.py | 20/20 lines, 100% | 4/20 lines, 20% | dropped — 80.00 points |
utils.py | 72/80 lines, 90% | 77/80 lines, 96.25% | improved — -6.25 points |
| Overall | 92/100, 92% | 81/100, 81% | both above an 80% floor |
Against a flat --cov-fail-under=80 floor, both reports pass:
92% clears 80, and so does 81%. Nothing about the floor fails, and nothing
about it can even see that auth.py lost 16 of its 20 covered
lines — the check only ever looks at the one summed-up number.
Pasted into this tool, the same two reports resolve to a real finding:
line coverage dropped 11.00 points (92.00% → 81.00%), and
auth.py is named as the file that got worse, sorted to the
top of the report. utils.py's improvement is exactly what
kept the overall number above the floor at all — without it, this
PR would have failed on the floor alone and never needed a per-file
breakdown to catch.
| Question | Where to answer it |
|---|---|
| My CI's coverage floor passed — did this PR still make coverage worse? | Coverage Drop Check — a floor only checks the absolute number; this checks the direction and names which file moved. |
| Which specific file's tests got weaker in this PR? | Coverage Drop Check — the per-file table sorts dropped files first; a flat floor has no per-file breakdown at all. |
| This project has never had good test coverage and I want a baseline minimum enforced. | A flat floor — that is exactly the job it is built for, and this tool needs a baseline report to compare against, not just an absolute minimum. |
| Coverage dropped a little on this PR but stayed comfortably above our floor — is that worth blocking on? | Neither tool decides that for you. Coverage Drop Check reports the exact points-and-file delta so a reviewer can judge it; a floor cannot even show the number moved. |
Neither method is wrong on its own: a floor is the right guard against a codebase that never had coverage discipline, and this tool needs a real baseline report to compare against, which a brand-new project does not have yet. The gap is the case a floor is structurally unable to close — a number that stays technically fine while the file it actually matters for gets worse underneath it, masked by whatever else moved in the same report. That is exactly the gap this tool exists to close.
A floor like --cov-fail-under=80 asks one question: is the overall percentage still above 80? It cannot see direction or which file moved, only where the number sits right now. A PR can walk coverage down from 92% to 81%, one point at a time across many merges, and never once fail the floor — and within a single PR, one file's tests can be gutted while another file's coverage happens to improve enough that the overall percentage still clears 80, so the floor reports a clean pass while a real regression ships inside it.
A baseline report shows 92% overall (auth.py fully covered at 100%, utils.py at 90%). After a PR, the report shows 81% overall — auth.py dropped to 20% covered (16 of its 20 lines lost their tests), while utils.py improved slightly to 96.25%, which is what keeps the overall number above an 80% floor at all. A flat --cov-fail-under=80 check passes on both reports: 92 and 81 are each above 80. Pasted into this tool, the same two reports report a real finding: line coverage dropped 11.00 points (92.00% → 81.00%), and auth.py is named as the one file that got worse — the fact the floor has no way to express.
Noticing that one file's coverage fell is obvious once you already know to compare that specific file's line rate across the two reports, file by file. What a flat floor gets wrong is that it collapses every file into one number before you ever see it, and a codebase with dozens or hundreds of files makes re-deriving the per-file delta from two raw XML reports exactly the kind of bookkeeping nobody does by hand on every PR. This tool does that comparison mechanically and exhaustively, so a drop hiding behind an improvement elsewhere in the same report does not depend on someone happening to open the one file that got worse.
No. A floor still catches the case this tool cannot: a project that never had good coverage in the first place, or a brand-new file added with no tests and no baseline to compare against. The two catch different failures — a floor bounds the absolute number, this tool bounds the direction and names the file, and neither replaces reading the diff for why a file's tests changed at all.
More: why coverage drops on a PR that "didn't touch tests" · Coverage Drop Check vs. Codecov, for an ongoing connected dashboard rather than a one-off local delta.