Eyeballing two console logs is the default method, because
it needs no tool: open the first failed run, open its rerun, and compare
by eye. It works when a suite is small and exactly one test failed. It
stops working the moment a suite reruns only its failures —
pytest --lf, jest --onlyFailures, and most CI
"retry failed tests" features all do this by design — because the
second log then contains a different, smaller set of tests than the
first, and reading "the rerun looked mostly green" is not the same claim
as "every test that failed the first time passed the second time."
| Run 1 (full suite, 4 tests) | Run 2 (--lf rerun, 2 tests) | Verdict |
|---|---|---|
test_login — passed | not rerun | stable |
test_checkout — failed (cart total mismatch) | test_checkout — passed | flaky — flipped |
test_export — failed (export timeout) | test_export — failed (same timeout) | broken — failed both times |
test_search — passed | not rerun | stable |
Eyeballing this pair of logs, both failures look the same shape: "failed
in run 1, and run 2 is a rerun of just the failures." The natural read is
"the rerun is mostly green, this was probably all flakiness" — which
is true of test_checkout and false of test_export,
which failed with the identical TimeoutError both times it
ran. Pasted into this tool, the two reports are read test-by-test across
every run each test appeared in: test_checkout classifies as
flaky (1 pass, 1 fail), test_export
classifies as broken (2 fails, 0 passes) — a
real, consistent failure, not noise. Merging on "the rerun looked fine"
ships whatever test_export exists to catch.
| Question | Where to answer it |
|---|---|
| I have a failed run and its rerun open in two tabs right now — which failures are flaky and which are real? | Flaky Test Finder — paste both reports, get a verdict per test in seconds, no guessing which one caught your eye. |
| The suite only reran its failures, so the second log is smaller — does that mean the missing tests broke? | Neither, by hand or here: a test absent from a rerun because it was not retried is not evidence of anything. This tool already treats that as the retry it is, not a missing result. |
| I want to know if this is the exact same test that has been flaky for weeks, across many past builds. | Neither this tool nor eyeballing two logs — that needs history across many runs over time, which two reports pasted once cannot provide. A connected dashboard tool is built for that job. |
| I have three or more reruns and want to be sure before I mark a test as quarantined. | Flaky Test Finder — it accepts any number of reports at once and classifies from all of them together, which gets more reliable with more runs; eyeballing gets slower and more error-prone with each additional log. |
Neither method is wrong about small cases: one test, two logs, obviously different failure messages — a person can read that correctly every time. The gap opens exactly where CI already makes eyeballing hardest: a suite with dozens or hundreds of tests, a rerun that only covers the failures, and a reader under time pressure to just get the check green. That is precisely when "mostly green now" and "every specific test that failed actually flipped" stop being the same sentence, and it is the gap this tool exists to close.
It conflates two different facts that look identical in a console: a test that failed once and passed on rerun (flaky), and a test that failed the same way in both runs (broken, not flaky). Once a suite reruns only its failed tests -- pytest --lf, jest --onlyFailures, and most CI 'retry failed tests' features all do this -- the second log legitimately contains only a fraction of the first, and a person skimming it tends to read 'mostly green now' as 'must have just been flaky' without checking whether every specific test that failed actually flipped, or whether one of them failed again.
A 4-test run fails test_checkout and test_export; test_login and test_search pass. The team reruns only the two failures with --lf. In the rerun, test_checkout passes and test_export fails again with the same TimeoutError. Eyeballing two logs, both failures look the same: 'failed in run 1'. Pasted into this tool, the two reports classify differently: test_checkout is flaky (1 pass, 1 fail across 2 runs), test_export is broken (2 fails, 0 passes across 2 runs) -- a real, consistent failure. Merging past it because 'the rerun looked mostly fine' ships the regression test_export exists to catch.
The individual fact -- did this one test flip -- is obvious once you already know to check it test-by-test across every run it appeared in, including the runs where it did not appear. What eyeballing gets wrong is scale and attention, not logic: a report with 40, 200, or 2,000 tests makes 'read every test's status across every run and note which ones appeared in which' the kind of bookkeeping a person skips by pattern-matching on the overall shape of the log instead ('mostly green' or 'still red'). This tool does the same bookkeeping mechanically and exhaustively, so the answer does not depend on which test happened to catch the reader's eye.
No. It replaces the specific step of deciding, test by test, whether a failure is flaky or a real regression. You still need the JUnit-XML reports in hand -- most CI systems already produce or can be configured to produce them -- and you still decide what to do about a broken test once it is correctly identified as broken. This tool only removes the guessing from that one classification step.
More: why tests go flaky in CI and what to do about each one · Flaky Test Finder vs. BuildPulse, for ongoing team-wide tracking rather than a one-off verdict.