← Changepoint Detector · FREE · MIT · RUNS IN YOUR OWN ACCOUNT
Get alerted when a metric really shifts, not every time it wobbles
Daily sign-ups, orders, a sensor reading, a conversion rate. You do not want a ping for every odd day. You want to hear once, within a few days, when the level has moved for good. This watch refits your series after each new row with Bayesian online changepoint detection (Adams and MacKay, 2007; PELT, Killick et al. 2012, as an option) and posts to Slack, Discord, Teams or Mattermost once per change, after five points of evidence.
Set it up in five minutesOr watch it in your browser, no setup
What a message looks like
orders changed from 2026-05-01: level up from 49.9 to 54.9 (4.29 noise sd), Bayesian probability 99%; 7 points since.
From the tool's own test: 120 quiet days at about 50, then a step to 55. The run on day 124 says "watching: a possible change from 2026-05-01"; the run on day 127 posts the message above; later runs stay silent about it.
Who it is for
Founders and growth teams
Did weekly active users or trial starts step down after the pricing change, or is this week just slow?
Operations and SRE
A lasting rise in tickets, refunds or p95 latency, as opposed to one bad hour your paging already covers.
Analysts with a shared sheet
Point it at a published Google Sheet's CSV link and it checks the sheet every morning.
Labs and field sensors
A drift or regime change in a daily reading, dated, with a method you can cite.
How well it works, measured
We replayed series one point at a time, the way a daily job sees them, and counted what a person would have been sent.
| Stream | Default | Previous default | PELT level |
|---|---|---|---|
| False alerts per 1,000 checks, independent noise, no change | 0.5 | 0 | 2.5 |
| False alerts per 1,000 checks, sticky noise (AR(1) 0.5) | 0.75 | 6.25 | 49.8 |
| False alerts per 1,000 checks, very sticky noise (AR(1) 0.8) | 1.5 | 32.3 | - |
| Steps of 2 noise sd reported (independent noise) | 10 of 10 | 7 of 10 | 10 of 10 |
| Steps of 2 noise sd reported (sticky noise, AR(1) 0.5) | 6 of 10 | 8 of 10 | 9 of 10 |
| Steps of 1 noise sd reported | 2 of 10 | 0 of 10 | 9 of 10 |
| Real series: alerts per 1,000 checks | 18 | 30 | 98 |
| Real series: alerts at a change an annotator marked | 41% | 33% | 21% |
| Real series: changes most annotators marked, reached | 33 of 52 | 38 of 52 | 52 of 52 |
| Median points from the change to the message | 5 | 5 | 5 |
So the default is the quiet one. It allows for sticky noise, where today's value leans on yesterday's, which most business metrics have. That cut false alerts on very sticky noise from 32 per 1,000 checks to 1.5. The price is real: on the labelled real series it reached 33 of the 52 changes most annotators marked, against 38 for the plain Bayesian pass. It was 29 before round 4 found why: on smooth or wandering series (a price, a population) the estimated stickiness sat at its 0.95 limit, where any real move reads as persistence, so it is now capped at 0.8 (--rho-cap; 0.95 restores the old behaviour, which is quieter on a level that wanders like a random walk). If missing a change costs you more than a false alarm, run with --detector bocpd (the previous default) or --detector pelt (all 52 marked changes reached, about six times the alerts). With --model poisson (counts) or meanvar the default is the plain Bayesian pass, since the sticky-noise term is for levels only.
The rule for picking the defaults was written down before the first run, and the labelled real series were scored once afterwards without changing anything. Script and full output: eval/watch-replay.mjs, eval/watch-replay.out.txt (round 1), eval/watch-replay-round3.out.txt (round 3, which changed the default), eval/watch-rho.mjs and eval/watch-replay-round4.out.txt (round 4, the rho cap; the only round where the real series counted in the rule, because the fault only shows there).
Two guards against flicker are on by default. A change found later but dated back inside the history the first check called quiet is held, not sent. And a new change is sent only when the next check still finds it, so a detection that appears for one check and is gone the next never reaches you; the price is one check of delay (a day, on a daily watch). On the labelled real series the second guard took 2 of 105 alerts off, and 2 of 61 on series held out from every choice, without losing a marked change (--persist 1 sends at once). It is a small gain, and the README says how small.
No setup: watch it in your browser
If you look at the metric yourself every few days, you do not need a scheduled job. On the changepoint detector page, paste your rows (or a published Google Sheet's CSV link) under Keep watching a series and press Check for new changes. It runs the same watch with the same defaults; your browser remembers which changes it has already shown you, so each one is reported once. Nothing is uploaded, and nothing messages you: you have to come back and press the button. For a message in chat without opening a page, set up the daily run below.
Set it up in your own GitHub account
- Make the folder (no clone of our repository needed; Node 18+ and curl):
curl -fsSL https://raw.githubusercontent.com/Duncan-Haywood/oss-monorepo/main/ventures/changepoint-detector/deploy/make-watch-repo.sh | sh -s -- metric-watch
- Put your series in
metric-watch/data/metric.csv(a date column and a value column, oldest first), or plan to use a CSV link instead. - Push the folder to a new GitHub repository. In its Settings > Secrets and variables > Actions, add the variable
METRIC_CSV(data/metric.csvor the link) and, for chat messages, the secretWATCH_WEBHOOK_URL. With no webhook, a new change fails the run and GitHub emails you. - Run the
watch-metricworkflow once by hand. The first run records the changes already in your history as the baseline and does not alert on them, nor later on a change dated back inside that history; after that it runs daily.
Or anywhere with Node: node watch.mjs data.csv --state watch-state.json, from cron, a CI job or a notebook. Options for counts (--model poisson), volatility (--model meanvar), a weekly cycle (--period 7) and growth (--log) are in node watch.mjs --help.
Watch your Stripe payments, revenue and card declines
If you sell through Stripe, you do not need a CSV at all. Add a Stripe restricted key with read access to Charges (Dashboard > Developers > API keys > Create restricted key) as the secret STRIPE_API_KEY in the repository from the steps above, and leave METRIC_CSV unset. Each run fetches your complete days into data/stripe.csv: succeeded payments, their gross revenue, and failed payments, with zeros for quiet days and today left out so a half day never reads as a fall. It then watches all three, so you hear when sales, revenue or declines move to a new level and stay there: a broken checkout, a card-network problem, a price change that worked or did not.
Your key stays in your own GitHub account and calls Stripe directly; nothing passes through us. Keep the repository private: the state file records your revenue around each change. Set STRIPE_OPTIONS to --utc-offset -5 (or your own) so days start at your midnight, --currency usd to choose the currency summed into revenue, or --customers to add new customers per day. Locally: STRIPE_API_KEY=rk_live_... node stripe-metrics.mjs --out data/stripe.csv.
How well, measured on synthetic shops (40 payments a day, about $30 each; no real Stripe account was used): when declines jumped from 2% to 25% of attempts, all 20 shops heard it 5 days later; from 2% to 10%, 16 of 20 did, 4 to 7 days later. On 20 shops with no change, watched daily for a month, no alert was sent on any of the three columns. Real shops have a weekly cycle and promotions these synthetic ones lack, so replay your own history first (--suggest) before trusting the quiet (README, "Stripe as a source").
Several metrics from one sheet
Most teams track more than one number: orders, revenue, sign-ups, tickets. If they sit as columns of one CSV, one repository watches them all. Add --columns all to WATCH_OPTIONS, or name them, each with its own model if it needs one: --columns orders:poisson,revenue,signups:poisson. Each column keeps its own record of what was already reported, and a day's new changes arrive as one message that names the columns that moved.
Watching five numbers instead of one means five chances a day for noise to look like a change. We tested whether to split the alert budget across the columns, and the test said no: on five columns of pure noise the default settings sent 1.3 alerts per 100 checks of the whole watch without any split, and on five real road-casualty columns the split sent more alerts, not fewer (README, "Several metrics in one watch"). So --detector auto --alerts-per 1/month picks each column's setting on its own history, as for a single metric.
Each column you add is one more chance for noise, and the default budget is per column, not for the whole watch. On a real normal run of an eight-sensor water-pump testbed (the Skoltech Anomaly Benchmark, 2.6 hours in 5-second steps), the quiet default on all eight columns sent 8.5 alert lines per 100 checks, over four times the budget, and every one of them was false. On the same testbed's 33 fault runs it caught all 33 faults, mostly through the flow-rate column. So replay your history first (--suggest, or the replay on this page), see which columns are loud on an ordinary stretch, and watch the ones that matter (README, "Round 12"). Asking for two columns to agree before an alert was tested on the same testbed and does not ship: sensors that move together, like the two accelerometers or the two temperatures, agree on noise too, so the normal run still sent 2.7 messages per 100 checks, and 9 of the 33 faults that one column alone had seen were missed (README, "Round 13"). A smallest-move floor works better: --min-shift temperature=0.5 drops any move smaller than that in that column. Read off the normal run for every column at once, the floors cut false alert lines on the fault runs' normal stretches from 6.3 to 2.3 per 100 checks, and the alerts left sat at a fault mark 47% of the time against 37%; but they lost 5 of the 33 faults, because the flow-rate column that sees the faults was floored as hard as the rest. So set a floor by hand on the loud columns that do not matter and leave the one that sees the problem alone (README, "Round 14"). A single floor for every column, measured against how much each column wandered in the five minutes before the move, did better on false alerts (1.4 per 100 checks, and 59% of the alerts left sat at a fault mark) but lost 6 of the 33 faults, so it is not offered either (README, "Round 15"). Giving the loudest column its own floor and the rest a shared, lower one caught 30 of the 33 faults but let 2.5 false alert lines per 100 checks through, most of them from a slowly drifting motor temperature whose ordinary day did not look like the fault days (README, "Round 16"). You can set that kind of floor yourself: --min-spread 2.5 drops any move smaller than 2.5 times how much the column wiggled in the 60 points before it, and --min-spread temperature=8 sets it for one column. Re-reading the floor from each column's own last five minutes, alerting only on a move bigger than any it made there, caught 32 of the 33 faults but let 4.7 false alert lines per 100 checks through: the temperature's moves on the fault days were bigger than its recent wander, not drift a rolling window explains (README, "Round 17"). On a fresh test, 28 servers with about 30 metrics each (the Server Machine Dataset, hourly), the quiet default on every metric sent 76 false alert lines per 100 checks on normal days, and the three automatic floors that came closest on the pump all lost most of the labelled incidents: the best kept 106 of the 290 that no floor caught (README, "Round 18"). Finer data did not change that: on three of those servers in 10-minute steps the floors still missed most incidents under an hour, though every setting reported the incidents it did catch about three hours sooner (README, "Round 19"). So this watch is for level changes in the few metrics you pick, not for anomaly detection across everything on a machine.
What you would use otherwise
- Stripe analytics apps (Baremetrics, ChartMogul, ProfitWell and the like). Dashboards of subscription metrics, mostly paid, and they hold your data. This sends one message when a daily number moves to a new level, runs in your account and costs nothing.
- A fixed threshold. Simple, but you have to know the level in advance, and a slow series crosses it on noise.
- Anomaly detection in a monitoring or BI product. Good at single odd points and usually part of a paid plan. A lasting step in level is a different question, and it is the one this answers.
- R's
changepointor Python'srupturesin a scheduled script. More complete methods, but you write the scheduling, the "already told you" memory and the message yourself. This is that glue, done and tested.
Limits
- A series whose level wanders all the time (a bank balance, an exchange rate, a price) will keep alerting: on TCPD's
bankseries the default sent 42 alerts in 465 checks, none at a marked change. Watch a quantity that holds a level between changes, or its daily change instead of its level. - Five points of evidence means a change is reported no sooner than five rows after it. Use
--confirm 3to hear sooner at the cost of more false alarms. - It reports when the data changed, not why. Whether your release, a holiday or an outage caused it needs what you know about that date.
Questions
How is this different from a threshold or anomaly alert?
A threshold or anomaly alert fires on single odd points. This watch asks whether the level of the series has changed for good: it refits the series each day with Bayesian online changepoint detection (Adams and MacKay 2007) and reports a change once five new points sit on its far side, then never again for that change.
Where does it run, and who sees my data?
In a GitHub repository you own, on GitHub Actions, or on any machine with Node 18 or later. Your data and your webhook URL stay in your account; nothing is sent to the people who made the tool.
How many false alarms should I expect?
Measured by replaying series one point at a time: on 4,000 daily checks of independent noise with no change, 2; on sticky (autocorrelated) noise, about 1 per 1,000 checks. On 31 real labelled series from the Turing Change Point Dataset it raised 18 alerts per 1,000 checks, 41% of them at a change a human annotator marked, and reached 33 of the 52 changes most annotators marked; the plain Bayesian pass reached 38 at 30 alerts per 1,000 checks.
What does it cost?
Nothing from us: the code is MIT licensed. A once-a-day run takes well under a minute, inside GitHub's free Actions minutes for most accounts.