Written by an AI agent working in this repository.
The change-point detector can watch a growing metric and send an alert when its level moves. Yesterday's test ran that watch on the 47 real series of the Numenta Anomaly Benchmark (NAB): server metrics, ad-exchange clicks, taxi counts, traffic sensors and Twitter volumes, averaged to hourly points. It was not flattering. Only 13.7% of the default setting's alerts fell inside a labelled anomaly window, against 11.8% for an alert at a random point. A drift model ("Wandering level") did better: 20.5%, at half the alerts.
That test had an obvious gap. Most of these series rise and fall every day, and the watch was run without removing that cycle. Maybe it was alerting on the evening dip.
The test. The rules were written down and committed before any scored run. Removing the cycle (period 24) would count as helping the default if it reached at most one window fewer, sent no more alerts, and raised the share of alerts at a window by 2 points. It would count as beating drift if it matched drift's share and reach.
What happened. Both rules failed.
| setting | alerts | at a window | windows reached (of 61) |
|---|---|---|---|
| default, no period | 797 | 13.6% | 50 |
| default, period 24 | 833 | 14.9% | 52 |
| drift | 404 | 20.5% | 48 |
Removing the cycle reached two more windows. On the NYC taxi series it caught all five holiday dips, against four. But it sent slightly more alerts, and its alerts were barely more often at an anomaly than before. The cycle was not the problem. On these metrics the level watch marks real level moves that NAB does not label as anomalies. NAB labels spikes and failures, not every level move.
The wrong guesses. I predicted the season would cut alerts by about a third and lift the share to 16-18%. It did neither. The run also turned up a bug in the test harness first: one script's worker answered another script's jobs, and the first run crashed before printing a score. If it had run to the end, the period settings could have been scored without their period. Each script's worker now answers only its own jobs, and a test holds it there.
What changes. Nothing in the tool's defaults or advice, as the rules said in advance. The trade-off stands: drift's alerts are the ones that point at anomalies, and the level settings are the ones that reach more of them. The next test needs a third labelled set, not another look at NAB. The full numbers are in the README, "Round 11".