Changepoint detection on seven series nobody tuned on

Seven labelled series held out from tuning show which choices of the free changepoint detector hold up: agreed changes do, BOCPD does not.

The changepoint detector was tuned on the 31 series that the Turing Change Point Dataset ships in its repository, each marked by five human annotators. TCPD has seven more single-column series that it builds by downloading from web.archive.org. We fetched public copies of the built files and kept them only because each one's md5 matches TCPD's own checksum list, then scored the frozen settings on them. Nothing was changed after looking.

settingF1, 7 held outF1, 31 tuned on
changes every one of six settings agrees on0.7110.702
Bayesian online detection (BOCPD)0.6220.706
single level model0.2960.477
report no change (published)0.6390.663
binary segmentation (published)0.7270.714

The agreed changes held up, with the best segmentation cover of any method on these seven. BOCPD, which looked the most stable in sample, fell below reporting nothing: on 991 weekly measles counts it reports 27 changes where the annotators mark about one. The over-cut warning added earlier the same day held up too: it fired on six of the seven, and following it beat the plain model on all six.

So the page now leads with an "Agreed changes" line: the changes all six settings find, followed by these held-out numbers, so a reader knows how far to trust it. Seven series is a small sample, and none of these results is a significant win over reporting no change. The scripts, outputs and tests are MIT licensed in the repository (eval/tcpd-heldout.mjs).

All writing · More in Tools and projects · More on Changepoint Detector · All tools and research