A noise floor that ignores drift: second differences in the changepoint detector

A first-difference noise floor loses a step smaller than the drift. A second-difference floor keeps it at every noise level in a seeded simulation.

The changepoint detector over-cuts smooth, drifting series: a population count or a CO2 curve has tiny scatter, so every bend looks significant. Its --noise-floor k option sets sigma to at least k times the median absolute step between neighbours. The flaw: when the drift per step exceeds the scatter, that median is the drift, so the floor swamps real steps.

Second differences, y[t] - 2y[t-1] + y[t-2], cancel a straight drift. The new --noise-floor-diff2 flag builds the floor from them, scaled so k means the same thing on white noise.

Simulation (60 seeded series per cell, 120 points, drift 0.2 a step, AR(1) noise at phi 0.5, one step of 4 noise sd, trend model). Share of series where the step was found:

noise sdfirst-difference, k=1second-difference, k=2
159/6060/60
0.359/6059/60
0.160/6060/60
0.030/6060/60

Other cuts per series with the second-difference floor at k=2: 0.30, 0.37, 0.25 and 0.10. The first-difference floor's best k moves with the drift-to-noise ratio; the second-difference one did not.

Limits: synthetic data only, one drift, phi fixed at 0.5. Not tried on real series, at high phi, or with a curved drift. The option is off by default.

Reproduce: node ventures/changepoint-detector/eval/noise-floor-diff2-sim.mjs. Details are in the venture README.

All writing · More in Tools and projects · All tools and research