KALMAN FILTER AND SMOOTHER · LOCAL LEVEL, TREND AND SEASONAL MODELS · MAXIMUM LIKELIHOOD · MISSING VALUES · OUTLIERS AND LEVEL SHIFTS · RUNS IN YOUR BROWSER

What is the real level under this noisy series, and did it shift?

Paste a series: monthly sales, weekly sign-ups, a sensor reading, a river's yearly flow, a lab value over visits. A Kalman smoother on a structural time-series model (Durbin & Koopman 2012; Harvey 1989) splits every value into an underlying level that drifts and noise around it, with a band saying how sure it is at each point. It decides from the data how much to smooth, fills missing values, removes a seasonal pattern if there is one, forecasts ahead, and tells apart a single outlier from a real level shift.

Nothing you enter leaves this browser tab. Statistical output from data you supply, not advice. Not legal, tax, or financial advice.

Your series

Blank, NA or “.” cells are missing values; keep them as rows so the periods stay equally spaced.


Method and citation

The series yt is modelled as a level plus noise, yt = μt + γt + εt, where the level is a random walk μt+1 = μt + βt + ηt, the slope β is absent, fixed (a drift) or itself a random walk (the local linear trend), and γ is an optional dummy seasonal whose s values sum to about zero over a year (Harvey 1989). Every disturbance is normal with its own variance. Written in state-space form, the Kalman filter gives each period's one-step prediction and its error variance; the smoother runs backwards to give the level at each period from all the data, with its variance (Durbin & Koopman 2012, chapter 4).

The starting level, slope and seasonal values are unknown, so they get the exact diffuse initialisation (Koopman 1997; Durbin & Koopman 2012, sections 5.2 and 5.3): the state variance is split into an infinite and a finite part, and the first observations are spent pinning the start down instead of guessing a huge prior variance. The variances are chosen by maximum likelihood on the diffuse loglikelihood with the irregular variance concentrated out, by Nelder-Mead from a grid of starts. With the trend on automatic, the three trend forms are compared by AIC with the diffuse elements counted as parameters (Durbin & Koopman 2012, section 7.4). Missing values skip the filter's update step. Outliers and level shifts are read from the auxiliary residuals, the smoothed disturbances divided by their standard errors (Harvey & Koopman 1992; de Jong & Penzer 1998). The one-step errors are checked for autocorrelation (Ljung-Box), normality (Bowman-Shenton) and a change in noise level (the H test).

Verified by tests in the source against statsmodels 0.15.0's UnobservedComponents with exact diffuse initialisation, at equal variances, on five cases: the Nile series as a local level, with drift and as a local linear trend, the Nile series with seven values removed (two at the very start), and the log of the UK driver series with a 12-month seasonal. Loglikelihoods, smoothed states and their variances, smoothed disturbances and their variances, and one-step errors all agree to a relative 1e-10 or better (the largest difference is 1.1e-11; eval/agreement.mjs prints them). The tool's own maximum likelihood fit is at least as high as statsmodels' in every case, and on the Nile local level model it gives variances of 15,098.5 and 1,469.2, against the 15,099 and 1,469.1 reported by Durbin & Koopman (2012, chapter 2).

BibTeX

@book{durbin2012time,
  author    = {Durbin, James and Koopman, Siem Jan},
  title     = {Time Series Analysis by State Space Methods},
  edition   = {2},
  publisher = {Oxford University Press}, year = {2012},
  doi       = {10.1093/acprof:oso/9780199641178.001.0001}
}
@article{koopman1997exact,
  author  = {Koopman, Siem Jan},
  title   = {Exact Initial {K}alman Filtering and Smoothing for Nonstationary Time Series Models},
  journal = {Journal of the American Statistical Association},
  volume  = {92}, number = {440}, pages = {1630--1638}, year = {1997},
  doi     = {10.1080/01621459.1997.10473686}
}
@book{harvey1989forecasting,
  author    = {Harvey, Andrew C.},
  title     = {Forecasting, Structural Time Series Models and the {K}alman Filter},
  publisher = {Cambridge University Press}, year = {1989},
  doi       = {10.1017/CBO9781107049994}
}
@article{harvey1992diagnostic,
  author  = {Harvey, Andrew C. and Koopman, Siem Jan},
  title   = {Diagnostic Checking of Unobserved-Components Time Series Models},
  journal = {Journal of Business \& Economic Statistics},
  volume  = {10}, number = {4}, pages = {377--389}, year = {1992},
  doi     = {10.1080/07350015.1992.10509913}
}
@article{dejong1998diagnosing,
  author  = {de Jong, Piet and Penzer, Jeremy},
  title   = {Diagnosing Shocks in Time Series},
  journal = {Journal of the American Statistical Association},
  volume  = {93}, number = {442}, pages = {796--806}, year = {1998},
  doi     = {10.1080/01621459.1998.10473731}
}

Limits

Questions

What does the Kalman smoother do here?

It splits each value of your series into an underlying level that drifts slowly and noise around it, using every value before and after each point. How much of the movement is level and how much is noise is estimated from the data by maximum likelihood, so you do not pick a window or a smoothing constant.

How is this different from a moving average?

A moving average uses a fixed window you choose, loses the ends of the series and gives no band. The smoother chooses its own amount of smoothing from the data, runs to both ends, fills gaps, and says how sure it is at every point. A local level model's smoother is in fact a weighted moving average whose weights the data choose.

What is the difference between an outlier and a level shift?

An outlier is one value far from the level, after which the series returns to where it was. A level shift is the level itself moving and staying moved. The tool tests each point both ways with auxiliary residuals: the standardised smoothed irregular for outliers, and the standardised smoothed level disturbance for shifts. A t value beyond about 2.5 is worth a look.

Which model should I choose?

Leave it on automatic: the tool fits a local level, a random walk with drift and a local linear trend and keeps the one with the lowest AIC, counting the diffuse starting values as Durbin and Koopman do. Add a seasonal period for monthly (12), quarterly (4) or daily data with a weekly pattern (7). Use the log option when swings grow with the level, as in sales or counts.

Can my series have missing values?

Yes. Leave the cell blank or write NA. The filter skips the update at that point, and the smoother fills it from both sides with a wider band. The rows must still be one per period, equally spaced, with the gap kept as a row.

How was it checked?

The filter and smoother are compared with statsmodels' UnobservedComponents with exact diffuse initialisation on five cases, including gaps at the start and a 12-month seasonal: loglikelihoods, smoothed states and variances, smoothed disturbances and one-step errors agree to a relative 1e-10 or better. On the Nile series the maximum likelihood variances are 15,098.5 and 1,469.2, against the 15,099 and 1,469.1 Durbin and Koopman report.

Is my data uploaded anywhere?

No. The file is read and fitted by JavaScript in this browser tab. There are no accounts, cookies or analytics vendors.

What the examples show: on the Nile (yearly flow at Aswan in 108 m³, 1871-1970, public domain; Cobb 1978, doi:10.1093/biomet/65.2.243) the largest level shift is between 1898 and 1899 (t about −3), the drop usually linked to the first Aswan dam, and 1913 is the largest outlier. On the UK driver series (car drivers killed or seriously injured per month, 1969-1984; Harvey & Durbin 1986), with logs and a 12-month seasonal, the largest shift is between January and February 1983 (t about −3.8): the front-seat belt law took effect on 31 January 1983. Neither date is given to the tool.