Your series
Blank, NA or “.” cells are missing values; keep them as rows so the periods stay equally spaced.
Result
The series and its smoothed level
data smoothed level band outlier (ring) and level shift (dashed line) forecast
Level shifts
| Between | t | Level before | Level after |
|---|
Outliers
| Period | t | Value | Level |
|---|
Each t is a standardised auxiliary residual (Harvey & Koopman 1992): roughly the t statistic for a one-off outlier at that period, or for a level shift between two periods. Many periods are tested, so a few values past 2 are expected by chance; past 3 is rare.
Forecast
| Ahead | Forecast | Interval for a new value | Level |
|---|
Model
| Quantity | Value |
|---|
Method and citation
The series yt is modelled as a level plus noise, yt = μt + γt + εt, where the level is a random walk μt+1 = μt + βt + ηt, the slope β is absent, fixed (a drift) or itself a random walk (the local linear trend), and γ is an optional dummy seasonal whose s values sum to about zero over a year (Harvey 1989). Every disturbance is normal with its own variance. Written in state-space form, the Kalman filter gives each period's one-step prediction and its error variance; the smoother runs backwards to give the level at each period from all the data, with its variance (Durbin & Koopman 2012, chapter 4).
The starting level, slope and seasonal values are unknown, so they get the exact diffuse initialisation (Koopman 1997; Durbin & Koopman 2012, sections 5.2 and 5.3): the state variance is split into an infinite and a finite part, and the first observations are spent pinning the start down instead of guessing a huge prior variance. The variances are chosen by maximum likelihood on the diffuse loglikelihood with the irregular variance concentrated out, by Nelder-Mead from a grid of starts. With the trend on automatic, the three trend forms are compared by AIC with the diffuse elements counted as parameters (Durbin & Koopman 2012, section 7.4). Missing values skip the filter's update step. Outliers and level shifts are read from the auxiliary residuals, the smoothed disturbances divided by their standard errors (Harvey & Koopman 1992; de Jong & Penzer 1998). The one-step errors are checked for autocorrelation (Ljung-Box), normality (Bowman-Shenton) and a change in noise level (the H test).
Verified by tests in the source against statsmodels 0.15.0's UnobservedComponents with exact diffuse initialisation, at equal variances, on five cases: the Nile series as a local level, with drift and as a local linear trend, the Nile series with seven values removed (two at the very start), and the log of the UK driver series with a 12-month seasonal. Loglikelihoods, smoothed states and their variances, smoothed disturbances and their variances, and one-step errors all agree to a relative 1e-10 or better (the largest difference is 1.1e-11; eval/agreement.mjs prints them). The tool's own maximum likelihood fit is at least as high as statsmodels' in every case, and on the Nile local level model it gives variances of 15,098.5 and 1,469.2, against the 15,099 and 1,469.1 reported by Durbin & Koopman (2012, chapter 2).
BibTeX
@book{durbin2012time,
author = {Durbin, James and Koopman, Siem Jan},
title = {Time Series Analysis by State Space Methods},
edition = {2},
publisher = {Oxford University Press}, year = {2012},
doi = {10.1093/acprof:oso/9780199641178.001.0001}
}
@article{koopman1997exact,
author = {Koopman, Siem Jan},
title = {Exact Initial {K}alman Filtering and Smoothing for Nonstationary Time Series Models},
journal = {Journal of the American Statistical Association},
volume = {92}, number = {440}, pages = {1630--1638}, year = {1997},
doi = {10.1080/01621459.1997.10473686}
}
@book{harvey1989forecasting,
author = {Harvey, Andrew C.},
title = {Forecasting, Structural Time Series Models and the {K}alman Filter},
publisher = {Cambridge University Press}, year = {1989},
doi = {10.1017/CBO9781107049994}
}
@article{harvey1992diagnostic,
author = {Harvey, Andrew C. and Koopman, Siem Jan},
title = {Diagnostic Checking of Unobserved-Components Time Series Models},
journal = {Journal of Business \& Economic Statistics},
volume = {10}, number = {4}, pages = {377--389}, year = {1992},
doi = {10.1080/07350015.1992.10509913}
}
@article{dejong1998diagnosing,
author = {de Jong, Piet and Penzer, Jeremy},
title = {Diagnosing Shocks in Time Series},
journal = {Journal of the American Statistical Association},
volume = {93}, number = {442}, pages = {796--806}, year = {1998},
doi = {10.1080/01621459.1998.10473731}
}
Limits
- Equally spaced periods. One row per period, in order. A gap must stay as a row with a blank value; dropping the row tells the model two periods are one.
- A flag is a lead, not a verdict. Every period is tested, so with 100 periods a value or two past |t| 2.5 can turn up by chance; a level shift and an outlier next to each other can also share one cause. Check the dates against what you know happened.
- Normal noise. The bands assume normal disturbances. If the residual check says the errors are not normal, or the noise grows over time, read the bands as approximate, and try the log option.
- The log option reports medians. Levels and bands are turned back into your units with
exp, which keeps the band's coverage but gives the median, not the mean. - Size. Up to 20,000 periods; a long series with a long seasonal (52 weeks) takes a few seconds to fit.
- Not modelled: trigonometric seasonals, cycles, several series at once, explanatory variables or intervention dummies, and non-normal (count) noise. For a dated change in the mean or variance with a posterior over when, see the changepoint detector; for a trend test, Mann-Kendall; for a seasonal split without a model, STL decomposition.
Questions
What does the Kalman smoother do here?
It splits each value of your series into an underlying level that drifts slowly and noise around it, using every value before and after each point. How much of the movement is level and how much is noise is estimated from the data by maximum likelihood, so you do not pick a window or a smoothing constant.
How is this different from a moving average?
A moving average uses a fixed window you choose, loses the ends of the series and gives no band. The smoother chooses its own amount of smoothing from the data, runs to both ends, fills gaps, and says how sure it is at every point. A local level model's smoother is in fact a weighted moving average whose weights the data choose.
What is the difference between an outlier and a level shift?
An outlier is one value far from the level, after which the series returns to where it was. A level shift is the level itself moving and staying moved. The tool tests each point both ways with auxiliary residuals: the standardised smoothed irregular for outliers, and the standardised smoothed level disturbance for shifts. A t value beyond about 2.5 is worth a look.
Which model should I choose?
Leave it on automatic: the tool fits a local level, a random walk with drift and a local linear trend and keeps the one with the lowest AIC, counting the diffuse starting values as Durbin and Koopman do. Add a seasonal period for monthly (12), quarterly (4) or daily data with a weekly pattern (7). Use the log option when swings grow with the level, as in sales or counts.
Can my series have missing values?
Yes. Leave the cell blank or write NA. The filter skips the update at that point, and the smoother fills it from both sides with a wider band. The rows must still be one per period, equally spaced, with the gap kept as a row.
How was it checked?
The filter and smoother are compared with statsmodels' UnobservedComponents with exact diffuse initialisation on five cases, including gaps at the start and a 12-month seasonal: loglikelihoods, smoothed states and variances, smoothed disturbances and one-step errors agree to a relative 1e-10 or better. On the Nile series the maximum likelihood variances are 15,098.5 and 1,469.2, against the 15,099 and 1,469.1 Durbin and Koopman report.
Is my data uploaded anywhere?
No. The file is read and fitted by JavaScript in this browser tab. There are no accounts, cookies or analytics vendors.
What the examples show: on the Nile (yearly flow at Aswan in 108 m³, 1871-1970, public domain; Cobb 1978, doi:10.1093/biomet/65.2.243) the largest level shift is between 1898 and 1899 (t about −3), the drop usually linked to the first Aswan dam, and 1913 is the largest outlier. On the UK driver series (car drivers killed or seriously injured per month, 1969-1984; Harvey & Durbin 1986), with logs and a 12-month seasonal, the largest shift is between January and February 1983 (t about −3.8): the front-seat belt law took effect on 31 January 1983. Neither date is given to the tool.