Forecasting in State Space: theory and practice

Siem Jan Koopman

http://personal.vu.nl/s.j.koopman

Department of EconometricsVU University Amsterdam

Tinbergen Institute2012

Program

Lectures :

• Introduction to UC models

• State space methods

• Forecasting time series with different components

• Practice of Forecasting with Illustrations

Exercises and assignments will be part of the course.

2 / 42

Time Series

A time series is a set of observations yt , each one recorded at aspecific time t.

The observations are ordered over time.We assume to have n observations, t = 1, . . . , n.

Examples of time series are:

• Number of cars sold each year

• Gross Domestic Product of a country

• Stock prices during one day

• Number of firm defaults

Our purpose is to identify and to model the serial or “dynamic”correlation structure in the time series.

Time series analysis may be relevant for economic policy, financialdecision making and forecasting

3 / 42

Example: Nile data

1870 1880 1890 1900 1910 1920 1930 1940 1950 1960 1970

500

600

700

800

900

1000

1100

1200

1300

1400 Nile Data

4 / 42

Example: GDP growth, quarter by quarter

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

−4

−3

−2

−1

0

1

2

3

4

5

5 / 42

Example: winner boat races Cambridge/Oxford

1840 1860 1880 1900 1920 1940 1960 1980 2000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

6 / 42

Example: US monthly unemployment

7 / 42

Sources for time series data

Data sources :

• US economics :http://research.stlouisfed.org/fred2/

• DK book data : http://www.ssfpack.com/dkbook.html

• Financial data : Datastream, Yahoo Finance

8 / 42

9:

9 / 42

White noise processes

Simplest example of a stationary process is a white noise (WN)process which we usually denote as εt .

A white noise process is a sequence of uncorrelated randomvariables, each with zero mean and constant variance σ2

ε :

εt ∼ WN(0, σ2ε ).

The autocovariance function is equal to zero for lags h > 0:

γY (h) =

{

σ2ε if h = 0,

0 if h 6= 0.

9 / 42

White noise realisations

0 50 100 150 200 250 300 350 400 450 500

−3

−2

−1

0

1

2

White Noise, 500 observations

10 / 42

White noise ACF and SACF

0 10 20 30 40 50

−0.05

0.00

0.05

0.10 Theoretical ACF max lag = 50

0 100 200 300 400 500

−0.05

0.00

0.05

0.10 Theoretical ACF max lag = 500

0 10 20 30 40 50

−0.5

0.0

0.5

1.0 Sample ACF n = 50ACF−

0 100 200 300 400 500

−0.5

0.0

0.5

1.0 Sample ACF n = 500ACF−

11 / 42

Random Walk processes

If ε1, ε2, . . . come from a white noise process with variance σ2,then the process {Yt} with

Yt = ε1 + ε2 + . . .+ εt for t = 1, 2, . . .

is called a random walk.

A recursive way to define a random walk is:

Yt = Yt−1 + εt for t = 2, 3, . . .

Y1 = ε1

12 / 42

Random Walk properties I

A random walk is not stationary, because the variance of Yt istime-varying:

E(Yt) = E(ε1 + . . .+ εt) = 0

Var(Yt) = E(Y 2t ) = E[(ε1 + . . . + εt)

2] = tσ2

The autocovariance function is equal to:

γ(t, t − h) = E(YtYt−h)

= E[(

t−h∑

j=1

εj +

t∑

j=t−h+1

εj)(

t−h∑

j=1

εj )]

= (t − h)σ2

This means that the variance and the autocovariances go toinfinity if t → ∞.

13 / 42

Random Walk properties II

The autocorrelation of Yt and Yt−h is

ρ(t, t − h) =γ(t, t − h)

√

Var(Yt)Var(Yt−h)

=(t − h)σ2

√

(tσ2)((t − h)σ2)=

√t − h√t

14 / 42

RW realisation

0 50 100 150 200 250 300 350 400 450 500

−30

−25

−20

−15

−10

−5

0

5

Random Walk, 500 observations

15 / 42

RW sample ACF

0 5 10 15 20 25 30 35 40 45 50

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0 Sample ACF 50 lags

ACF−

16 / 42

Seatbelt Law

70 75 80 85

7.0

7.1

7.2

7.3

7.4

7.5

7.6

7.7

7.8

7.9

17 / 42

Classical Decomposition

A basic model for representing a time series is the additive model

yt = µt + γt + εt , t = 1, . . . , n,

also known as the Classical Decomposition.

yt = observation,

µt = slowly changing component (trend),

γt = periodic component (seasonal),

εt = irregular component (disturbance).

In a Structural Time Series Model (STSM) or UnobservedComponents Model (UCM), the RHS components are modelledexplicitly as stochastic processes.

18 / 42

Nile data

1870 1880 1890 1900 1910 1920 1930 1940 1950 1960 1970

500

600

700

800

900

1000

1100

1200

1300

1400 Nile Data

19 / 42

Local Level Model

• Components can be deterministic functions of time (e.g.polynomials), or stochastic processes;

• Deterministic example: yt = µ+ εt with εt ∼ NID(0, σ2ε ).

• Stochastic example: the Random Walk plus Noise, orLocal Level model:

yt = µt + εt , εt ∼ NID(0, σ2ε )

µt+1 = µt + ηt , ηt ∼ NID(0, σ2η),

• The disturbances εt , ηs are independent for all s, t;

• The model is incomplete without a specification for µ1 (notethe non-stationarity):

µ1 ∼ N (a,P)

20 / 42

Local Level Model

yt = µt + εt , εt ∼ NID(0, σ2ε )

µt+1 = µt + ηt , ηt ∼ NID(0, σ2η),

µ1 ∼ N (a,P)

• The level µt and the irregular εt are unobserved;

• Parameters: a,P , σ2ε , σ

2η ;

• Trivial special cases:• σ2

η= 0 =⇒ yt ∼ NID(µ1, σ

2ε) (WN with constant level);

• σ2ε= 0 =⇒ yt+1 = yt + ηt (pure RW);

• Local Level is a model representation for EWMA forecasting.

21 / 42

Simulated LL Data

0 10 20 30 40 50 60 70 80 90 100

−6

−4

−2

0

2

4

6

σε2=0.1 ση

2=1 y µ

22 / 42

Simulated LL Data

0 10 20 30 40 50 60 70 80 90 100

−6

−4

−2

0

2

4

6

σε2=1 ση

2=1

23 / 42

Simulated LL Data

0 10 20 30 40 50 60 70 80 90 100

−6

−4

−2

0

2

4

6

σε2=1 ση

2=0.1

24 / 42

Simulated LL Data

0 10 20 30 40 50 60 70 80 90 100

−5

0

5 σε2=0.1 ση

2=1 y µ

0 10 20 30 40 50 60 70 80 90 100

−5

0

5 σε2=1 ση

2=1

0 10 20 30 40 50 60 70 80 90 100

−2

0

2 σε2=1 ση

2=0.1

25 / 42

Properties of the LL model

yt = µt + εt , εt ∼ NID(0, σ2ε ),

µt+1 = µt + ηt , ηt ∼ NID(0, σ2η),

• First difference is stationary:

∆yt = ∆µt +∆εt = ηt−1 + εt − εt−1.

• Dynamic properties of ∆yt :

E(∆yt) = 0,

γ0 = E(∆yt∆yt) = σ2η + 2σ2

ε ,

γ1 = E(∆yt∆yt−1) = −σ2ε ,

γτ = E(∆yt∆yt−τ ) = 0 for τ ≥ 2.

26 / 42

Properties of the LL model

• The ACF of ∆yt is

ρ1 =−σ2

ε

σ2η + 2σ2

ε

= − 1

q + 2, q = σ2

η/σ2ε ,

ρτ = 0, τ ≥ 2.

• q is called the signal-noise ratio;

• The model for ∆yt is MA(1) with restricted parameters suchthat

−1/2 ≤ ρ1 ≤ 0

i.e., yt is ARIMA(0,1,1);

• Write ∆yt = ξt + θξt−1, ξt ∼ NID(0, σ2) to solve θ:

θ =1

2

(

√

q2 + 4q − 2− q)

.

27 / 42

Local Level Model

• The model parameters are estimated by Maximum Likelihood;

• Advantages of model based approach: assumptions can betested, parameters are estimated rather than “calibrated”;

• Estimated model can be used for signal extraction;

• The estimated level µt is obtained as a locally weightedaverage;

• The distribution of weights can be compared with Kernelfunctions in nonparametric regressions;

• Within the model, our methods yield MMSE forecasts.

28 / 42

Signal Extraction and Weights for the Nile Data

1880 1900 1920 1940 1960

500

750

1000

1250data and estimated level

−20 −15 −10 −5 0 5 10 15 20

0.01

0.02

weights

1880 1900 1920 1940 1960

500

750

1000

1250

−20 −15 −10 −5 0 5 10 15 20

0.05

0.10

0.15

1880 1900 1920 1940 1960

500

750

1000

1250

1500

−20 −15 −10 −5 0 5 10 15 20

0.25

0.50

29 / 42

Local Linear Trend Model

The LLT model extends the LL model with a slope:

yt = µt + εt , εt ∼ NID(0, σ2ε ),

µt+1 = βt + µt + ηt , ηt ∼ NID(0, σ2η),

βt+1 = βt + ξt , ξt ∼ NID(0, σ2ξ ).

• All disturbances are independent at all lags and leads;

• Initial distributions β1, µ1 need to specified;

• If σ2ξ = 0 the trend is a random walk with constant drift β1;

(For β1 = 0 the model reduces to a LL model.)

• If additionally σ2η = 0 the trend is a straight line with slope β1

and intercept µ1;

• If σ2ξ > 0 but σ2

η = 0, the trend is a smooth curve, or anIntegrated Random Walk;

30 / 42

Trend and Slope in LLT Model

0 10 20 30 40 50 60 70 80 90 100

−2.5

0.0

2.5

5.0µ

0 10 20 30 40 50 60 70 80 90 100

−0.25

0.00

0.25

0.50

0.75 β

31 / 42

Trend and Slope in Integrated Random Walk Model

0 10 20 30 40 50 60 70 80 90 100

0

5

10 µ

0 10 20 30 40 50 60 70 80 90 100

−0.25

0.00

0.25

0.50

0.75 β

32 / 42

Local Linear Trend Model

• Reduced form of LLT is ARIMA(0,2,2);

• LLT provides a model for Holt-Winters forecasting;

• Smooth LLT provides a model for spline-fitting;

• Smoother trends: higher order Random Walks

∆dµt = ηt

33 / 42

Seasonal Effects

We have seen specifications for µt in the basic model

yt = µt + γt + εt .

Now we will consider the seasonal term γt . Let s denote thenumber of ‘seasons’ in the data:

• s = 12 for monthly data,

• s = 4 for quarterly data,

• s = 7 for daily data when modelling a weekly pattern.

34 / 42

Dummy Seasonal

The simplest way to model seasonal effects is by using dummyvariables. The effect summed over the seasons should equal zero:

γt+1 = −s−1∑

j=1

γt+1−j .

To allow the pattern to change over time, we introduce a newdisturbance term:

γt+1 = −s−1∑

j=1

γt+1−j + ωt , ωt ∼ NID(0, σ2ω).

The expectation of the sum of the seasonal effects is zero.

35 / 42

Trigonometric Seasonal

Defining γjt as the effect of season j at time t, an alternativespecification for the seasonal pattern is

γt =

[s/2]∑

j=1

γjt ,

γj ,t+1 = γjt cos λj + γ∗jt sinλj + ωjt ,

γ∗j ,t+1 = −γjt sinλj + γ∗jt cos λj + ω∗

jt ,

ωjt , ω∗

jt ∼ NID(0, σ2ω), λj = 2πj/s.

• Without the disturbance, the trigonometric specification isidentical to the deterministic dummy specification.

• The autocorrelation in the trigonometric specification laststhrough more lags: changes occur in a smoother way;

36 / 42

Unobserved Component Models

• Different specifications for the trend and the seasonal can befreely combined.

• Other components of interest, like cycles, explanatoryvariables, interventions effects, outliers, are easily added.

• UC models are Multiple Source of Errors models. The reducedform is a Single Source of Errors model.

• We model non-stationarity directly.

• Components have an explicit interpretation: the model is notjust a forecasting device.

37 / 42

Seatbelt Law

70 75 80 85

7.0

7.1

7.2

7.3

7.4

7.5

7.6

7.7

7.8

7.9

38 / 42

Seatbelt Law: decomposition

drivers Level+Reg

70 75 80 85

7.25

7.75drivers Level+Reg

drivers−Seasonal

70 75 80 85

0.0

0.2drivers−Seasonal

drivers−Irregular

70 75 80 85

−0.1

0.0

0.1drivers−Irregular

39 / 42

Seatbelt Law: forecasting

70 75 80 85

7.0

7.1

7.2

7.3

7.4

7.5

7.6

7.7

7.8

7.9

40 / 42

Textbooks

• A.C.Harvey (1989). Forecasting, Structural Time SeriesModels and the Kalman Filter. Cambridge University Press

• G.Kitagawa & W.Gersch (1996). Smoothness Priors Analysisof Time Series. Springer-Verlag

• J.Harrison & M.West (1997). Bayesian Forecasting andDynamic Models. Springer-Verlag

• J.Durbin & S.J.Koopman (2001). Time Series Analysis byState Space Methods. Oxford University Press

• J.J.F.Commandeur & S.J.Koopman (2007). An Introductionto State Space Time Series Analysis. Oxford University Press

41 / 42

Exercises

1. Consider LL model (see slides, see DK chapter 2).• Reduced form is ARIMA(0,1,1) process. Derive the

relationship between signal-to-noise ratio q of LL model andthe θ coefficient of the ARIMA model;

• Derive the reduced form in the case ηt =√qεt and notice the

difference in the general case.• Give the elements of the mean vector and variance matrix of

y = (y1, . . . , yn)′ when yt is generated by a LL model for

t = 1, . . . , n.

2. Consider LLT model (see slides, see DK section 3.2.1).• Show that the reduced form is an ARIMA(0,2,2) process;• Discuss the initial values for level and slope of LLT;• Relate the LLT model forecasts with the Holt-Winters method

of forecasting. Comment.

42 / 42