Post on 13-Jan-2016
TIME SERIES ANALYSISTime Domain Models: Red Noise; AR
and ARMA models
LECTURE 7
Supplementary Readings:
Wilks, chapters 8
Recall:
Statistical Model
[Observed Data] = [Signal] + [Noise]
“noise” has to satisfy certain properties!
If not, we must iterate on this process...
We will seek a general method of specifying just such a model…
Statistics of the time series don’t change with time
We conventionally assume Stationarity
•Weak Stationarity (Covariance Stationarity)
statistics are a function of lag k but not absolute time t
•Strict Stationarity
•Cyclostationarity
statistics are a periodic function of lag k
Time Series ModelingAll linear time series models can be written in the form:
1111
11
)(
mt
M
mtkt
K
kkt
mxx
We assume that are Gaussian distributed.
Autoregressive Moving-Average Model(“ARMA”)
Box and Jenkins (1976)
ARMA(K,M) MODEL
111
1)(
tkt
K
kkt
xx Consider Special case of simple Autoregressive AR(k) model (m=0)
Suppose k=1, and define 1
11)(
tttxx
This should look familiar!
Special case of a Markov Process (a process for which the state of system depends on previous states)
Time Series ModelingAll linear time series models can be written in the form:
1111
11
)(
mt
M
mtkt
K
kkt
mxx
Suppose k=1, and define 1
11)(
tttxx
Time Series Modeling
Assume process is zero mean, then
11
tttxx
Lag-one correlated process or “AR(1)” process…
111
1)(
tkt
K
kkt
xx Consider Special case of simple Autoregressive AR(k) model (m=0)
All linear time series models can be written in the form:
1111
11
)(
mt
M
mtkt
K
kkt
mxx
11
kky
ky
ky
kky
ky
ky
12
1
ky
kky
ky
ky
n
k
n
k
n
k 12
11
1
1
1
1
1
21
1
1
1
1
ky
ky
ky
n
k
n
k
For simplicity, we assume zero mean
2
1
y
yy
nlii
l
AR(1) Process
-4 -3 -2 -1 0 1 2 3 40
20
40
60
80
100
120
140
Let us take this series as a random forcing
0 100 200 300 400 500 600 700 800 900 1000-4
-3
-2
-1
0
1
2
3
4
85.02
181120 N
AR(1) Process
0 100 200 300 400 500 600 700 800 900 1000-4
-3
-2
-1
0
1
2
3
4
0 10 20 30 40 50 60 70 80 90 100-3
-2
-1
0
1
2
3
4
AR(1) Process
Blue: =0.4Red: =0.7
Let us take this series as a random forcing
What is the standard deviation of an AR(1) process?
2222
yy
2)1
(21
kk
yk
y
11
kky
ky
222
112
kk
yk
y
2
2
1
2
y
AR(1) Process
0 10 20 30 40 50 60 70 80 90 100-6
-4
-2
0
2
4
6
8
10
Blue: =0.4Red: =0.7Green: =0.9
2
2
1
2
y
AR(1) Process
010020030040050060070080090010000
5
10
15
20
25
30
35
40
45
50
Suppose =1
Random Walk (“Brownian Motion”)
Not stationary
How might we try to turn this into a stationary time
series?
AR(1) Process
2
2
1
2
y
11
kky
ky
Variance is infinite!
11
kky
ky
Let us define the lag-k autocorrelation:
We can approximate:
2)(1
xxvari
n
i
i
n
i
xx 1
varkn
xxxx
kr kii
kn
i
)(
))((1
AR(1) ProcessAutocorrelation Function
i
kn
i
xx
1
i
n
ki
xx
1
2)(1
xxvar
i
kn
i
2)(1
xxvar
i
n
ki
varvarkn
xxxx
kr kii
kn
i
)(
))((1
Let us assume the series x has been de-meaned…
Let us define the lag-k autocorrelation:
2
1i
n
i
xvar
varkn
xx
kr kii
kn
i
)(1
AR(1) ProcessAutocorrelation Function
Let us define the lag-k autocorrelation:
Then:
i
kn
i
xx
1
i
n
ki
xx
1
2)(1
xxvar
i
kn
i
2)(1
xxvar
i
n
ki
varvarkn
xxxx
kr kii
kn
i
)(
))((1
0 50 100 150 200 250-0.4
-0.2
0
0.2
0.4
0.6
0.8
1Serial correlation function
Autocorrelation Function
1976 1978 1980 1982 1984 1986 1988 1990 1992 1994 1996336
338
340
342
344
346
348
350
352
354
356
CO2 since 1976
varkn
xx
kr kii
kn
i
)(1
11
kky
ky
Recursively, we thus have for an AR(1) process,k
kr
AR(1) ProcessAutocorrelation Function
kky
ky
1
1]
1[
kkky
)1
(1
2
kkky
kky
ky '
112
)/exp()lnexp( kkk
rk
(Theoretical)
varkn
xx
kr kii
kn
i
)(1
0 5 10 15 20 25-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
“tcf.m” scf.m
0 10 20 30 40 50 60 70 80 90 100-2
-1
0
1
2
3
4
=0.5 N=100
“rednoise.m”
)/exp()lnexp( kkk
rk
AR(1) ProcessAutocorrelation Function
varkn
xx
kr kii
kn
i
)(1
0 5 10 15 20 25-0.2
0
0.2
0.4
0.6
0.8
1
1.2
=0.5 N=100
“rednoise.m”
)/exp()lnexp( kkk
rk
AR(1) ProcessAutocorrelation Function
0 50 100 150 200 250 300 350 400 450 500-4
-3
-2
-1
0
1
2
3
4
5
=0.5 N=500
“rednoise.m”
varkn
xx
kr kii
kn
i
)(1
0 5 10 15 20 25-0.2
0
0.2
0.4
0.6
0.8
1
1.2
0 100 200 300 400 500 600 700 800 900 1000-4
-3
-2
-1
0
1
2
3
4
)/exp()lnexp( kkk
rk
AR(1) ProcessAutocorrelation Function
Glacial Varves
=0.23 N=1000
varkn
xx
kr kii
kn
i
)(1
0 5 10 15 20 250
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 50 100 150-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
)/exp()lnexp( kkk
rk
AR(1) ProcessAutocorrelation Function
Northern Hem Temp
=0.75 N=144
varkn
xx
kr kii
kn
i
)(1
0 5 10 15 20 25-0.2
0
0.2
0.4
0.6
0.8
1
1.2
0 50 100 150-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
)/exp()lnexp( kkk
rk
AR(1) ProcessAutocorrelation Function
Northern Hem Temp (linearly detrended)
=0.54 N=144
varkn
xx
kr kii
kn
i
)(1
1 2 3 4 5 6 7 8 9 10 11-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
)/exp()lnexp( kkk
rk
AR(1) ProcessAutocorrelation Function
0 50 100 150-2
-1
0
1
2
3
4
Dec-Mar Nino3
AR(1) Process
The sampling distribution for is given by the sampling distribution for the slope parameter in linear regression!
n/)1( 22
11
kky
ky
2/1
)(1
2
xxesbi
2
2
1
2
y
AR(1) Process
The sampling distribution for is given by the sampling distribution for the slope parameter in linear regression!
n/)1( 22
How do we determine if is significantly non-zero?
n
t
/21
0
This is just the t test!
11
kky
ky
When Serial Correlation is Present, the variance of the mean must be adjusted,
nxVx
2)var(
121k k
rV
k
k Recall for AR(1) series,
AR(1) Process
Variance inflation factor
This effects the significance of regression/correlation as we saw previously…
Vnn /'
111
121
1121V
11
211
121
V?V
This effects the significance of regression/correlation as we saw previously…
111
121
1121V
AR(1) Process
12 2
)/exp()lnexp( kkk
k
ln/1
Suppose 1 1ln 1/1
Vnn /'
Multiply this equation by xt-k’ and sum,
111
1)(
tkt
K
kkt
xx
Now consider an AR(K) Process
1'11
'1'
tktkt
K
kkkttkt
xxxxx
For simplicity, we assume zero mean
111
1
tkt
K
kktxx
11
'1'
kt
K
kkkttktxxxx
kKKKK
kk
kk
kk
rrrr
rrrr
rrrr
rrrr
ˆ...ˆˆˆ...
ˆ...ˆˆˆ
ˆ...ˆˆˆ
ˆ...ˆˆˆ
332211
3312213
2132112
1231211
Yule-Walker Equations
Use
varkn
xx
kr kii
kn
i
)(1
11
'1'
kt
K
kkkttktxxxx
AR(K) Process
kk
x
rK
k
11
22
Several results obtained for the AR(1) model generalize readily to the AR(K) model:
kmk
K
km
rr
1 nxVx
2)var(
121k k
rV
AR(K) Process
kKKKK
kk
kk
kk
rrrr
rrrr
rrrr
rrrr
ˆ...ˆˆˆ...
ˆ...ˆˆˆ
ˆ...ˆˆˆ
ˆ...ˆˆˆ
332211
3312213
2132112
1231211
Yule-Walker Equations
AR(K) Process
kKKKK
kk
kk
kk
rrrr
rrrr
rrrr
rrrr
ˆ...ˆˆˆ...
ˆ...ˆˆˆ
ˆ...ˆˆˆ
ˆ...ˆˆˆ
332211
3312213
2132112
1231211
Yule-Walker Equations
is particularly important because of the range of behavior it can describe w/ a parsimonious number of parameters
11211
ttttxxx The AR(2) model
The Yule-Walker equations give:
2112
1211
ˆˆ
ˆˆ
rr
rr
kk
x
rK
k
11
22
AR(2) Process
is particularly important because of the range of behavior it can describe w/ a parsimonious number of parameters
11211
ttttxxx The AR(2) model
The Yule-Walker equations give:
2112
1211
ˆˆ
ˆˆ
rr
rr
Which readily gives:
2
1
2
122
2
1
211
1ˆ
1
)1(ˆ
r
rr
r
rr
)1)(1(
2
2
1
2
2
2
rx
)1/(
)1/(
22122
211
rr
kk
x
rK
k
11
22
kmk
K
km
rr
12
AR(2) ProcessWhich readily gives:
2
1
2
122
2
1
211
1ˆ
1
)1(ˆ
r
rr
r
rr
)1)(1(
2
2
1
2
2
2
rx
)1/(
)1/(
22122
211
rr
0 1 2-1-2
+1
-1 1
2
1ˆˆ1ˆˆ
1ˆ
12
21
2
For stationarity, we must have:
kmk
K
km
rr
12
AR(2) ProcessWhich readily gives:
)1/(
)1/(
22122
211
rr
0 1 2-1-2
+1
-1 1
2
1ˆˆ1ˆˆ
1ˆ
12
21
2
For stationarity, we must have:
Note that this model allows for independent lag-1 and lag-2
correlation, so that both positive correlation and negative correlation
are possible...kmk
K
km
rr
12
AR(2) ProcessWhich readily gives:
)1/(
)1/(
22122
211
rr
1ˆˆ1ˆˆ
1ˆ
12
21
2
For stationarity, we must have:
Note that this model allows for independent lag-1 and lag-2
correlation, so that both positive correlation and negative correlation
are possible...
0 2 4 6 8 10 12 14 16 18 200
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
4.0ˆ3.0ˆ
2
1
“artwo.m”
kmk
K
km
rr
12
Selection Rules
AR(K) Process
nmmsmnnnmBIC )ln1()(
1ln)( 2
Bayesian Information Criterion
)1(2)(1
ln)( 2
mmsmnnnmAIC
Akaike Information Criterion
The minima in AIC or BIC represent an ‘optimal’ tradeoff between degrees of freedom and variance explained
ENSO
Multivariate ENSO Index
(“MEI”)
AR(K) Process
varkn
xx
kr kii
kn
i
)(1
1 2 3 4 5 6 7 8 9 10 11-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
AR(1) FitAutocorrelation Function
0 50 100 150-2
-1
0
1
2
3
4
Dec-Mar Nino3
)/exp()lnexp( kkk
rk
varkn
xx
kr kii
kn
i
)(1
1 2 3 4 5 6 7 8 9 10 11-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
AR(2) FitAutocorrelation Function
0 50 100 150-2
-1
0
1
2
3
4
Dec-Mar Nino3
)1/(
)1/(
22
122
211
rr kmk
K
km
rr
1
(m>2)
varkn
xx
kr kii
kn
i
)(1
1 2 3 4 5 6 7 8 9 10 11-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
AR(3) FitAutocorrelation Function
0 50 100 150-2
-1
0
1
2
3
4
Dec-Mar Nino3
Theoretical AR(3) Fit
varkn
xx
kr kii
kn
i
)(1
1 2 3 4 5 6 7 8 9 10 11-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
AR(K) FitAutocorrelation Function
0 50 100 150-2
-1
0
1
2
3
4
Dec-Mar Nino3
Favors AR(K) Fit for K=?
Minimum in BIC?
Minimum in AIC?
MA model
1111
11
)(
mt
M
mtkt
K
kkt
mxx
Now, consider the case k=0
1111
mt
M
mttm
x
Pure Moving Average (MA) model, represents a running mean of the past M values.
Consider case where M=1…
MA(1) model
1111
11
)(
mt
M
mtkt
K
kkt
mxx
Now, consider the case k=0
1111
mt
M
mttm
x tt
11
)1( 2
122
x
)1/( 2
111 r
Consider case where M=1…
ARMA(1,1) model
ttttxx
1111)(
2
1
1112
1
)221(2
x
11
2
1
1111
1 21
)()1(
r