Frequency Domain Techniques i n Forecasting with ARIMA Model

Frequency Domain Techniques in

Forecasting with ARIMA Model

Dr. M. Jahanur RahmanAssociate Professor

Department of StatisticsUniversity of Rajshahi, Rajshahi-6205, Bangladesh

[email protected]

July 28, 2012

Dr. M. Jahanur Rahman, R.U.

Contents

Time Series?

Time domain forecasting with ARIMA model

Transformations from time domain to

frequency domain

Frequency domain techniques in forecasting

with ARIMA model

2

Dr. M. Jahanur Rahman, R.U. 3

Time Series

A sequence of observations indexed by time, denoted

by or

Monthly relative humidity time series of Rajshahi


𝑦 𝑡+2

𝑦 𝑡+1𝑦 𝑡

𝑦 𝑡−1

𝑦 𝑡−2

= Realization

orSample pathTime Series?

A finite realization of a time series process

Let be a sequence of random variable indexed by time t.

Random Process to Time Series


Note that

Time series is only one of the many possible

realizations (sample paths) that the history might

have generated.

Time series analysis is trying to draw statistical

inference from a single outcome (time series),

even partially observed.


How can we make sensible

inferences on the underlying

time series process with a single

observation?We need a strong assumption:

Stationarity (or ergodicity). This assumption treat the time series as a random sample

from the same underlying population (process).


A time series is weak stationary if its mean, variance, and autocovariance (at various lags) remain the same no matter at what point of time we measure them. That is, they are time invariant.

Stationary Time Series

Mea

n =

Varia

nce

= Co

varia

nce

=

Mea

n =

Varia

nce

= Co

varia

nce

=

Mea

n =

Varia

nce

= Co

varia

nce

=

Mea

n =

Varia

nce

= Co

varia

nce

=

Mea

n =

Varia

nce

= Co

varia

nce

=


Ergodicity

a realization(a sample path)

Mea

n =

Varia

nce

= Co

varia

nce

=

Mea

n =

Varia

nce

= Co

varia

nce

=

Mea

n =

Varia

nce

= Co

varia

nce

=

Mea

n =

Varia

nce

= Co

varia

nce

=

Mea

n =

Varia

nce

= Co

varia

nce

=

Dr. M. Jahanur Rahman, R.U.

Mea

n =

Varia

nce

= Co

varia

nce

=

Mea

n =

Varia

nce

= Co

varia

nce

=

Mea

n =

Varia

nce

= Co

varia

nce

=

Mea

n =

Varia

nce

= Co

varia

nce

=

Mea

n =

Varia

nce

= Co

varia

nce

=

Covariance = Variance =

Mean =

The covariance-stationary process is ergodic for the mean.

The covariance-stationary process is ergodic for the variance.

The covariance-stationary process is ergodic for the covariance.9

Ergodicity


Non-stationary time series Non-stationary time series

-3

-2

-1

0

1

2

3

10 20 30 40 50 60 70 80 90 00

Stationary time series

(Non)Stationay Time Series


Non-stationary time series contain two kinds of trends: Deterministic trend Stochastic (random) trend

Non-stationary Time Series


Trend Stationary: If y(t) contains a deterministic trend and

{y(t) – trend}

becomes stationary. Then y(t) is known as trend stationary.

Difference Stationary: If y(t) contains a stochastic trend and

{y(t) – y(t-1)}

becomes stationary. Then y(t) is known as difference stationary.



Any time series can contain some or all of the following

components:

1. Trend (T)

2. Cyclical (C)

3. Seasonal (S)

4. Irregular (I)

Trend component

The trend is the long term pattern of a time series. A trend can be

positive or negative depending on whether the time series exhibits an

increasing long term pattern or a decreasing long term pattern.

Components of a Time Series


Cyclical component

Any pattern showing an up and down movement around a given trend

is identified as a cyclical pattern. The duration of a cycle depends on

the type of business or industry being analyzed.


components:

1. Trend (T)

2. Cyclical (C)

3. Seasonal (S)

4. Irregular (I)


Seasonal component

Seasonality occurs when the time series exhibits regular fluctuations

during the same month (or months) every year, or during the same

quarter every year. For instance, retail sales peak during the month of

December.


components:

1. Trend (T)

2. Cyclical (C)

3. Seasonal (S)

4. Irregular (I)




Irregular component

This component is unpredictable. Every time series has some

unpredictable component that makes it a random variable. In prediction,

the objective is to “model" all the components to the point that the only

component that remains unexplained is the random component.


components:

1. Trend (T)

2. Cyclical (C)

3. Seasonal (S)

4. Irregular (I)



Two DomainsTraditionally, there are two ways to analyze time series data:

Time domain analysis

Time domain analysis examines how a time series process evolves

through time, with tools such as autocorrelation function.

Frequency domain analysis

Frequency domain analysis, also known as spectral analysis,

studies how periodic components at different frequencies

describe the evolution of a time series.


Time

Time domainFrequency

Frequency domain

Two Domains


The Autoregressive Integrated Moving Average (ARIMA)

models, or Box-Jenkins models, are a class of linear models

that is capable of representing stationary as well as non-

stationary time series.

Time Series Model


ARIMA(p,d,q) Model

∆𝑑𝑌 𝑡=𝑐+𝜙1∆𝑑𝑌 𝑡− 1+𝜙2∆

𝑑𝑌 𝑡 −2+⋯𝜙𝑝 ∆𝑑𝑌 𝑡−𝑝+𝜀𝑡+𝜃1𝜀𝑡 −1+𝜃2𝜀𝑡− 2+⋯+𝜃𝑞𝜀𝑡 −𝑞

= Response (dependent) variable at time

= Respose variable at time lags

= difference operator ()

= White noise error term

= Errors in previous time periods

= parameters

= Number of autoregressive terms

= Number of moving average terms.


ARIMA(p,0,q) = ARMA(p, q) Model

ARIMA(p,0,0) = AR(p) Model

ARIMA(0,0,q) = MA(q) Model


ARIMA models rely heavily on both autocorrelation (AC) and

partial autocorrelation (PAC) patterns in time series data.


ARIMA(2,0,0) = AR(2)

AC PA

C

1 2 3 4 5 6 7 …….. Lags 1 2 3 4 5 6 7 …….. Lags

ARIMA(0,0,2) = MA(2)1 2 3 4 5 6 7 …….. Lags 1 2 3 4 5 6 7 …….. Lags

AC

PAC

ARIMA(2,0,2) = ARMA(2, 2)

AC

PAC

1 2 3 4 5 6 7 …….. Lags1 2 3 4 5 6 7 …….. Lags

Theoretical pattern of ACF and PACFACF 0PACF = 0 for lag > 2

ACF = 0 for lag > 2; PACF 0

ACF PACF 0


Choosing an ARIMA model

ARIMA modeling requires a great deal of skill, which of course come from practice.

Model Pattern of ACF Pattern of PACFAR(p) Spikes decays exponentially. Significance spikes through lags p

MA(q) Significance spikes through lags q Spikes decays exponentially

ARMA(p,q) Spikes decays exponentially Spikes decays exponentially

Theoretical pattern of ACF and PACF


1. Identification of the model

(choosing tentative p, d, p for ARIMA)

2. Parameter estimation of the chosen model.

4. Forecasting

3. Diagnostic checking

(are the estimated residual white noise?)

yes

no

Box-Jenkins (BJ) Methodology


Steps of modeling a observed time series

with ARIMA


Original time series [

Log transformation

Dr. M. Jahanur Rahman, R.U. 30Regular differencing

Seasonal differencing


Correlogram: Shows SAC & SPAC at different lags


ARIMA(2,0,0) = AR(2)

AC PA

C

1 2 3 4 5 6 7 …….. Lags

1 2 3 4 5 6 7 ……..

ARIMA(0,0,2) = MA(2)1 2 3 4 5 6 7 …….. Lags

1 2 3 4 5 6 7 ……..

AC

PAC

ARIMA(2,0,2) = ARMA(2, 2)

AC

PAC

1 2 3 4 5 6 7 ……1 2 3 4 5 6 7 …….. Lags

Competitive Models:

(1) ARMA(2, 0) (2) ARMA(0, 1) (3) ARMA(2, 1)

Choosing an ARIMA model


Model MSE

ARMA(2, 0) 3521

ARMA(0, 1) 4019

ARMA(2, 1) 3998

Looking for the best Model


Finally Forecasting

Frequency Domain Fourier Transformation Short-time Fourier Transformation Wavelet Transformation


Frequency Domain

To get further information from the time series that is not

readily available in the time domain.

Frequency Domain

Time Domain


Autocorrelation function or correlogram is used for analyzing time series in time domain.

rk

k

Correlogram

t

Yt

Periodic process with noise

Time Domain

Periodicities in data can be best determined by analyzing the time series in frequency domain.

Time series


Frequency Domain

Frequency Domain

Time Domain

Transformations:

Fourier Transformation (FT)

Short-time Fourier Transformation (SFT)

Wavelet Transformation (WT) ………


Time series analysis in frequency domain is known as

frequency domain analysis or spectral analysis.


The Basics

= Amplitude (height of the function), R > 0

= Frequency (cycle/unit time)

= Phase (the starting point of the cosine function

which lies in )

(period).

𝑌 𝑡=𝑅𝑐𝑜𝑠 (2𝜋 𝑓𝑡+𝜙 )



t = running from 1 to 100. Red curve has , = 4/100 = 1/25 and Blue curve has , = 4/100 = 1/25 and Black curve has , = 4/100 = 1/25 and

The Basics


as


The Basics


𝑦 2𝑡=2 cos (2𝜋 5 𝑓𝑡 )+3sin (2𝜋 5 𝑓𝑡 )

𝑦 1𝑡=2cos (2𝜋 2 𝑓𝑡 )+3sin (2𝜋 2 𝑓𝑡 )

𝑦 𝑡=𝑦1 𝑡+𝑦2 𝑡

Jean Baptiste Joseph Fourier(1768-1830)

The Basics


Mat

hem

atic

al C

once

pts


𝑓 (𝑥 )= ⟨ 𝑓 (𝑥) ,1 ⟩⟨1,1 ⟩

1+∑𝑛=1

∞ ⟨ 𝑓 (𝑥) ,cos (𝑛𝑥) ⟩⟨cos (𝑛𝑥 ) ,cos (𝑛𝑥) ⟩

cos (𝑛𝑥)+∑𝑛=1

∞ ⟨ 𝑓 (𝑥) , sin (𝑛𝑥 )⟩⟨sin (𝑛𝑥 ), sin (𝑛𝑥 )⟩

sin (𝑛𝑥 )

So 𝑎0=1

2𝜋 ∫−𝜋

𝜋

𝑓 (𝑥 )𝑑𝑥

𝑎𝑛=1𝜋 ∫

−𝜋

𝜋

𝑓 (𝑥 )cos (𝑛𝑥 )𝑑𝑥

𝑏𝑛=1𝜋 ∫

−𝜋

𝜋

𝑓 (𝑥 )𝑠𝑖𝑛 (𝑛𝑥 )𝑑𝑥 𝑛=1,2,3 ,……

𝑓 (𝑥 )=𝑎0+∑𝑛=1

∞

𝑎𝑛cos (𝑛𝑥 )+∑𝑛=1

∞

𝑏𝑛sin (𝑛𝑥)

According to Fourier, any function in can be defined as

Fourier Series


or equivalently,

Where

for

Fourier Series


Time (t)

y(t)

Frequency (f)

Y(f)

Time domain to Frequency Domain

Time Domain Frequency Domain


𝑦 2𝑡=2 cos (2𝜋 5 𝑓𝑡 )+3sin (2𝜋 5 𝑓𝑡 )

𝑦 1𝑡=2cos (2𝜋 2 𝑓𝑡 )+3sin (2𝜋 2 𝑓𝑡 )

𝑦 𝑡=𝑦1 𝑡+𝑦2 𝑡

2f 5f

5f

2f

Time Domain Frequency Domain

Time domain to Frequency Domain


Discrete Fourier Transformation (DFT)Let a time sequence

and a frequency sequence

The DFT of is

Where

Note that and are of the same size.

The DFT of a time series is another time series.


The inverse DFT (IDET) of , is given by

Inverse Discrete Fourier Transformation (IDFT)

DFT

IDFT

3f 40f6f

Fourier Transformation

Inverse Fourier Transformation.


Spectral Density

Spectral density () is the amount of variance per interval of frequency: Angular frequency:

The plot of vs is called spectrum and as a whole called spectral density.


Spectral Density

𝐼 (𝑘 )

𝜔𝑘

Spectral density transforms information from time domain to frequency domain

Spectral density


Total area under the spectrum is equal to the variance of the process.

All information in frequency domain is extracted from the spectral density (or spectrum).

Spectral density () is the amount of variance per interval of frequency.A peak in the spectrum indicates an important contribution to

the variance at frequencies close the peak.Prominent spikes indicate periodicity. Several expressions for spectrum exist in literature.

Spectral Density


Spectral Density For a complete random series the spectral density function is

constant – termed as white noise.White noise indicates that no frequency interval contains

more variance than any other frequency intervals.

𝐼 (𝑘 )

𝜔𝑘Spectral density of a White noise


3f 40f6f

DFT

Time Series Periodogram

IDFT

Filtered Series

Remove high

frequency component

ReconstructionDecomposition

Application of Fourier Transformation


0 0.2 0.4 0.6 0.8 1-3

-2

-1

0

1

2

3

0 5 10 15 20 250

100

200

300

400

500

600

All frequency components exist at all times

Stationary Time Series

Fourier Transformation


Frequency

Am

plitu

de

Fourier TransformationNon-stationary Series


0 0.5 1-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

0 5 10 15 20 250

50

100

150

Time Frequency0 0.5 1

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

0 5 10 15 20 250

50

100

150

Time Frequency

Different in Time Domain

Same in Frequency Domain

FT only gives frequency components exist in the seriesThe time and frequency Information can not be seen at the same

time which is required.

Fourier TransformationNon-stationary Series


Short-Time Fourier TransformationDennis Gabor (1946) In STFT, the signal is divided into small

enough segments, where these segments of the series can be assumed to be stationary.

For this purpose, a fixed window function "w" is chosen


FT X

Short-Time Fourier Transformation


FT X



FT X

Advantage & Disadvantage



FT X



Time stepFrequency

Am

plitu

deShort-Time Fourier Transformation


Time

Freq

uenc

y

Better time resolution; Poor frequency resolution

Better frequency resolution; Poor time resolution

Narrow windows give good time resolution, but poor frequency resolution.

Wide windows give good frequency resolution, but poor time resolution.




Wide windows do not provide good localization at high frequencies




Use narrower windows at high frequencies




Narrow windows do not provide good localization at low frequencies




Use wider windows at low frequencies




= Small value

(Wavelet coefficient)

.54321

TimeSc

ale

Scale = 1, time = 0

Changing Window at Multi-resolution Analysis


= Small value


.54321

TimeSc

ale

Scale = 1, time = 100



= Moderate value


.54321

TimeSc

ale




= Large value


.54321

TimeSc

ale




Mag

nitu

de

20 Hz 50 Hz 120 Hz



The Discrete Wavelet Transform (DWT)

𝐷𝑊𝑇 (𝑚 ,𝑛 )=∑𝑡𝑌 (𝑡 )Ψ (𝑚 ,𝑛 )

= wavelet function = mother wavelet m = is location parameter n = scale parameter = time series


Wavelet families


Multi-level Decomposition of a Series with Wavelets

Time Series

cA3 cD3

cA2 cD2

cA0 cD0The decomposition tree can be schematically described as:


0 0 0 0 0 00 0 0 0 0 00 0 0 0 0 00 0 0 0 0 0 - 0 0 0 0 0 00 0 - 0 0 0 00 0 0 0 - 0 00 0 0 0 0 0 -

0 0 0 00 0 0 0

- - 0 0 0 00 0 0 - -

- - - -

Basis Constriction


Example



Time Series1, 3, 5, 11, 12, 13, 0, 1

cA32.8, 11.3, 17.6, 0.7

cD3-1.4, -4.2, -0.7, -0.7

0 0 0 0 0 00 0 0 0 0 00 0 0 0 0 00 0 0 0 0 0 - 0 0 0 0 0 00 0 - 0 0 0 00 0 0 0 - 0 00 0 0 0 0 0 -

Level-1 orScale-1



Time Series1, 3, 5, 11, 12, 13, 0, 1

cA32.8, 11.3, 17.6, 0.7

cD3-1.4, -4.2, -0.7, -0.7

Level-1 orScale-1

Wavelet coefficients = {2.8, 11.3, 17.6, 0.7, -1.4, -4.2, -0.7, -0.7}



Time Series1, 3, 5, 11, 12, 13, 0, 1

cA32.8, 11.3, 17.6, 0.7

cD3-1.4, -4.2, -0.7, -0.7

cA210.0, 13.0

cD2-6.0, 12.0

Level-1 orScale-1

Level-2 orScale-2

0 0 0 00 0 0 0

- - 0 0 0 00 0 0 - -

Wavelet coefficients = {10.0, 13.0, -6.0, 12.0, -1.4, -4.2, -0.7, -0.7}



Time Series1, 3, 5, 11, 12, 13, 0, 1

cA32.8, 11.3, 17.6, 0.7

cD3-1.4, -4.2, -0.7, -0.7

cA210.0, 13.0

cD2-6.0, 12.0

cA016.26

cD0-2.12

Level-1 orScale-1

Level-2 orScale-2

Level-3 orScale-3

- - - -

Wavelet coefficients = {16.2, -2.1, -6.0, 12.0, -1.4, -4.2, -0.7, -0.7}


DWT Analysis

IDWT Reconstruction

Discrete Wavelet Transform (IDWT).

The input can be reconstructed using the Inverse

Frequency Domain Application-1 Fourier Transformation

Spectral density

Determining periods

Removing Periodicities

Modeling and Forecasting


Autocorrelation function or correlogram is used for analyzing time series in time domain.

Time series rk

k

Correlogram

t

Yt

Periodic process with noise

Why Frequency Domain?


The steps of analyzing the data

Plot the time series

Plot the correlogram (to suspect the periodicity)

Plot the spectrum (to identify the periodicities)

Ik

wk

rk

k

Yk

t


The spectrum shows prominent spikes which represents the periodicities inherent in the data.

The period corresponding the any value of may be computed by

While the correlogram indicate the presence of periodicities in the data, the spectral analysis helps identify the significant periodicities themselves

Periodicities


Ik

wk

Ik

wk

Ik

wk

Ik

wk

Reconstruct new series by

removing the significance

periodicity one after another as



The periodicities are tested for signoficance by defining the

following statistics (Kashyap and Rao 1976)

Where

Statistical significance of periodicities



Significant periodicities are removed from the original time series to get a new series , where

The series containing the previous periodicities

where d is no. of periodicities removed (which are known to be significant)

The spectrum of new series is plotted and the spikes are observed until the series become random.


Example A monthly time series over the period 1987-2011 is used. Objective: Modeling and forecasting

Time Series ACF

Periodogram PACF


Spectral density shows tree significant periodic patterns with frequencies at

with corresponding periods 30 months 12 months 06 months respectively.

Correlogram & SpectrumCorrelogram indicates the presence

of periodicity.

ACF

Periodogram


One-period removed series

Two-period removed series

Three-period removed series

Correlogram & Spectrum


Model Selection

Seasonally differenced seriesARIMA(2,0,0) = AR(2)

Periodicities removed seriesARIMA(2,0,0) = AR(2)

PAC

PACAC

AC


Trail data: January, 1964 – December, 2007Test data: January, 2008 – December, 2008

Forecasting


Comparison of Forecasting PerformanceMean Absolute Error (MAE)Root Mean Squared Error (RMSE)Mean Absolute Percentage Error (MAPE)

Lower-value implies best forecast


seasonally differenced series Parameters are significant R-square is high DW-statistic is near 2

Periodicities removed series Parameters are significant R-square is moderate DW-statistic is about 2

EstimationAR(2) model


Forecasting

RMSE MAE PR series 1.173 4.739D series 0.852 3.914


Decomposition

Discrete Wavelet Transformation

Modeling & Forecasting

Humidity & Maximum Temperature of Rajshahi

Data obtained from BARC

Frequency Domain Application-2


Monthly Humidity of Rajshahi(1964-2008)


Humidity y(t)

A1

A2

A3

A4

D1

D2

D3

D4

Y = A1 + D1

Y = A2 + D2 +D1

Y = A3 + D3 + D2 +D1

Y = A4 + D4 + D3 + D2 +D1

Decom

positi

on by

wavele

t

Trans

form

ation

Monthly humidity of Rajshahi over 1964-2008


DecompositionLevel =3 Wavelet : Doubasis-5



Input series

Wavelet Transformatio

nForecasting

Inverse Wavelet

Transformation

Output series

ForecastingInput series

Wavelet Transformatio

n Processing

Inverse Wavelet

Transformation

Output series

Method-1

Method-2

Forecasting Framework


Forecasting Framework

Proposed by: Rocha, T. et.al. (2010)


Method-1Step-1: The wavelet transformation of type Daubechies-5 and decomposition level 3 is applied to the series ( result in 4 series denoted by and .

Step-2: Use a specific ARIMA model of each one of the constitutive series to forcast its future values, that is,

Step-3: The inverse wavelet transform is used to reconstruct the forecast series as


Method-2Step-1: The wavelet transformation of type Daubechies-5 and decomposition level 3 is applied to the series ( result in 4 series denoted by and .

Step-2: Reconstruct the series by removing the high frequency component

Step-3: Use the appropriate ARIMA model to the reconstructed series to forecast the future values;


Trail data: January, 1964 – December, 2007

Test data: January, 2008 – December, 2008

Forecasting of Humidity


Comparison of Forecasting PerformanceMean Absolute Error (MAE)Root Mean Squared Error (RMSE)Mean Absolute Percentage Error (MAPE)

Lower-value implies best forecast


Method MAE RMSE MAPE

Time domain Method 0.1633 0.2187 18.566

Frequency Domain Method-1

0.1336 0.1591 15.3654


0.0654 0.0799 9.1472

Forecasting performance over Jan-2008 to Dec-2008

Forecasting of HumidityModels

Time Domain: ARIMA(0,1,1)(0,1,1)

Frequency Domain-1: ARIMA(8,2,8)(0,0,1), ARIMA(0,1,2)(0,1,1),

ARIMA(0,0,4)(0,1,0), ARIMA(2,0,2)(0,0,0)

Frequency Domain-2: ARIMA(2,1,1)(0,1,1)


Frequency Domain Method-2 Time Domain Method




0.16835 0.2296 19.001


0.0654 0.0799 9.1472

Forecasting of Humidity


Monthly Maximum Temperature of Rajshahi(1964-2009)

Trail data: Jan, 1964 – Dec, 2008

Test data: Jan, 2009 – Dec, 2009


Forecasting of Maximum Temperature






0.6995 0.8286 2.1402

Models

Time Domain: ARIMA(0,0,1)(0,1,1)

Frequency Domain-1: ARIMA(1,2,8)(0,0,1), ARIMA(4,1,4)(0,1,1),

ARIMA(0,0,6)(0,1,1), ARIMA(1,0,4)(0,1,1)

Frequency Domain-2: ARIMA(0,0,1)(0,1,1)


Forecasting of Maximum Temperature





0.6995 0.8286 2.1402


In modeling time series both time domain and

frequency domain analysis should be done

together.

For high frequency and noisy data, frequency

domain analysis performs better.

Recommendation


Than

ks

Frequency Domain Techniques i n Forecasting with ARIMA Model

Documents

Transcript of Frequency Domain Techniques i n Forecasting with ARIMA Model