Functional Linear Models - cuni.czmsekce.karlin.mff.cuni.cz/.../1314z/1314z_Petrasek.pdf · 2015....

Functional LinearModels

Jakub Petrasek

Introduction

Linear models

Motivation

Functional linearmodel

Fully FunctionalModel

Functionalprincipalcomponents

Estimator basedon FPC

Scalar ResponseModel

Estimator basedon roughnesspenalty

Asymptoticproperties

ModelDiagnostics

Bibliography

Functional Linear Models

Jakub Petrasek

Department of Probability and Mathematical StatisticsFaculty of Mathematics and Physics, Charles University in Prague

[email protected]

Seminar in Stochastic Modelling in Economics and Finance

December 2, 2013


Jakub Petrasek

Introduction

Linear models

Motivation








ModelDiagnostics

Bibliography

Outline

1 IntroductionLinear modelsMotivationFunctional linear model

2 Fully Functional ModelFunctional principal componentsEstimator based on FPC

3 Scalar Response ModelEstimator based on roughness penaltyAsymptotic properties

4 Model Diagnostics


Jakub Petrasek

Introduction

Linear models

Motivation








ModelDiagnostics

Bibliography

Outline




4 Model Diagnostics

1


Jakub Petrasek

Introduction

Linear models

Motivation








ModelDiagnostics

Bibliography

Standard linear model

Model

Y = Xβ + ε,

where

Y ... N x 1 vector,

X ... N x p matrix,

β ... p x 1 patameter vector,

ε ... N x 1 vector of zero mean errors independent of X.

OLS estimatorThe least squares estimator solves normal equations

XTXβ = XTY

For XTX regular, we can explicitly define the OLS estimator

β =(

XTX)−1

XTY

2


Jakub Petrasek

Introduction

Linear models

Motivation








ModelDiagnostics

Bibliography

Example

We observe asset price dynamics continuously. The aim is to exploitmarket inefficiencies using prices. A linear model can be of the form

Ptik+τ − Ptik

=

p(∆)∑j=1

∆Ptik−jβj + εi , 1 ≤ k ≤ N.

●

●

●●

●

●

●

●

●●

●●

●

●

●●

●

●

●

●

●●

●

●

●

●

●●●●●

●●

●

●●

●

●

●●●●

●

●

●●

●●

●

●

●

●

●

●

●●

●

●

●●

●

●

●●

●

●

●

●

●

●

●●

●

●●

●

●●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

0 2 4 6 8 10

−0.

010

−0.

005

0.00

00.

005

0.01

00.

015

0.02

0

Linear model (1 minute price prediction based on 5 seconds shots)

Lag in minutes

β

Figure: Estimated linear parameters{βj

}(120 parameters).

3


Jakub Petrasek

Introduction

Linear models

Motivation








ModelDiagnostics

Bibliography

Example

We observe asset price dynamics continuously. The aim is to exploitmarket inefficiencies, i.e. find a prediction model.

Ptik+τ − Ptik

=

p(∆)∑j=1

∆Ptik−jβj + εi , 1 ≤ k ≤ N.

●

●

●●

●

●

●

●

●●

●●

●

●

●●

●

●

●

●

●●

●

●

●

●

●●●●●

●●

●

●●

●

●

●●●●

●

●

●●

●●

●

●

●

●

●

●

●●

●

●

●●

●

●

●●

●

●

●

●

●

●

●●

●

●●

●

●●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

0 2 4 6 8 10

−0.

010

−0.

005

0.00

00.

005

0.01

00.

015

0.02

0

Linear model + Functional Model (1 minute price prediction based on 5 seconds shots)

Lag in minutes

β

Figure: Estimated linear parameters{βj

}(120 parameters). The blue line is

estimated function β(t) of appropriate functional linear model.

4


Jakub Petrasek

Introduction

Linear models

Motivation








ModelDiagnostics

Bibliography

Notes to example

Linear modelGrid is chosen arbitrarily.

The more regressors you use the less robust is the model.

For p ≥ N you achieve a perfect fit, which is typically a noisetype function (you have no insight into the true parameters).

The main issue is that the parameter in the model is infinitedimensional objects, but the sample is finite dimensional.

We have to smooth!

Idea in functional modelRestrict to some suitable finite dimensional space.

1 Use principal components of the data.

2 Approximate the data by a rich set of functions and impose aroughness penalty.

5


Jakub Petrasek

Introduction

Linear models

Motivation








ModelDiagnostics

Bibliography

Functional versions

Fully functional model

Yi (t) =

∫β(t, s)Xi (s)ds + εi (t).

The estimator is usually based on functional principalcomponents expansion.

Consistency of the estimator can be proved.

Scalar response model

yi =

∫β(s)Xi (s)ds + εi .

Some estimate methods use roughness penalty.Asymptotic properties are known.

Functional response model

Yi (t) = β(t)xids + εi (t).6


Jakub Petrasek

Introduction

Linear models

Motivation








ModelDiagnostics

Bibliography

Outline




4 Model Diagnostics

7


Jakub Petrasek

Introduction

Linear models

Motivation








ModelDiagnostics

Bibliography

Notation

Fully functional model

Yi (t) =

∫Xi (s)β(s, t)ds + εi (t).

Model assumptions are analogical to those of a classical linear model

Yi and Xi are observed functions

εi are zero mean functions independent of Xi

We assume that

Xi can be approximated by a set of functions {ηk}Kk=1

Yi can be approximated by a set of functions {θl}Ll=1.

IdeaConsider the estimator in the form

β∗(s, t) =K∑

k=1

L∑l=1

bklηk(s)θl(t).

We reduced the dimension to ”only” K × L number of parameters,{bkl}K ,Lk,l=1.

8


Jakub Petrasek

Introduction

Linear models

Motivation








ModelDiagnostics

Bibliography

Normal equations

In vector form we can write

β∗(s, t) = η(s)TBθ(t)

Y(t) =

∫X(s)β(s, t)ds + ε(t) (N × 1)

Y(t) =

∫X(s)η(s)TBθ(t)ds + ε(t)

Y(t) = ZBθ(t) + ε(t).

We multiply the equation by θ(t)T and integrate with respect to t∫Y(t)θT (t)dt =

∫ZBθ(t)θT (t)dt +

∫ε(t)θ(t)Tdt (N × L).

Finally we multiply by ZT , use the fact that error terms are independent ofX and obtain K × L normal equations

ZTZBJ = ZT

∫Y(t)θT (t)dt (K × L).

The procedures is unnecessary complicated, we can use more suitable

approximation!9


Jakub Petrasek

Introduction

Linear models

Motivation








ModelDiagnostics

Bibliography

Reminder of FPC

MotivationDecrease the dimension of objects for better manipulation. We aremoving from infinite dimension to p-dimensional with p typicallysingle digit.

Mathematical formulation

Assume we observe x1, . . . , xN zero mean functions (we plan todescribe variance not location). Fix p < N, we search fororthonormal basis u1, . . . , up which minimize

S2 =N∑i=1

∥∥∥∥∥xi −p∑

k=1

〈xi , uk〉 uk

∥∥∥∥∥2

.

10


Jakub Petrasek

Introduction

Linear models

Motivation








ModelDiagnostics

Bibliography

Reminder of FPC II

Sample covariance operator

Definition

C (x) = N−1N∑i=1

〈xi , x〉 xi .

Operator is symmetric and positive-definite and admitsdecomposition

C (x) =∞∑k=1

λk 〈x , vk〉 vk ,

where{λk

}are eigenvalues and {vk} are eigenfunctions.

Proposition

The functions u1, . . . , up are equal (up to a sign) to the first peigenfunctions of the sample covariance operator.

11


Jakub Petrasek

Introduction

Linear models

Motivation








ModelDiagnostics

Bibliography

Theoretical estimator

Functional principal compomentsAssume that random functions X and Y satisfy expansion

X (s) =∞∑i=1

ξivi (s), Y (t) =∞∑j=1

ζjuj(t),

where {vi} and {uj} are functional principal compoments (FPCs).

Lemma

X ,Y , ε ∈ L2 are zero mean, X and ε independent and

Y (t) =

∫ψ(t, s)X (s)ds + ε(t),

∫ ∫ψ(t, s)2 <∞

ψ(t, s) =∞∑k=1

∞∑l=1

E [ξkζl ]

E ξ2k

uk(t)vl(s), in L2([0, 1]2).

12


Jakub Petrasek

Introduction

Linear models

Motivation








ModelDiagnostics

Bibliography

Proof of Lemmaui (t)vj(s), 1 ≤ s, t ≤ 1 form an orthonormal basis in L2([0, 1]2).

ψ(t, s) =∞∑k=1

∞∑l=1

ψkluk(t)vl(s)

and∞∑k=1

∞∑l=1

ψ2kl =

∫ ∫ψ2(t, s)dtds <∞.

We can derive ψkl by the following operations

Y (t) =

∫ψ(t, s)X (s)ds + ε(t)

∞∑j=1

ζjuj(t) =

∫ ∞∑i=1

ξivi (s)∞∑k=1

∞∑l=1

ψkluk(t)vl(s)ds + ε(t)

∞∑j=1

ζjuj(t) =∞∑i=1

ξi

∞∑k=1

ψkiuk(t) + ε(t)

ζl =∞∑i=1

ψliξi + 〈ul , ε(t)〉 , for any l

E [ζlξk ] = ψlkE [ξ2k ], for any l , k

13


Jakub Petrasek

Introduction

Linear models

Motivation








ModelDiagnostics

Bibliography

Estimator based on EFPC

E [ξ2k ]

Note the relation between scores and eigenvalues of the covarianceoperator E [ξ2

k ] = λk .

λk = 〈C(vk), vk〉 = 〈E [〈X , vk〉X ] , vk〉 = E 〈X , vk〉2 = E [ξ2k ]

Thus we can estimate E [ξ2k ] by λk .

E [ζlξk ]

Asume that data are approximated by finite sums Xi (s) =∑K

j=1 ξij vj(s)

and Yi (t) =∑L

j=1 ζij uj(t). Proposed estimator for E [ζlξk ] is

σlk =1

N

N∑i=1

〈Xi , vl(s)〉〈Yi , uk(s)〉 .

14


Jakub Petrasek

Introduction

Linear models

Motivation








ModelDiagnostics

Bibliography

Estimator based on EFPC II

ψ(t, s)

If we put the partial estimators together we can define the estimatorfor ψ(t, s)

ψKL(t, s) =K∑

k=1

L∑l=1

λ−1l σlk uk(t)vl(s)

Consistency

We assume that the numbers of principal components K a L arefunctions of the sample size N, then under regularity conditions(details in the reference in the book [3], page. 138)∫ ∫ [

ψKL(t, s)− ψ(t, s)]2

dtdsP→ 0, N(K , L)→∞.

15


Jakub Petrasek

Introduction

Linear models

Motivation








ModelDiagnostics

Bibliography

Outline




4 Model Diagnostics

16


Jakub Petrasek

Introduction

Linear models

Motivation








ModelDiagnostics

Bibliography

Notation

ModelRegressors are curves but the responses are scalars

Yi =

∫Xi (s)g(s)ds + εi , i = 1, . . . ,N.

EstimationWe can apply a simplified version of the described estimationprocedure. Or, we can use method based on regularization by aroughness penalty.The idea is to use rich set of base functions but control thesmoothness, i.e. searching for g which minimize

sum of squared errors + roughness penalty.

17


Jakub Petrasek

Introduction

Linear models

Motivation








ModelDiagnostics

Bibliography

Formally

Set of base functionsWe search for

gK (s) =K∑

k=1

bkφk(s),

where φk , k = 1, 2, ...,K is Fourier basis, K is assumed to be large.We denote g (m) m-th derivative of g .

Minimization problem

N∑i=1

[Yi −

∫Xi (s)gK (s)ds

]2

+λ

∫g

(m)K (s)2ds.

Using orthogonality of Fourier basis we can rewrite the above formulaas

N∑i=1

[Yi −

K∑k=1

〈Xi , φk〉 bk

]2

+ λK∑

k=1

b2k

∥∥∥φ(m)k

∥∥∥2

.

18


Jakub Petrasek

Introduction

Linear models

Motivation








ModelDiagnostics

Bibliography

Smoothing parameter

The estimator gK ,λ depends both on the number of basis function K andon the roughness parameter λ

gK ,λ(s) =K∑

k=1

bkφk(s)

Optimal λ

Let g (−i) be the estimated function without using the i-th observation.

CV (λ) =N∑i=1

[Yi −

∫Xi (s)g (−i)ds

]2

.

Optimal smoothing parameter λ minimize CV (λ) (cross-validation).

Asymptotic propertiesIn order to derive some asymptotic properties we need to restrict the set ofpossible solutions g to a subset of L2 of sufficiently smooth and periodicfunctions.

19


Jakub Petrasek

Introduction

Linear models

Motivation








ModelDiagnostics

Bibliography

Sobolev space

Definition (Sobolev space)

Sobolev space Wm,p([0, 1]) is a space of functions u ∈ Lp([0, 1]) whoseweak derivatives up to order m also belong to Lp([0, 1]). m is called theorder of Sobolev space.

Note

Sobolev space uses the notion of weak derivative. v ∈ LP([0, 1]) is called aweak derivative of u ∈ LP([0, 1]) if∫

u(t)φ(t)′dt = −∫

v(t)φ(t)dt

for any infinitely differentiable functions φ with φ(0) = φ(1) = 0.

For p = 2 it is a Hilbert space, usually denoted by Hm([0, 1]) with innerproduct

〈u, v〉 =

∫ (u(t)v(t) +

m∑ν=1

u(ν)(t)v (ν)(t)

)dt.

20


Jakub Petrasek

Introduction

Linear models

Motivation








ModelDiagnostics

Bibliography

Asymptotic property

Hmper([0, 1])

We define a set Hmper ([0, 1]) a subset of Hm([0, 1]) with additional

condition that for every 1 ≤ ν ≤ m − 1 u(ν)(0) = u(ν)(1).

Asymptotic property

If g ∈ Hmper ([0, 1]) for m ≥ 2 then

E ‖g∞,λ − g‖2 = OP

(N−1/2 + λ+ N−1λ−1/2m

)

21


Jakub Petrasek

Introduction

Linear models

Motivation








ModelDiagnostics

Bibliography

Hybrid estimator

We can combine the FPC method with roughness penalty to getFunctional Principal Component Regression with a roughness penalty.The estimation proceeds as follows

1 Define regressor scores Xik = 〈Xi , φk〉.2 Compute K (>> p) multivariate principal components (note that Xi

are real-valued vectors). Let vj , j = 1, . . .K denote the eigenvectors.

3 We are estimating the linear regression model

Yi =K∑

k=1

Xikbk ,

where we further project the parameters bk on the first p principalcomponents

bk =

p∑j=1

βj vjk .

The minimization is of the form

N∑i=1

[Yi −

p∑j=1

βj

K∑k=1

Xjk vjk

]2

+ λ

[p∑

j=1

βj

K∑k=1

vjk

∫φ

(m)k (s)ds

]2

.

22


Jakub Petrasek

Introduction

Linear models

Motivation








ModelDiagnostics

Bibliography

Outline




4 Model Diagnostics

23


Jakub Petrasek

Introduction

Linear models

Motivation








ModelDiagnostics

Bibliography

Overview

The aim is to describe the key diagnostics methods, which includes

Verification of linearity

Independence of residuals and regressors

Stationarity of residuals

Coefficient of determination

Principle

When dealing with linear model we deal with vectors, that are easy toplot (e.g. scatter plots). The main idea is again to move from infinitedimension to finite dimension.Instead of working with raw observations we use estimated scores,that is

Yi (t) =

∫Xi (s)β(s, t)ds + εi (t) −→ ζil =

K∑k=1

ψlkξik + ηl(K ),

24


Jakub Petrasek

Introduction

Linear models

Motivation








ModelDiagnostics

Bibliography

Linearity

Recall thatξik = 〈Xi , vk(s)〉 , ζil = 〈Yi , ul(s)〉 ,

For any j and l the dependence between scores should be linear.

Figure: Scatterplots of response and regressor scores. The left one clearly showsquadratic trend.

25


Jakub Petrasek

Introduction

Linear models

Motivation








ModelDiagnostics

Bibliography

Residuals

Independence of regrossors

We define residuals as

εi = Yi (t)−∫ψ(t, s)X (s)ds, i = 1, . . . ,N.

We can decompose estimated residuals and verify that the scores areindependent of ξij . Moreover the scores should exhibit equal variance.

Functional coefficient of determinationWe can define pointwise the function

R2(t) =var [E [Y (t)|X (t)]]

var [Y (t)]

and a global measure can be defined as

R2 =

∫R2(t)dt.

26


Jakub Petrasek

Introduction

Linear models

Motivation








ModelDiagnostics

Bibliography

Summary

Functional vs classical modelsAvoid linear regression when working with functional data

- Grid is arbitrary- Not robust results- Poor insight into the true dependency

Functional linear models can deal with functional both theresponse and the regressors

Functional models allow smoothness penalties

Estimation procedures

Based on dimension reduction, i.e. instead of (infinitedimension) functions we deal with (finite dimension)coordinates. Main methods

1 Approximate data by empirical functional principal components2 Approximate data by a rich set of basis functions and then

impose a roughness penalty in order to smooth the results3 Functional principal components regression with roughness

penalty27


Jakub Petrasek

Introduction

Linear models

Motivation








ModelDiagnostics

Bibliography

Bibliography

Ramsay J.O. and Silverman B.W.Functional Data Analysis.Springer Series in Statistics, 2005.

Ramsay J.O., Hooker G., and S. Graves.Functional Data Analysis with R and Matlab.Springer Series in Statistics, 2009.

Horvath L. and Kokoszka P.Inference for Functional Data with Applications.Online, 2011.

28


Jakub Petrasek

Introduction

Linear models

Motivation








ModelDiagnostics

Bibliography

...

Thank you for attention

29

Functional Linear Models - cuni.czmsekce.karlin.mff.cuni.cz/.../1314z/1314z_Petrasek.pdf · 2015....

Documents

Transcript of Functional Linear Models - cuni.czmsekce.karlin.mff.cuni.cz/.../1314z/1314z_Petrasek.pdf · 2015....