A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional...

87
A Bayesian Multivariate Functional Dynamic Linear Model David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764 October 30, 2015 Joint work with Daniel R. Kowal, Laura L. Tupper & David Ruppert Support from ABP, NYSTAR, Xerox PARC, and NSF DMS-1455172 David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764 A Bayesian Multivariate Functional Dynamic Linear Model

Transcript of A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional...

Page 1: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

A Bayesian Multivariate FunctionalDynamic Linear Model

David S. Matteson

Cornell University

[email protected]

http://arxiv.org/abs/1411.0764

October 30, 2015

Joint work with Daniel R. Kowal, Laura L. Tupper & David Ruppert

Support from ABP, NYSTAR, Xerox PARC, and NSF DMS-1455172

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic Linear Model

Page 2: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Outline

• Multiple functional time series.• Two examples.

• Yield curves in four economies• Local field potentials measured in rats’ brains

• Bayesian dynamic factor model.• Constrained spline modeling of factor loading curves• Markov switching model for the effects of changes in US yield

curves on the yield curves of the other economies• Stochastic volatility• Time-frequency analysis of the local field potentials.

Page 3: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Data

Assume that we observe multiple time series of functional data:

Y (c)t (τ).

(a) For each c and t, Y (c)t (τ) is a function of τ ∈ T ;

(b) For each c and τ , Y (c)t (τ) is a time series for t = 1, . . . , T ; and

(c) For each t and τ , Y (c)t (τ) is a multivariate observation with

outcomes c = 1, . . . , C .

• T ⊆ Rd is a compact set.• d = 1 and T = [0, 1] are typical.

Page 4: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

ECIS: Functional Time Series

freq

time

z

WI38

5000

10000

15000

20000

25000

freqtime

z

HUTU80

2000400060008000100001200014000

freq

time

z

VA13

200040006000800010000120001400016000

freq

time

z

BSC1

2000400060008000100001200014000

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Functional Time Series Analysis

Page 5: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Example: yield curves

First example:

Y (c)t (τ) is the yield curve (function of maturity τ) in tth week and

cth economy.

• 1 unit of currency invested in week t will grow to

exp(τ/12)Y (c)t (τ) units in τ months.

• Therefore, Y (c)t (τ) is the average yearly interest rate that is

earned on a zero-coupon bond maturing in τ months and

purchased during the tth week.

Page 6: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Multi-economy yield curves

We have yield curves from

1 Federal Reserve (Fed)

2 Bank of England (BOE)

3 European Central Bank (ECB)

4 Bank of Canada (BOC)

So C = 4.

Page 7: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Functional: Y (τ)

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Functional Time Series Analysis

Page 8: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Functional + Time Series: Yt(τ)

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Functional Time Series Analysis

Page 9: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Functional + Time Series + Multivariate: Y (c)t (τ)

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Functional Time Series Analysis

Page 10: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Example: time-frequency analysis

• Suppose we observe C times series.

• Bin the time dimension into T bins.

• Within each bin, and for each time series, compute the

Fourier transform to obtain a function of frequency τ .

• t is the bin index

• Y (c)t (τ) is the smoothed Fourier transform (spectrum) of the

cth series in the tth bin.

Page 11: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Local Field Potential Data

Our example: local field potential (LFP) data collected on rats attwo brain locations.

• A pair of electrodes is placed in each brain region.• In feature binding, a rat’s brain must amalgamate distinct

sensory information into a single neural representation.• There is interest in the synchronization between the regions,

in particular, whether• synchronization is increased during feature binding.

• c = 1 or 2 indicates brain region.• t is the bin index.• τ is frequency.

Page 12: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

LFP Signals from a Single Trial

Page 13: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Time-frequency Plots for FS Trials

Page 14: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

MFDLM

DefineYt(τ) =

Y (1)

t (τ), Y (2)t (τ), . . . , Y (C)

t (τ)

.

The multivariate functional dynamic linear model (MFDLM) is:

Yt(τ) = F(τ)βt + t(τ),t(τ)

Et indep∼ N (0, Et)

βt = Xtθt + νt ,νt

Vt indep∼ N (0, Vt)

θt = Gtθt−1 + ωt ,ωt

Wt indep∼ N (0, Wt)

T F(τ)F(τ) dτ = IKC×KC for identifiability.

We will look more closely at each of the four equations in thismodel.

Page 15: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Factor Model

The first equation is:

Yt(τ) C×1

= F(τ)C×KC

βtKC×1

+t(τ),t(τ)

Et indep∼ N (0, Et)

• The KC columns of F(τ) are factor loading curves, K for eachoutcome.

• βt contains KC factors, K for each outcome.

F(τ) =

F(1)(τ) 1×K

0 . . . 0

0 F(2)(τ) · · · 00 0 . . . 00 0 · · · F(C)(τ)

and βt =

β(1)t

K×1...

β(C)t

Page 16: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Factor Model, continued

From the previous frame:

Yt(τ)

C×1

= F(τ) C×KC

βtKC×1

+t(τ),t(τ)

Et indep∼ N (0, Et)

The covariance matrix Et could be

• constant (independent of t), or

• might follow a stochastic volatility model.

Page 17: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Covariates

The second equation is:

βtKC×1

= XtKC×p

θtp×1

+νt ,νt

Vt indep∼ N (0, Vt)

• Xt is a matrix of known covariates that might affect βt .

• θt contains the p regression coefficients.

• Again, the covariance matrix Vt could be constant.

Page 18: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Dynamic Model for Regression Coefficients

The third equation is what makes the model dynamic and is:

θtp×1

= Gtp×p

θt−1 p×1

+ωt ,ωt

Wt indep∼ N (0, Wt).

Gt is an evolution matrix.

• Gt might be independent of t or could follow its own dynamic

model.

• If Gt ≡ G, then θt is a VAR(1) process.

Page 19: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Orthonormal Factor Loading Curves

The fourth equation is:

TF(τ)F(τ) dτ = IKC×KC for identifiability.

This is a typical constraint used in factor models.

• For example, in PCA.

Page 20: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Modeling the factor loading curves

We model the factor loading curves with splines:

fk(τ) = φ(τ)dk

• fk(τ) is the loading curve for the kth factor and for alloutcomes.

• for simplicity, in our examples we assume the same loadingcurves across all outcomes

• φ(τ) is a vector of spline basis functions.

• dk is a vector of spline coefficients for the kth loading curve.

Page 21: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic
Page 22: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Spline Penalty

We use a penalty

P(dk) = d kΩφdk =

τ∈T

fk(τ)

2dτ (1)

In our Bayesian approach, the penalty is implemented by an

O’Sullivan spline basis (Wand and Ormerod, 2008) .

• The first two basis functions are 1 (constant) and x.

• The remaining basis functions are orthogonal to the first two.

(Nonlinear)

• Penalty (1) is equal to the sum of squared coefficients of the

nonlinear basis functions.

• The prior on dk is N (0, diag(106, 106, λ−1k , . . . , λ−1

k )).

Page 23: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic
Page 24: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Selecting a Penalty Parameter

Three common methods for selecting tuning parameter:

1 CV, GCV, AIC, or Mallow Cp.2 Mixed model/REML (empirical Bayes).3 Fully Bayes.

• There was been considerable research comparing 1. and 2.starting with Wahba (1985).

• See Krivobokov (2013) for a review and recent results.

• The tuning parameter selected by 1. is more variable and morelikely to undersmooth compared to 2. or 3.

• Approach 3. accounts for uncertainty in the smoothingparameter when making inference about other parameters.

Page 25: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Fitting the MFDLM

We used a rather standard approach to a Bayesian analysis:

• Noninformative priors mostly, but

• hierarchical priors for the splines

• Gibbs sampling, with

• Metropolis steps where needed.

Page 26: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Yield Curves

There is an extensive literature on yield curve modeling:

• Nelson-Siegel (1987): three-parameter parametric model

• Svensson (1994): four-parameter extension of Nelson-Siegel

model

• Hays et al. (2012): functional dynamic model—for single

economies

These will be discussed in more detail later.

Page 27: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Yield Curves: Advantages of MFDLM

With the multivariate functional dynamic linear model it is easy to:

• handle multiple economies, especially interactions betweenthem,

• add a hidden Markov model for regime-switching,

• add covariates, e.g., indicators of changes in governmentpolicies,

• allow conditional heteroscedasticity, e.g., with stochasticvolatility models.

Page 28: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Common loading curves

We assume common functional loading curves, f1, . . . , fK , so that

the conditional expectation of the yield curve for economy c,

conditional on past information, is

µ(c)t (τ) ≡

K

k=1β(c)

k,t fk(τ)

Page 29: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Estimates of Common Loading Curves

0 50 100 150 200 250 300 350

−0.3

−0.2

−0.1

0.0

0.1

Common Factor Loading Curves

Maturity (months)

Yiel

d (%

)

k = 1k = 2k = 3k = 4

• k = 1: parallel shifts• k = 1: changes in slope• k = 3: changes in convexity (curvature)

Page 30: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Common Trend Hidden Markov Model

Recall the general form of our model:

Yt(τ) = F(τ)βt + t(τ),t(τ)

Et indep∼ N (0, Et)

βt = Xtθt + νt ,νt

Vt indep∼ N (0, Vt)

θt = Gtθt−1 + ωt ,ωt

Wt indep∼ N (0, Wt)

TF(τ)F(τ) dτ = IKC×KC for identifiability.

We will eliminate θt by assuming that Xt is the identity matrixand Vt = 0 so that βt = θt .

Page 31: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

The Dynamic Model

c = 1 is the Fed. We will investigate how this dominant economy

affects the other three economies. Our model is

∆β(1)k,t = ω(1)

k,t (1)

∆β(c)k,t = s(c)

k,t (γ(c)k ∆β(1)

k,t ) + ω(c)k,t c = 2, . . . , C (2)

In equation (1):

• ∆ is the differencing operator

• The ω(c)k,t are AR(r) processes

• so β(1)k,t is an ARIMA(1,1,0) process

Page 32: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

The Dynamic Model

From the previous frame:

∆β(1)k,t = ω(1)

k,t (1)∆β(c)

k,t = s(c)k,t (γ(c)

k ∆β(1)k,t ) + ω(c)

k,t c = 2, . . . , C (2)

In the equation (2):

s(c)k,t : t = 1, . . . , T

is a discrete Markov chain with states

0, 1.• s(c)

k,t = 1 implies that at time t, the kth factor for the ctheconomy is affected by the kth factor of the US economy.

• γ(c)k ∈ R is the economy-specific slope term for the kth factor.

Page 33: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

The Dynamic Model

From the previous frame:

∆β(1)k,t = ω(1)

k,t (3)

∆β(c)k,t = s(c)

k,t (γ(c)k ∆β(1)

k,t ) + ω(c)k,t c = 2, . . . , C (4)

• This is the autoregressive regime switching model of Albertand Chib (1993) and McCulloch and Tsay (1993).

• Define q(c)ii,k to be the probability of switching from state i to

state i , i, i ∈ 0, 1.

Page 34: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Posterior estimates of P(s(c)k,t = 1).

0.0

0.2

0.4

0.6

0.8

1.0

k = 1

Dates

Prob

abilit

y of

Sta

te 1

12/07 02/08 04/08 06/08 08/08 10/08 12/08 02/09 04/09

BOE

ECB

BOC

0.0

0.2

0.4

0.6

0.8

1.0

k = 2

Dates

Prob

abilit

y of

Sta

te 1

12/07 02/08 04/08 06/08 08/08 10/08 12/08 02/09 04/09

0.0

0.2

0.4

0.6

0.8

1.0

k = 3

Dates

Prob

abilit

y of

Sta

te 1

12/07 02/08 04/08 06/08 08/08 10/08 12/08 02/09 04/09

0.0

0.2

0.4

0.6

0.8

1.0

k = 4

Dates

Prob

abilit

y of

Sta

te 1

12/07 02/08 04/08 06/08 08/08 10/08 12/08 02/09 04/09

We see that changes in the first factor for the US economy are

associated with similar changes in the first factor for the other

three economies, especially BOE and BOC.

Page 35: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Functional Time Series Analysis

Page 36: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Functional Time Series Analysis

Page 37: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Overview

FDLM

Loading Curves

Gibbs Sampler

Application

David S. Matteson Cornell University [email protected] A Bayesian Multivariate Functional Dynamic Linear Model

Page 38: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Overview

FDLM

Loading Curves

Gibbs Sampler

Application

David S. Matteson Cornell University [email protected] A Bayesian Multivariate Functional Dynamic Linear Model

Page 39: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Stochastic Volatility

Recall the dynamic model:

∆β(c)k,t = s(c)

k,t (γ(c)k ∆β(1)

k,t ) + ω(c)k,t , c = 2, . . . , C .

We modeled the noise ω(c)k,t as AR(r) with stochastic volatility:

ω(c)k,t =

r

i=1ψ(c)

k,i ω(c)k,t−i + σk,(c),t z(c)

k,t

where, following Kim et al. (1998) and Chib et al. (2002),

• log(σk,(c),t) is an AR(1) process.

Page 40: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Other Factor Models: Jungbacker et al.

Jungbacker et al. (2014) also use splines for the factor loading

curves

• They use Wald, Lagrange multiplier, and likelihood ratio tests

to reduce the number of knots.

• They work with only the US economy.

• They assume constant conditional volatility.

Page 41: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Other Factor Models: Nelson-Siegel Model

The Nelson-Siegel (1987) model for the yield curve is:

Y (τ) = θ0 + (θ1 + θ2τ) exp(−θ3τ).

• This model is widely used in the banking industry.

• Empirically, it does not fit as well as spline models, which arealso widely used.

Svensson’s (1994) model adds θ4τ exp(−θ5τ) to the Nelson-Siegelmodel.

• This improves the fit, but still not as much as using splines.

Page 42: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Diebold and Li (2006)

Nelson and Siegel formulated their model as a static one.

• Diebold and Li (2006) reformulated the Nelson-Siegel modelin dynamic form.

• λ determines the location of the maximum of the third factorloading curve.

• Diebold and Li fixed λ in advance so that the location of thismaximum was sensible.

• With λ fixed, the Nelson-Siegel model is linear.

Page 43: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Nelson-Siegel Model in Factor Form

The Nelson-Siegel model can be expressed as a dynamic model

with three parametric factor loading curves:

1 (constant)

1 − exp(−λτ)λt

1 − exp(−λτ)λτ

− exp(−λτ).

Here λ = θ3 is fixed and the factors are linear combinations of θ0,

θ1, and θ2.

Recall: The Nelson-Siegel model is θ0 + (θ1 + θ2τ) exp(−θ3τ).

Page 44: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Empircal Versus Nelson-Siegel Factor Loading Curves

0 50 100 150 200 250 300 350

0.0

0.2

0.4

0.6

0.8

1.0

maturity (months)

fact

or lo

adin

g cu

rve

f1f2f3

Nelson-Siegel withλ = 0.4/12

0 50 100 150 200 250 300 350

−0.3

−0.2

−0.1

0.0

0.1

Common Factor Loading Curves

Maturity (months)

Yiel

d (%

)

k = 1k = 2k = 3k = 4

Empirical from the MFDLM

Page 45: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Overview

FDLM

Loading Curves

Gibbs Sampler

Application

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic Linear Model

Page 46: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Hays et al. (2012

Hays et al. (2012) developed a functional dynamic factor model

and applied it to yield curve forecasting.

• We view our work at least somewhat as a generalization of

theirs.

• They worked with one economy.

• Their analysis is non-Bayesian.

• A stochastic EM-algorithm was used.

• Smoothing parameters were fit by GCV.

• Stochastic volatility and covariates seem more difficult to add

compared to within a Bayesian analysis.

• They found that their model outperformed a dynamic

Nelson-Siegel in terms of forecasting accuracy.

Page 47: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Local Field Potentials

• Neural activity in a region of the brain can be detected byrecording from a pair of electrode in that region.

• The electrodes measure the local field potential (LFP) in the

region.

• Usually done with rodents.• Performed on humans only rarely and only for medical reasons.

• LFP data are similar to data obtained from EEG whereelectrodes are placed on the scalp.

• LPF signals are more localized than EEG signals.

Page 48: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Our Data

• LFP data are usually analyzed in the frequency domain.

• Our data were collected by PhD student Vladmir Ljubojevicworking with Professor Eve De Rosa at U. Toronto.

• They are now both at Cornell where Vlad is a postdoc.

• Pairs of electrodes where placed in two locations:• prefrontal cortex (PFC)• posterior parietal cortex (PPC)

• These two regions have been proposed as the neural substrate

of attention.

• Sampling was at 4000 Hertz.

Page 49: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

FS and FC

• There were two experimental conditions:• feature singleton (FS) — the rat processes a single stimulus.• feature conjunction (FC) — the rat must integrate two stimuli.

• In the experiments, the rat attempts to select the one of twobowls with a treat.

• In FS, the correct bowl is indicated by a single smell.• In FC, the correct bowl is indicated by two smells.

• FS serves as a baseline.

• We are interested in synchronization of the two brain regionsduring FC.

Page 50: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Data Processing

• Tasks are repeated for 20 trials per rat for each of FS and FC.

• For each trial, we look at a 3 sec window centered at the time

the rat makes a decision.

• The data are binned into 15 time bins.

• Each is of length 3/8 sec.

• 50% overlap.

Page 51: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

LFP Signals from a Single Trial

Page 52: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Spectra

• The periodgram (discrete Fourier transform) of the bivariatetime series is computed in each bin.

• The periodgrams are smoothed with a modified Daniell kernelto obtain spectra.

• I (c)t (τ) is• the spectrum of PFC for c = 1,• the spectrum of PPC for c = 2,• the cross-spectrum for c = 3.

• The squared coherence is

κ2t (τ) ≡ |I (3)

t (τ)|2

I (1)t (τ)I (2)

t (τ)

Page 53: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Time-frequency Analysis

• Y (c)t (τ) = log

I (c)t (τ)

for c = 1, 2.

• 0 ≤ κ2t (τ) ≤ 1

• Let Y (3)t (τ) = Φ−1(κ2

t (τ))• The link function Φ−1

is the standard normal quantile function.

The indices are:

• t = 1, . . . , 15.

• c = 1, 2, 3.

• τ ranges from 0.1 to 80 Hertz.

• 30 observation points after truncation above 80 Hertz.

• There were about 750 frequencies, but most were above 80

Hertz and not of interest.

Page 54: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Time-frequency Plots for FS Trials

Page 55: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

MFDLM SpecificationWe assume

• common factor loading curves, and• a random walk model for the factors.

Y (c)i,s,t(τ) =

K

k=1β(c)

k,i,s,t fk(τ) + (c)i,s,t(τ),

(c)

i,s,t(τ)σ2

(c)

indep∼ N (0, σ2(c))

βk,i,s,t = βk,i,s,t−1 + ωk,i,s,t ,ωk,i,s,t

Wk indep∼ N (0, Wk)

• i = 1, . . . , 8 is the rat index.• s = 1, . . . , 40 is the index of the trial within a rat.• t = 1, . . . , 15 is the index of the time bins.• c = 1, 2, 3 is the outcome index.• k = 1, . . . , K is factor index.

Page 56: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Comparing FS and FC

• DIC selected K = 10 factors.

• µ(c)i,s,t(τ) ≡

10k=1 β(c)

k,i,s,tfk(τ) is the estimated mean for the ith

rat, sth trial within that rat, and tth time bin.

• We focused on the difference in squared coherence between

FC and FS.

Page 57: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Posterior and 95% HPD Confidence Intervals for Difference

in Squared Coherene

−0.10

−0.05

0.00

0.05

0.10

2 4 6 8 10 12 14

10

20

30

40

50

60

70

80Lower 95% HPD Interval

Time Bin

Freq

uenc

y (H

z)

−0.

06

−0.06

−0.04

−0.04

−0.04

−0.04

−0.04

−0.0

4

−0.04

−0.04

−0.04

−0.04

−0.04

−0.02

−0.02

−0.02

−0.02

−0.02

−0.02

−0.02

−0.02

−0.02

−0.02

−0.02

0

0

0

0

0

0

0

0

0

0

0

0

0.02

0.0

2

0.0

2

0.02

0.02

0.02

0.0

2

0.02

0.0

2

0.04 0.04

0.0

4 0

.06

2 4 6 8 10 12 14

10

20

30

40

50

60

70

80Posterior Mean

Time Bin

−0.

04

−0.04

−0.02

−0.02 −0.02

−0.02

−0.02

−0.02

−0.

02

−0.02

−0.02

−0.02 −0.02

0

0

0

0

0

0

0

0

0

0.02

0.02

0.02

0.02

0.0

2

0.02

0.02

0.02

0.02

0.02

0.02

0.02

0.02

0.04

0.0

4

0.0

4

0.04

0.04

0.04

0.0

4

0.04

0.06

0.06

0.06

0.0

8

2 4 6 8 10 12 14

10

20

30

40

50

60

70

80Upper 95% HPD Interval

Time Bin

−0.02

−0.02

−0.

02

−0.02

−0.02

0

0

0

0

0

0

0

0

0

0 0

0

0

0.02

0.0

2

0.0

2

0.02

0.02

0.02 0.0

2

0.02

0.0

2

0.04

0.04

0.04

0.0

4

0.04

0.04

0.0

4

0.04

0.04

0.06

0.06

0.06

0.06

0.06 0.06 0

.08

0.08

FB is associated with increased coherence in the Theta range (4–8

Hz), Alpha range (8–13 Hz), and Beta range (13–30 Hz).

Page 58: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

References

Albert, J. H. and Chib, S. (1993).

Bayes inference via Gibbs sampling of autoregressive time series subject to

Markov mean and variance shifts.

Journal of Business & Economic Statistics, 11(1):1–15.

Chib, S., Nardari, F., and Shephard, N. (2002).

Markov chain Monte Carlo methods for stochastic volatility models.

Journal of Econometrics, 108(2):281–316.

Diebold, F. X. and Li, C. (2006).

Forecasting the term structure of government bond yields.

Journal of Econometrics, 130(2):337–364.

Hays, S., Shen, H., and Huang, J. Z. (2012).

Functional dynamic factor models with application to yield curve forecasting.

The Annals of Applied Statistics, 6(3):870–894.

Page 59: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

References

Jungbacker, B., Koopman, S. J., and van der Wel, M. (2013).

Smooth dynamic factor analysis with application to the US term structure of

interest rates.

Journal of Applied Econometrics.Kim, S., Shephard, N., and Chib, S. (1998).

Stochastic volatility: likelihood inference and comparison with ARCH models.

The Review of Economic Studies, 65(3):361–393.

Krivobokova, T. (2013)

Smoothing parameter selection in two frameworks for penalized splines

Journal of the Royal Statistical Society, Series B, 75, 725–741.

McCulloch, R. E. and Tsay, R. S. (1993).

Bayesian inference and prediction for mean and variance shifts in autoregressive

time series.

Journal of the American Statistical Association, 88(423):968–978.

Page 60: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

References

Nelson, C. R. and Siegel, A. F. (1987).

Parsimonious modeling of yield curves.

Journal of Business, 60(4):473

Svensson, L. E. (1994).

Estimating and interpreting forward interest rates: Sweden 1992-1994.

Technical report, National Bureau of Economic Research.

Wahba, G. (1985)

A comparison of GCV and GML for choosing the smoothing parameter in the

generalized spline smoothing problem.

Annals of Statistics, 13, 1378–1402.

Wand, M. P. and Ormerod, J. (2008).

On semiparametric regression with O’Sullivan penalized splines.

Australian and New England Journal of Statistics, 50:179–198.

Page 61: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Original Motivation Methodology Applications Future work References

Applied BioPhysics ECIS data

ElectronicCell-substrateImpedance Sensingprocess, withelectrode assubstrate. Currenthas two possiblepath types.

Image: Applied BioPhysics,

www.biophysics.com, c2015.

Laura L. Tupper

Cell Phenotyping and Classification with Time Series Data

Page 62: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Original Motivation Methodology Applications Future work References

Applied BioPhysics ECIS data

Laura L. Tupper

Cell Phenotyping and Classification with Time Series Data

Page 63: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Original Motivation Methodology Applications Future work References

Applied BioPhysics ECIS data

We focus on

resistance time

series

Examine early

behavior and

spreading

Use time series

collected using

9 frequencies

from 250 to

64000 Hz

Laura L. Tupper

Cell Phenotyping and Classification with Time Series Data

Page 64: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

ECIS: Multivariate Time Series

0 10 20 30 40 50 60

5000

15000

25000

WI38

time

R

0 10 20 30 40 50 60

5000

15000

25000

HUTU80

time

R

0 10 20 30 40 50 60

5000

15000

25000

VA13

time

R

0 10 20 30 40 50 60

5000

15000

25000

BSC1

time

R

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Functional Time Series Analysis

Page 65: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

ECIS: Functional Data

0 10000 20000 30000 40000 50000 60000

5000

15000

25000

WI38

freq

R

0 10000 20000 30000 40000 50000 60000

5000

15000

25000

HUTU80

freq

R

0 10000 20000 30000 40000 50000 60000

5000

15000

25000

VA13

freq

R

0 10000 20000 30000 40000 50000 60000

5000

15000

25000

BSC1

freq

R

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Functional Time Series Analysis

Page 66: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

ECIS: Functional Time Series

freq

time

z

WI38

5000

10000

15000

20000

25000

freqtime

z

HUTU80

2000400060008000100001200014000

freq

time

z

VA13

200040006000800010000120001400016000

freq

time

z

BSC1

2000400060008000100001200014000

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Functional Time Series Analysis

Page 67: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

ECIS: Functional Time Series

freq

time

z

BSC1

2000

4000

6000

8000

10000

12000

14000

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Functional Time Series Analysis

Page 68: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

ECIS: A Common Functional Basis

0 20000 40000 60000 80000

−0.6

−0.2

0.2

0.6

Common Factor Loading Curves

!

f k(!)

k = 1k = 2k = 3k = 4

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Functional Time Series Analysis

Page 69: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

ECIS: Time Series of Weights (Factors/Scores)

0 10 20 30 40 50 60

−0.3

0.0

0.2

k = 1

BSC−1Time

Beta

0 10 20 30 40 50 60

02

46

k = 2

BSC−1Time

Beta

0 10 20 30 40 50 60

−3−2

−10

1

k = 3

BSC−1Time

Beta

0 10 20 30 40 50 60

−0.3

−0.1

0.1

k = 4

BSC−1Time

Beta

0 10 20 30 40 50 60

−0.3

0.0

0.2

k = 1

HUTU−80Time

Beta

0 10 20 30 40 50 60

02

46

k = 2

HUTU−80Time

Beta

0 10 20 30 40 50 60

−3−2

−10

1

k = 3

HUTU−80Time

Beta

0 10 20 30 40 50 60

−0.3

−0.1

0.1

k = 4

HUTU−80Time

Beta

0 10 20 30 40 50 60

−0.3

0.0

0.2

k = 1

WI−38Time

Beta

0 10 20 30 40 50 60

02

46

k = 2

WI−38Time

Beta

0 10 20 30 40 50 60−3

−2−1

01

k = 3

WI−38Time

Beta

0 10 20 30 40 50 60

−0.3

−0.1

0.1

k = 4

WI−38Time

Beta

0 10 20 30 40 50 60

−0.3

0.0

0.2

k = 1

VA−13Time

Beta

0 10 20 30 40 50 60

02

46

k = 2

VA−13Time

Beta

0 10 20 30 40 50 60

−3−2

−10

1

k = 3

VA−13Time

Beta

0 10 20 30 40 50 60

−0.3

−0.1

0.1

k = 4

VA−13Time

Beta

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Functional Time Series Analysis

Page 70: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

FDLM: Random Effects

Model with many replicates (i):

Function(i)t = Curves,Weightst + Curves,Weights(i)

t + Noiset

Weightst = Weightst−1 + Noiset (common signal)

Weights(i)t = Weights(i)

t−1 + Noise(i)t (∀i)

No Redundancy in Curves

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Functional Time Series Analysis

Page 71: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Overview

FDLM

Loading Curves

Gibbs Sampler

Application

Natural Cubic Splines

• Suppose we observe (xi , yi )ni=1

• Goal: model y as a function of x , assuming only that theunderlying function is twice continuously differentiable

Natural cubic splines (NCS) arise as the minimizer of

S(g) =n

i=1

(yi − g(xi ))2 + λ

b

a(g(x))2 dx

= ||y − g||2 + λgΩg

where g = [g(x1), ..., g(xn)] and Ω is a a known matrix that

depends only on the xi.• λ→∞⇒ g is linear; λ→ 0⇒ g is smoothest interpolator

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic Linear Model

Page 72: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Overview

FDLM

Loading Curves

Gibbs Sampler

Application

Natural Cubic Splines

So the NCS is smooth, optimal, and computationallyconvenient:

g = (In + λΩ)−1y

Note:• g is restricted to an infinite-dimensional space,

but the solution g is an n-dimensional vector• We can compute g(x) for any x ∈ [a, b] using g

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic Linear Model

Page 73: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Overview

FDLM

Loading Curves

Gibbs Sampler

Application

Bayesian Splines

For Bayesian penalized regression, we usually have1 A normal likelihood for the least squares criterion; and2 A prior for the penalty term.

Consider the common regression model

yi = g(xi ) + i

where iiid∼ N(0, σ2).

Since our penalty is a quadratic form, a Gaussian like priorshould be appropriate...

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic Linear Model

Page 74: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Overview

FDLM

Loading Curves

Gibbs Sampler

Application

Partially Informative Normal Distribution

We say z is partially informative normal, written z ∼ PIN(µ,Σ), if

[z] ∝ |Σ|1/2+ exp

−1

2(z− µ)Σ(z− µ)

where the operator | · |+ takes the product of the nonzeroeigenvalues of Σ, which is restricted to be symmetric andpositive semidefinite.

• If Σ is positive definite, then z ∼ Nµ,Σ−1

• More useful when Σ is positive semidefinite but notpositive definite, in which case the prior is improper

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic Linear Model

Page 75: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Overview

FDLM

Loading Curves

Gibbs Sampler

Application

NCS Posterior Distribution

Letting g ∼ PIN(0, λΩ), we obtain the posterior

[g|y, λ] ∝ [y|g, λ][g|λ]

∝ exp

−1

2(y − g) σ−2In (y − g)

exp

−1

2g (λΩ) g

∼ N(In + λΩ)−1

y, (In + λΩ)−1

The posterior mean is g.

But more generally, we can compute posterior quantities forfunctionals of g , e.g. max/min, derivatives, etc.

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic Linear Model

Page 76: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Overview

FDLM

Loading Curves

Gibbs Sampler

Application

More about the Prior

Should we be concerned about the prior g ∼ PIN(0, λΩ)?

• rank(Ω) = n − 2, so the prior is improper...but this makessome sense:

• Constant and linear terms are unpenalized

• Flat prior over this space (although not necessary)

• But there are other ways to arrive at the same posteriorusing proper priors and then taking limits in the posterior

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic Linear Model

Page 77: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Overview

FDLM

Loading Curves

Gibbs Sampler

Application

Constrained Loading Curves

For the loading curves fk , we extend the penalized regression

framework to incorporate additional constraints

(orthonormality), and then move everything into the Bayesian

setting.

Again, ignore c for simplicity.

We need an observation model:

Yt(τ) =K

k=1

fk(τ)βk,t + t(τ)

where t(τ)iid∼ N(0, σ2). Here, we condition on βk,t and σ2

.

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic Linear Model

Page 78: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Overview

FDLM

Loading Curves

Gibbs Sampler

Application

Identifiability and Orthonormality

For identifiability, we impose the constraint

τ∈Tfk(τ)f(τ)dτ = 1(k=)

1 Unit-norm constraint allows for comparisons among the fk2 Orthogonality across k reduces information overlap among

the curves (and thus the number of curves necessary)

• “Basis-like” with a functional inner product

3 As before, we can write the integral in terms of vectors:

τ∈Tfk(τ)f(τ)dτ = f

k f

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic Linear Model

Page 79: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Overview

FDLM

Loading Curves

Gibbs Sampler

Application

Sequential Orthogonality

In practice, we impose the orthogonality constraint sequentially:

τ∈Tfk(τ)f(τ)dτ = f

k f = 0, for < k

which satisfies joint orthogonality, but also:

1 Allows for easy estimation within our Gibbs sampler

2 Only imposes k − 1 constraints on each curve, rather than

K − 1• So f1 is a NCS, and fk is “close” to a NCS

3 Suggests a natural ordering of the curves, which guards

against label-switching

Think PCA!

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic Linear Model

Page 80: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Overview

FDLM

Loading Curves

Gibbs Sampler

Application

The Loading Curve Constrained Optimization Problem

In integral form, our constrained optimization problem for eachloading curve fk is

minfk1σ2

Tt=1

mj=1

Yt(τj)−

Kk=1 fk(τj)βk,t

2+ λk

τ∈T

fk(τ)

2dτ

subject to

τ∈T fk(τ)f(τ)dτ = 1(k=), for ≤ k

The top line is the familiar penalized least squares criterion fornatural cubic splines (already in Lagrange multiplier form), andwe add our identifiability constraint.

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic Linear Model

Page 81: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Overview

FDLM

Loading Curves

Gibbs Sampler

Application

Transitioning to Vectors and Matrices

Letting Y be the T ×m matrix of yield curve observations and

Bk = (βk,1, ...,βk,T ), we can rewrite the likelihood in matrix

form as

Y =K

k=1

Bk fk +

or perhaps more useful in the Bayesian setting,

Y−k ≡ Y −

l =k

Bf = Bk f

k +

since we will condition on BkKk=1 and f=k (LHS known).

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic Linear Model

Page 82: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Overview

FDLM

Loading Curves

Gibbs Sampler

Application

The Loading Curve Constrained Optimization Problem

Putting it all together in vector/matrix notation, we get

minfk σ−2Y−k − Bk f

k

2 + λk fkΩfk

subject to f k f = 0 for l < k

and f k fk = 1

For simplicity (and later, interpretability), we drop the unitnorm constraint and solve the resulting quadratic program;then we normalize the solution.

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic Linear Model

Page 83: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Overview

FDLM

Loading Curves

Gibbs Sampler

Application

Constrained Bayesian Splines

As before, we choose a prior so that the posterior distribution

will be normal with expectation equal to the solution of our

optimization problem:

[fk ] ∝ PIN(0, λkΩ) exp−f

kF

1:(k−1)λ

∗1:(k−1)

Here, F1:(k−1) and λ∗1:(k−1) are from the Lagrangian.

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic Linear Model

Page 84: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Overview

FDLM

Loading Curves

Gibbs Sampler

Application

Exponential Tilting.

So what’s special about this?

[fk ] ∝ PIN(0, λkΩ) exp−f

kF

1:(k−1)λ

∗1:(k−1)

is the usual partially informative normal prior for natural cubicsplines, but exponentially tilted by our constraint.

• Connections to exponential family theory

Should hold at least for any constraints that can be written inLagrange multiplier form.

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic Linear Model

Page 85: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Overview

FDLM

Loading Curves

Gibbs Sampler

Application

Constrained NCS Posterior Distribution

The resulting posterior distribution is N (Vfkufk ,Vfk ) where

ufk =Y−kBk

σ2− F

1:(k−1)λ

∗1:(k−1)

and

V−1fk

=||Bk ||2

σ2Im + λkΩ

As with the Bayesian spline, the posterior mean solves theconvex optimization problem.

Extensions to C > 1 are easy.

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic Linear Model

Page 86: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Overview

FDLM

Loading Curves

Gibbs Sampler

Application

Summarizing the Constrained NCS

Some observations:

1 Our spline-based constrained optimization approach is

immediately recognizable in the prior

2 The posterior distribution follows naturally from Bayesian

spline theory

3 The unit-norm constraint is intuitive as a projection of a

normal distribution onto the unit sphere

• Does not change the shape of the curves

• More appropriate for sampling

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic Linear Model

Page 87: A Bayesian Multivariate Functional Dynamic Linear Model · A Bayesian Multivariate Functional Dynamic Linear ... arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic

Overview

FDLM

Loading Curves

Gibbs Sampler

Application

Gibbs Sampler

Briefly:1 Initialize by computing “optimal” B and F using SVDs;

then estimate everything else via conditional MLEs2 Sample the loading curves fk using eigendecompositions

(and a regression for the Lagrange multipliers)• Sample the smoothing parameters λk like precisions

3 Sample the βt using DLM algorithms4 Sample the variance parameters and any covariate

parameters (individually)

David S. Matteson Cornell University [email protected] http://arxiv.org/abs/1411.0764A Bayesian Multivariate Functional Dynamic Linear Model