Stochastic Collocation Methods for Polynomial Chaos: Analysis and Applications

Dongbin

Xiu

Department of Mathematics, Purdue University

Support:

AFOSR FA9550-08-1-0353 (Computational Math)NSF CAREER DMS-0645035 (Computational Math)DOE DE-FC52-08NA28617 (PSAAP)

Overview

• Generalized polynomial chaos

• Stochastic collocation• Lagrange interpolation• Pseudo spectral gPC

• Applications• Bayesian inverse problem• Data assimilation

0

ˆ( , , ) ( , ) ( ), # of basisN

zN

z

n Nu t x Z u t x Z

nk kk=

⎛ ⎞+ ⎟⎜ ⎟Φ = ⎜ ⎟⎜ ⎟⎝ ⎠∑• Nth-order gPC

expansion:

( ) ( ) ( ) ( ) ( )Z Z Z Z Z Z dZi j i j ijE ρ δ⎡ ⎤Φ Φ Φ Φ =⎣ ⎦ ∫• Orthogonal basis:

Uncertainty Quantification via gPC

0

( , , ) ( ), (0, ]

( ) 0, [0, ]( , ), { 0}

z

z

z

n

n

n

u t x Z u T Dtu T D

u u x Z t D

∂⎧ = × ×⎪ ∂⎪

= ×∂ ×⎨⎪ = = × ×⎪⎩

L

B

R

RR

• Stochastic PDE:

ˆ [ ( ) ( )] ( ) ( ) ( ) , 0 ,u u Z Z u Z Z Z dZ Nk k k kE ρ= Φ = Φ ≤ ≤∫

2 2( ) ( )inf

nzN

N L Z L Zu u u

ρ ρΨ∈Π− = − Ψ• Optimality:

Generalized Polynomial Chaos (gPC)

• Implementations:Stochastic GalerkinStochastic collocation

• Properties:Rigorous mathematicsHigh accuracy, fast covergenceCurse-of-dimensionality

• Basis functions:Hermite polynomials: seminal work by R. GhanemGlobal orthogonal polynomials (Xiu & Karniadakis, 02)Wavelet basis (Le Maitre et al, 04)Piecewise basis (Babuska et al 04, Wan & Karniadakis, 05)

• Lagrange interpolation• Can not be constructed for any given nodes• Interpolation error hard to control

• Pseudo spectral• Utilize gPC

polynomial basis

• Becomes multivariate integration

Stochastic Collocation

• Sampling:

(solution statistics only)• Random (Monte Carlo)• Deterministic (lattice rule, tensor grid, cubature)

• Response surface method• Multivariate interpolation• Many ad hoc approaches

• Collocation:

To satisfy governing equations at nodes

• Stochastic collocation:

To construct polynomial approximations

Stochastic Collocation –

Lagrange Interpolation

1

( ) ( ) ( )Q

Q jj

j

u Z u Z L Z=∑ ( ) , 1 ,j

i ijL Z i j Qδ= ≤ ≤• Lagrange interpolation:

{ }1

zQ ni

Q iZ R

=Θ = ∈• Nodal set:

0

( , , ) ( ), in (0, ] ,

( ) 0, [0, ] ,( , ), { 0}

j

j

u t x Z u T Dtu T D

u u x Z t D

∂= ×

∂= ×∂

= = ×

L

B

• Solution:

for j=1,…,Q,

(Xiu & Hesthaven, SIAM J. Sci. Comput., 05)

( )1 nziiU U⊗ ⊗

• Tensor product:

1

1

1( 1) ( )nziq i

q N q

NU U

qi

i i−

− + ≤ ≤

⎛ ⎞− ⎟⎜ ⎟− ⋅ ⋅ ⊗ ⊗⎜ ⎟⎜ ⎟⎜ −⎝ ⎠∑• Sparse grid (Smolyak):

Stochastic Collocation: Pseudo Spectral Approach

• Nth-order gPC

projection

2 ( )Q N N L Zu w

ρε −• Aliasing Error:

0

ˆ( , , ) ( , ) ( ),N

Nw t x Z w t x Zk kk=

= Φ∑

1

ˆ ( , , ) ( ) ( ) ( ) ( )Q

j j j

j

w u t x Z Z u Z Z Z dZk k kα ρ=

= Φ ≈ Φ∑ ∫

• gPC-collocation approximation

0

ˆ( , , ) ( , ) ( ),N

Nu t x Z u t x Zk kk=

= Φ∑ ˆ ( ) ( ) ( ) .u u Z Z Z dZk k ρ= Φ∫

ˆ ˆ( , ) ( , ),w t x u t x Q→ →∞k k

(Xiu, Comm. Comput Phys, vol. 2, 07)

gPC-Collocation: Algorithm{ }

11. Choose a nodal set , in z

Q nj j

jZ Rα

=

1

3. Evaluate the approximate gPC expansion coefficient

ˆ ( , , ) ( ) , 0 ;Q

j j j

j

w u t x Z Z Nk k kα=

= Φ ≤ ≤∑th

1

4. Construct the -order gPC approximation

ˆ ( , , ) ( ).N

N

N

w t x Z w Zk kk=

= Φ∑

Post-process

Deterministic solver

0

2. Solve for each 1, , ,

( , , ) ( ), in (0, ] ,

( ) 0, [0, ] ,( , ), { 0}

j

j

j Qu t x Z u T Dtu T D

u u x Z t D

=∂

= ×∂

= ×∂

= = ×

L

B

( )2

1/ 22 2 2 2( )N N Q QL Z

u w M Cρ

ε ε ε εΔ− ≤ + +

Finite-term projection error aliasing error Numerical errorError ≤ + +

• Error bound (Xiu, 07):

Parameter Estimation: Bayesian Inverse Approach

0

( , , ) ( ), (0, ]

( ) 0, [0, ]( , ), { 0}

z

z

z

n

n

n

u t x Z u T Dtu T D

u u x Z t D

∂⎧ = × ×⎪ ∂⎪

= ×∂ ×⎨⎪ = = × ×⎪⎩

L

B

R

RR

• Stochastic PDE:

( , , ) : [0, ] uz nnu t x Z T D× ×R R• Solution:

1

( ) ( )zn

Z i ii

z zπ π=

=∏• Prior distribution:

• Estimation of the prior distributionRequires direct measurements of the parametersNo/not enough direct measurements? (Use experience/intuition …)How to take advantage of measurements of other variables?

( ) , is i.i.d.dnd G Z e e= + ∈R• Data:

Bayesian Inference

( | ) ( )( ) ( | )( | ) ( )

d d Z ZZ Z dd Z Z dZ

π ππ ππ π

=∫

1

( ) ( | ) ( ( ))d

i

n

e i ii

L Z d Z d G Zπ π=

= −∏• Likelihood function:

• Posterior distribution:

• Notes:Difficult to manipulateClassical sampling approaches can be time consuming (MCMC, etc)GPC (Galerkin) based approach: (Marzouk, Najm, Rahn, JCP, 07)

( ) ( )( )( ) ( )

d NN

N

L Z ZZL Z Z dZ

πππ

=∫

,1

( ) ( | ) ( ( ))d

i

n

N N e i N ii

L Z d Z d G Zπ π=

= −∏

• gPC

approximation:

• Properties:Allows direct sampling in term of Z with arbitrarily large samples(Virtually) no additional computational cost – forward problem solver onlyConvergence seems natural

2(0, ), i.i.d. Normale N σ I∼

( ) 11 2 1

2

( )( ) log( )zD z dzz

ππ π ππ∫

Convergence of gPC

Bayesian Inference

• Kullback-Leibler

divergence:

• Observation error:

( )

2

2

i ,

. If the gPC expansion converges to in , then the posterior density

converges to in the sense

D 0, .

Moreover, if

( ) ( )

Z

z

N

d dN

d dN

N i L

G G L

N

G Z G Zπ

π

π π

π π → →∞

− ≤

Theorem

( ) 2

, 1 , 0, independent of ,

then for sufficiently large ,

.

d

d dN

CN i n C N

N

D N

α

α

α

π π

−

−

≤ ≤ >

∼

• Notes:Fast (exponential) convergence rate is retainedFactor of 2 in the convergence rates

(Marzouk & Xiu, Comm. Comput. Phys. 08)

Parameter Estimation: Supersensitivity

Example

[ ]2

2 , 1,1u u uu xt x x

ν∂ ∂ ∂+ = ∈ −∂ ∂ ∂

• Burgers’

equation :

( 1) 1 ( ); (1) 1; 0 1u Z uδ δ− = + =− < <<• Boundary conditions :

(Xiu & Karniadakis, Int. J. Numer. Eng., 04)1% uncertainty in left BC

Deterministic results with no uncertainty

0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10

5

10

15

20

25

δ

p(δ

| dat

a )

exact posteriorgPC, p=4gPC, p=8δ

true

Measurement noise: e~N(0,0.052)

Prior distribution is uniform

6 7 8 9 10 11 12 13 14 15 16

10−3

10−2

N

erro

r

D(πN

|| π )

||G − GN

||L

2

Factor = 2.10 (theory = 2)

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−0.2

0

0.2

0.4

0.6

0.8

1

1.2

z

G(z

) or

GN

(z)

exact forward solutiongPC approximation

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 10

0.5

1

1.5

2

2.5

3

z

πd (z)

exact posteriorgPC posterior

Parameter Estimation: Step Function

• Assume the forward model is a step function• Posterior distribution is discontinuous• Gibb’s oscillations exist• Slow convergence with global gPC

basis functions

Forward model and its approximation Posterior distribution and its approximation

2 4 6 8 10 12 14 16 18 20

10−0.9

10−0.8

10−0.7

10−0.6

10−0.5

10−0.4

10−0.3

10−0.2

10−0.1

p

erro

r

D(π

N || π )

||G − GN

||L

2

Factor = 1.99 (theory = 2)

Kalman

Filter for Data Assimilation

• Forecast:

• Observation:

• Analysis:

0

( , ) ( , ), (0, ]

(0, )

ff

f

d t Z t t Tdt

Z

⎧= ∈⎪

⎨⎪ =⎩

u F u

u u

, :t m= + ∈ →d Hu ε HR R R

• True

state (unknown): , 1t m m∈ ≥u R

( )a f f= + −u u K d Hu

1( )f T f T −= +K P H HP H R

( )( )f f t f t T⎡ ⎤= − −⎣ ⎦P u u u uE T⎡ ⎤= ⎣ ⎦R εεE

(Kalman

gain matrix)

• Properties:• Straightforward for linear dynamic equations• Extension to nonlinear equations: Extended KF (EKF)• Optimal for Gaussian• Explicit calculation of covariance can be costly

Ensemble Kalman

Filter (EnKF)

( )( )f ff f f T f

e = − − ≈P u u u u P Te = ≈R εε R

( ) ( ) ( )( ) , 1, ,a f fe ii i i

i M⎡ ⎤= + − =⎣ ⎦u u K d H u …

1( )f T f Te e e e

−= +K P H HP Η R

• Properties:nonlinear dynamicssampling errors

Measurement. Can be eliminated by square-root filter (EnSRF)Solution states.

Computational cost is of great concern

( ) ( , ), 1, ,f f i

it Z i M=u u … ( ) ( ) , 1, ,ii

i N= + =d d ε …• Ensemble:

Error Analysis of the EnKF

1n nT t t+Δ = −

( )1

,

fn t t

p

e

O t N α

ε ε ε

σ

+ Δ Δ

−

≤ ⋅ + Δ ⋅ + Δ ⋅ ⋅

Δ

M K K H

∼

= −M I KH eΔ = −K K K

( )01

expn

n k nk

E E e t=

⎛ ⎞≤ + Λ ⋅⎜ ⎟⎝ ⎠

∑

• Lemma

(local error):

• Theorem

(global error):

1T −Λ ∝ Δ

• Note the inverse dependence

on assimilation step size

• Assimilation step size:

(Li & Xiu, vol. 197, CMAME 08)

EnKF

Example: Linear Wave Equation

0 100 200 300 400 500 600 700 800 900 10000

5

10

0 100 200 300 400 500 600 700 800 900 10000

5

10

0 100 200 300 400 500 600 700 800 900 10000

2

4

6

8

Long-term Wave propagation

t=5

t=100

t=500

50( , )x Z ∈ ×R R

Model description:Linear advection equation;Periodic domain of length L=1000;Wave speed = 1; grid spacing=1; time step = 1;True states are sampled from a Gaussian process, with zero mean and unit variance, and a spatial de-correlation length of 20. The dimension of random space is nz=50.Four measurements uniformly in space are made every 5 time units.Measurement variance is 0.01.No model error.

Error Behavior of EnKF

4.5 5 5.5 6 6.5 7 7.5 8−8.5

−8

−7.5

−7

−6.5

−6

−5.5

−5

log (N)

log

(Err

or)

T=100T=500

0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.50

0.005

0.01

0.015

Standard Deviation of Measurements

Err

or

T=1000

T=500

w.r.t. ensemble size w.r.t. data noise level

0 0.5 1 1.5 2 2.5 3 3.5 4−11.5

−11

−10.5

−10

−9.5

−9

−8.5

−8

−7.5

−7

−6.5

log(Δ T)

log(

Err

or)

EnKFEnSRFqEnSRF−2qEnSRF−3

EnKF

Error Behavior

qEnSRF: EnSRF

combined with deterministic sampling using optimal cubature

GPC Collocation based Kalman

Filter

T=100 T=500 T=1,000 T=1,500

EnKF

(N=100) 5.3×10-3 3.4×10-3 2.3×10-3 1.7×10-3

EnKF

(N=103) 1.5×10-3 1.0×10-3 7.4×10-4 6.2×10-4

EnKF

(N=104) 4.6×10-4 2.8×10-4 2.4×10-4 1.9×10-4

gPC-KF (N=51) 3.5×10-4 1.6×10-4 1.1×10-4 9.0×10-5

gPC-KF (N=100) 1.8×10-4 7.9×10-5 5.6×10-5 4.6×10-5

Errors of assimilated results

• 50 dimensional random space

(Li & Xiu, vol. 197, CMAME 08)

Accuracy Improvement of EnKF

via gPC

• Use cubature –

equally weighted

• Use pseudo-spectral gPC

0

ˆ( , ) ( ) ( ),N

f fN t Z t Zk k

k

u u=

Φ∑

( )0

ˆ ˆ ˆ,Tf f f f

N NN< ≤

⎡ ⎤= = ⎢ ⎥⎣ ⎦∑0 k kk

u u P u u

Analytical expression in Z

Statistics

(Li & Xiu, J. Comput. Phys. 08)

( ) ( )0

ˆ ( ) , 1, , , 1N

f f iN ki

t Z i M M=

= Φ =∑ kk

u u …

( ) ( ) ( ) ( )' ', , 1, ,f f f a a a

N N N N N Ni i i ii M= + = + =u u u u u u …

( ) ,a f fN N N N= + −u u K d Hu 1( )f T f T

N N N−= +K P H HP Η R

( ) ( ) ( )' ' ', 1, ,a f f

N N N Ni i ii M= + =u u K H u …

( ) ( )1 1Tf T f T f T

N N N N

− −⎛ ⎞= + + +⎜ ⎟⎝ ⎠

K P H HP H R HP H R R

GPC Based Ensemble Kalman

Filter

• Ensemble:

• Square-root update:

Mean state update:

Perturbation update:

01 , (0)f f

f fdu ur u u udt A

⎛ ⎞= − − =⎜ ⎟

⎝ ⎠

0 0.2 0.4 0.6 0.8 12

2.1

2.2

2.3

2.4

2.5

2.6

2.7

2.8

Time

True stateMeasurementEstimate

Example: Nonlinear Population Dynamics

1 2 3 4 5 6 7 8 9−11

−10

−9

−8

−7

−6

−5

−4

−3

Degree of the gPC expansion (N)

log(

Err

or)

MeanStandard deviation

4 5 6 7 8 9 10 11 12 13−12

−10

−8

−6

−4

−2

0

Number of quardrature points (Q)lo

g(E

rror

)

MeanStandard deviation

Error Convergence

N=8, Q=10 is sufficient

0 1 2 3 4 5 6−12

−10

−8

−6

−4

−2

0

log(Ensemble size)

log(

Err

or)

Mean (gPC EnSRF)Std (gPC EnSRF)Mean (EnSRF)Std (EnSRF)

Comparison: gPC-KF vs

EnKF

( )dx y xdtdy x y xzdtdz xy zdt

σ

ρ

β

⎧ = −⎪⎪⎪ = − −⎨⎪⎪ = −⎪⎩

0 0 0( , , ) (1.508870, 1.531271, 25.46091)x y z = −

10, 28, 8 / 3σ ρ β= = =

0 2 4 6 8 10 12 14 16 18 20−20

0

20

Time

X

0 2 4 6 8 10 12 14 16 18 20−50

0

50

Time

Y

0 2 4 6 8 10 12 14 16 18 200

20

40

60

Time

Z

Nonlinear System Example: Lorenz Equations

Small deviation in initial condition (0.001 in x0 ) causes large deviation

0 5 10 15 200

0.5

1

1.5

2

2.5

3

Time

||Est

imat

e−T

rue

stat

e||

N=2 (gPC EnSRF)N=3 (gPC EnSRF)EnSRF

Qualitative Comparison: gPC

EnKF

vs

EnKF

• GPC EnKF: Q=53=125• EnKF: ensemble size = 104

Summary

• Point selection is crucial for the efficacy of stochastic collocation

• GPC expansion is much more than a forward UQ method• Bayesian inverse (Marzouk & Xiu, Comm. Comput. Phys, 08)• Kalman

filter for data assimilation (Li & Xiu, CMAME 08; JCP 08)