Using Small Ensembles in High Dimensions: Hierarchical ...

7/17/08

h Dimensions:roaches

Filters

tion (DAReS)

u, Kevin Raeder

Anderson: Joint Stats Meeting, Denver, 4 August, 2008 1

Using Small Ensembles in HigHierarchical Bayesian App

to Adaptive Ensemble

Jeffrey AndersonIMAGe Data Assimilation Research Sec

Thanks to Nancy Collins, Tim Hoar, Hui Li

7/17/08

ata Assimilation

re)ilable.


How an Ensemble Filter Works for Geophysical D

Ensemble stateestimate after usingprevious observation(analysis).

Ensemble state attime of next obser-vation (prior).

tk tk+1

1. Use model to advanceensemble (3 members heto time at which next observation becomes ava

****

7/17/08

ata Assimilation

=h(x), byember.

servationsments withed errors canequentially.



2. Get prior ensemble sample of observation, yapplying forward operator h to each ensemble m

Theory: obfrom instruuncorrelatbe done s

y

****

h hh

7/17/08

ata Assimilation

ibution



3. Getobserved valueandobservational error distrfrom observing system.

y

****

h hh

7/17/08

ata Assimilation

mbleation errors).

y

ce betweenrs of ensem-imarily increment.



4. Findincrement for each prior observation ense(this is a scalar problem for uncorrelated observ

y

****

h hh Note: Differen

different flavoble filters is probservation in

7/17/08

ata Assimilation

ariable to linearlyble increments.

y

impact oftion increments onte variable can be independently!



5. Use ensemble samples of y and each state vregress observation increments onto state varia

y

****

h hh

Theory:observaeach stahandled

7/17/08

ata Assimilation

variable are updated,bservation...

y

tk+2



6. When all ensemble members for each state have a new analysis. Integrate to time of next o

y

****

h hh

tk

7/17/08

le Model

latitude band.

35 40


Ensemble Filter for Lorenz-96 40-Variab

40 state variables: X1, X2,..., X40.

dXi / dt = (Xi+1 - Xi-2)Xi-1 - Xi + F.Acts ‘something’ like synoptic weather around a

5 10 15 20 25 30−10

−5

0

5

10

time 91

State Variable

7/17/08

le Model

latitude band.

35 40





5 10 15 20 25 30−10

−5

0

5

10

time 92

State Variable

7/17/08

le Model

latitude band.

35 40





5 10 15 20 25 30−10

−5

0

5

10

time 93

State Variable

7/17/08

le Model

latitude band.

35 40





5 10 15 20 25 30−10

−5

0

5

10

time 94

State Variable

7/17/08

le Model

latitude band.

35 40





5 10 15 20 25 30−10

−5

0

5

10

time 95

State Variable

7/17/08

le Model

latitude band.

35 40





5 10 15 20 25 30−10

−5

0

5

10

time 96

State Variable

7/17/08

le Model

latitude band.

35 40





5 10 15 20 25 30−10

−5

0

5

10

time 97

State Variable

7/17/08

le Model

latitude band.

35 40





5 10 15 20 25 30−10

−5

0

5

10

time 98

State Variable

7/17/08

le Model

latitude band.

35 40





5 10 15 20 25 30−10

−5

0

5

10

time 99

State Variable

7/17/08

le Model

latitude band.

35 40





5 10 15 20 25 30−10

−5

0

5

10

time 100

State Variable

7/17/08

bations

0.h’.

35 40ensembleensembleensembleensembleensembleensembleensembleensembleensembleensembleensembleensembleensembleensembleensembleensembleensembleensembleensembleensemble


Lorenz-96 is sensitive to small pertur

Introduce 20 ‘ensemble’ state estimates.Each is slightly perturbed for each Xi at time 10Refer to unperturbed control integration as ‘trut

5 10 15 20 25 30−10

−5

0

5

10

time 102 truth

State Variable

time 102 truthtime 102 truthtime 102 truthtime 102 truthtime 102 truthtime 102 truthtime 102 truthtime 102 truthtime 102 truthtime 102 truthtime 102 truthtime 102 truthtime 102 truthtime 102 truthtime 102 truthtime 102 truthtime 102 truthtime 102 truthtime 102 truth

7/17/08

bations

0.h’.





5 10 15 20 25 30−10

−5

0

5

10

time 104 truth

State Variable


7/17/08

bations

0.h’.





5 10 15 20 25 30−10

−5

0

5

10

time 106 truth

State Variable


7/17/08

bations

0.h’.





5 10 15 20 25 30−10

−5

0

5

10

time 108 truth

State Variable


7/17/08

bations

0.h’.





5 10 15 20 25 30−10

−5

0

5

10

time 110 truth

State Variable


7/17/08

bations

0.h’.





5 10 15 20 25 30−10

−5

0

5

10

time 112 truth

State Variable


7/17/08

bations

0.h’.





5 10 15 20 25 30−10

−5

0

5

10

time 114 truth

State Variable


7/17/08

bations

0.h’.





5 10 15 20 25 30−10

−5

0

5

10

time 116 truth

State Variable


7/17/08

bations

0.h’.





5 10 15 20 25 30−10

−5

0

5

10

time 118 truth

State Variable


7/17/08

bations

0.h’.





5 10 15 20 25 30−10

−5

0

5

10

time 120 truth

State Variable


7/17/08

ed stations each step.

station location. from N(0, 1) to each..

35 40le obsle obsle obsle obsle obsle obsle obsle obsle obsle obsle obsle obsle obsle obsle obsle obsle obsle obsle obsle obs


Assimilate ‘observations’ from 40 randomly locat

Observations generated by interpolating truth toSimulate observational error: Add random drawStart from ‘climatological’ 20-member ensemble

5 10 15 20 25 30−10

−5

0

5

10

time 201 truth ensemb

State Variable

time 201 truth ensembtime 201 truth ensembtime 201 truth ensembtime 201 truth ensembtime 201 truth ensembtime 201 truth ensembtime 201 truth ensembtime 201 truth ensembtime 201 truth ensembtime 201 truth ensembtime 201 truth ensembtime 201 truth ensembtime 201 truth ensembtime 201 truth ensembtime 201 truth ensembtime 201 truth ensembtime 201 truth ensembtime 201 truth ensembtime 201 truth ensemb

7/17/08







5 10 15 20 25 30−10

−5

0

5

10


State Variable


7/17/08


.


dent and right!



This isn’t working very well.Ensemble spread is reduced, but...,Ensemble is inconsistent with truth most places

5 10 15 20 25 30−10

−5

0

5

10


State Variable


ConfiConfident and WRONG.

7/17/08

Filters

y

tk+2

pling Error;ian Assumption

Error;inearelation


Some Error Sources in Ensemble

y

****

h hh

tk

1. Model Error

2. h errors;Representativeness

4. SamGauss

5. SamplingAssuming LStatistical R

3. ‘Gross’ Obs. Errors

7/17/08

gh sampling error.

hows expectedte value of sample

ation vs. trueation.

ted obs. reduced, increase error.

with localization.

let obs. impactted state.


Observations impact unrelated state variables throu

Plot sabsolucorrelcorrel

Unrelasprea

Attack

Don’t unrela

0 0.5 10

0.2

0.4

0.6

0.8

1

True Correlation

Exp

ecte

d |S

ampl

e C

orre

latio

n|

10 Members20 Members40 Members80 Members

7/17/08

bservation impact.

35 40le obs

lter

le obsle obsle obsle obsle obsle obsle obsle obsle obsle obsle obsle obsle obsle obsle obsle obsle obsle obsle obs



Lorenz-96 Assimilation with localization of o

5 10 15 20 25 30−10

0

10


Localization from Hierarchical Fi


5 10 15 20 25 30−10

0

10


State Variable

No Localization


7/17/08

bservation impact.

35 40le obs

lter





5 10 15 20 25 30−10

0

10




5 10 15 20 25 30−10

0

10


State Variable

No Localization


7/17/08

bservation impact.with truth.

35 40le obs

lter




Lorenz-96 Assimilation with localization of oEnsemble is much more consistent

5 10 15 20 25 30−10

0

10




5 10 15 20 25 30−10

0

10


State Variable

No Localization


7/17/08

hical filter.maximizes signal.

35 40le obs

lter


35 40

ons


Localization computed by adaptive hierarcA tuning run of 4, 20-member ensembles

5 10 15 20 25 30−10

0

10




5 10 15 20 25 300

0.5

1

State Variable

3 Sample Observation Localizati

7/17/08

ressure Obs. at 20N, 60E

XPERT USERS.

60 80 100

an factor level 3

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 60 80 100

s section at row 18


Localization in GCM can be very complex. Surface P

MUST HAVE ADAPTIVE HELP FOR NON-E

20 40 60 80 100−10

0

10

20

30

40

50

60u mean factor level 1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

20 40 60 80 100−10

0

10

20

30

40

50


0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

20 40−10

0

10

20

30

40

50

60u me

20 40 60 80 100−10

0

10

20

30

40

50


0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

20 40 60 80 100−10

0

10

20

30

40

50


0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

20 40

0.2

0.4

0.6

0.8

1cros

PS to U

7/17/08

Filters

y

tk+2

pling Error;ian Assumption

Error;inearelation


Some Error Sources in Ensemble

y

****

h hh

tk

1. Model Error

2. h errors;Representativeness

4. SamGauss

5. SamplingAssuming LStatistical R

3. ‘Gross’ Obs. Errors

7/17/08

d model error.

model.

10ys)


Assimilating in the presence of simulate

dXi / dt = (Xi+1 - Xi-2)Xi-1 - Xi + F.For truth, use F = 8.In assimilating model, use F = 6.

Time evolution for X1 shown.Assimilating model quickly diverges from ‘true’

0 5−5

0

5

10

model time (pseudo−da

F=8 F=6

7/17/08

d model error





5 10 15 20 25 30−10

−5

0

5

10


State Variable


7/17/08

d model error





This isn’t working again!It will just keep getting worse.

5 10 15 20 25 30−10

−5

0

5

10


State Variable


7/17/08

ance Inflation

=> ‘true’ distribution.ufficient prior variance.

ase spread of prior

.

−1 0

) x+


Model/Filter Error; Filter Divergence and Vari

1. History of observations and physical system 2. Sampling error, some model errors lead to ins

3. Naive solution is Variance inflation: just incre

4. For ensemble member i,

−4 −3 −20

0.5

1

Pro

babi

lity

"TRUE" Prior PDF

Variance Deficient PDF

inflate xi( ) λ xi x–(=

7/17/08

f model error

rithm.

35 40




Assimilating with Inflation in presence oInflation is a function of state variable and time.Automatically selected by adaptive inflation algo

5 10 15 20 25 30

1.2

1.4

1.6inflation

5 10 15 20 25 30−10

0

10


Adaptive State Space Inflation


5 10 15 20 25 30−10

0

10


State Variable

No Inflation


7/17/08

f model error

rithm.

35 40





5 10 15 20 25 30

1.2

1.4

1.6inflation

5 10 15 20 25 30−10

0

10




5 10 15 20 25 30−10

0

10


State Variable

No Inflation


7/17/08

f model error

rithm.rror.

35 40




Assimilating with Inflation in presence oInflation is a function of state variable and time.Automatically selected by adaptive inflation algoIt can work, even in presence of severe model e

5 10 15 20 25 30

1.2

1.4

1.6inflation

5 10 15 20 25 30−10

0

10




5 10 15 20 25 30−10

0

10


State Variable

No Inflation


7/17/08

ltering

observed inconsistency.

.

posed to be unbiased.

2 4

hood

S.D.n

obs2


Adaptive Inflation for Ensemble Fi

1. For observed variable, have estimate of prior-

2. Expected(prior mean - observation) =

Assumes that prior and observation are supIs it model error or random chance?

−4 −2 00

0.2

0.4

0.6

0.8

Pro

babi

lity

Prior PDF

S.D.

Obs. Likeli

Expected Separatio

Actual 4.714 SDs

σprior2 σ+

7/17/08

ltering


.

rior and observation.

2 4

hood

S.D.tion

obs2




2. Expected(prior mean - observation) =

3. Inflating increases expected separation.Increases ‘apparent’ consistency between p

−4 −2 00

0.2

0.4

0.6

0.8

Pro

babi

lity

Prior PDF Obs. Likeli

Inflated S.D.Expected Separa

Actual 3.698 SDs

σprior2 σ+

7/17/08

ltering


2 4

hood

S.D.tion

yo.

or σobs2

+ N 0 θ,( )=

2⁄D2 2θ2⁄–( )exp




Distance, D, from prior mean y to obs. is

Prob. yo is observed givenλ:

−4 −2 00

0.2

0.4

0.6

0.8

Pro

babi

lity

Prior PDF Obs. Likeli

Inflated S.D.Expected Separa

Actual 3.698 SDsy

λ

N 0 λσpri2

,

p yo λ( ) 2Πθ2( )1–

=

7/17/08

ltering

on factor,λ.

.p)



Use Bayesian statistics to get estimate of inflati

Assume prior is gaussian;

0 1 2 3 4 5 60

1

2

Obs. Space Inflation Factor: λ

Prior λ PDF

−1 0 1 2 3 40

0.2

0.4

0.6

Observation: y

Obs. LikelihoodPrior PDF

p λ Yprev( ) N λp σλ,2,(=

7/17/08

ltering

on factor,λ.

e've assumed aaussian for prior

.

ecall that

an be evaluatedrom normal PDF.

.

p λ Yprev( )

p yo λ( )

tion




Wg

R

cf

0 1 2 3 4 5 60

1

2


Prior λ PDF

−1 0 1 2 3 40

0.2

0.4

0.6

Observation: y

Obs. LikelihoodPrior PDF

p λ Yprev yo,( ) p yo λ( ) p λ Yprev( ) normaliza⁄=

7/17/08

ltering

on factor,λ.

et

rom normal PDF.

ultiply by

o get

.

p yo λ 0.75=( )

λ 0.75= Yprev( )

λ 0.75= Yprev˙ yo,( )

tion




G

f

M

t

0 1 2 3 4 5 60

1

2


Prior λ PDF

−1 0 1 2 3 40

0.2

0.4

0.6

Observation: y

Obs. LikelihoodInflated Prior λ = 0.75

p

p


7/17/08

ltering

on factor,λ.

et

rom normal PDF.

ultiply by

o get

.

p yo λ 1.5=( )

p λ 1.5= Yprev( )

p λ 1.5= Yprev˙ yo,( )

tion




G

f

M

t

0 1 2 3 4 5 60

1

2


Prior λ PDF

−1 0 1 2 3 40

0.2

0.4

0.6

Observation: y



7/17/08

ltering

on factor,λ.

et

rom normal PDF.

ultiply by

o get

.

p yo λ 2.2=( )

p λ 2.2= Yprev( )

p λ 2.2= Yprev˙ yo,( )

tion




G

f

M

t

0 1 2 3 4 5 60

1

2


Prior λ PDF

−1 0 1 2 3 40

0.2

0.4

0.6

Observation: y



7/17/08

ltering

on factor,λ.

epeat for a rangef values ofλ.

ow must get pos-erior in same forms prior (gaussian).

.tion




Ro

Nta

0 1 2 3 4 5 60

1

2


Prior λ PDF

Likelihood y observed given λPosterior

−1 0 1 2 3 40

0.2

0.4

0.6

Observation: y

Obs. Likelihood


7/17/08

ltering

on factor,λ.

ery little informa-ion aboutλ in aingle observation.

osterior and priorre very similar.

ormalized poste-ior indistinguish-ble from prior.

.tion




Vts

Pa

Nra

1 20

1

2


Prior λ PDF

Posterior

−1 0 1 2 3 40

0.2

0.4

0.6

Observation: y

Obs. Likelihood


7/17/08

ltering

on factor,λ.

ery little informa-ion aboutλ in aingle observation.

osterior and priorre very similar.

ifference showslight shift to largeralues ofλ.

.tion




Vts

Pa

Dsv

1 2

0.01

0

0.01


λ: Posterior − Prior

Max density shifted to right

−1 0 1 2 3 40

0.2

0.4

0.6

Observation: y

Obs. Likelihood


7/17/08

ltering

on factor,λ.

ne option is to useaussian prior for.

elect max (mode)f posterior asean of updatedaussian.

o a fit for updatedtandard deviation.

.tion




OGλ

SomG

Ds1 2

0

1

2


Prior λ PDFFind Max by search

Max is new λ mean

−1 0 1 2 3 40

0.2

0.4

0.6

Observation: y

Obs. Likelihood


7/17/08

ltering

ically!

6th order poly inθ

lved to give mode.

regressions.



A. Computing updated inflation mean, .

Mode of can be found analyt

Solving leads to

This can be reduced to a cubic equation and so

New is set to the mode.

This is relatively cheap compared to computing

λu

p yo λ( ) p λ Yprev( )

∂ p yo λ( ) p λ Yprev( ) ∂λ 0=⁄

λu

7/17/08

ltering

d point, e.g. .

and .

.

λu σλ p,+

p λu σλ p,+( )

λu σλ p,+ ) p λu( )⁄



A. Computing updated inflation variance,

1. Evaluate numerator at mean and secon

2. Find so goes through

3. Compute as where

σλ u,2

λu

σλ u,2 N λu σλ u,

2,( ) p λu( )

σλ u,2 σλ p,

2 2 rln⁄–= r p(=

7/17/08

tion space.

space inflation?

int distribution.

ss changes inn onto state vari-flation.


State Space Adaptive Inflation

Computations so far adapt inflation for observa

What is relation between observation and state

Have to use prior ensemble observation/state jo

Regreinflatioable in

y

****

h hh

y

tk+2

tk

7/17/08

orithm:

te variable,λs,i.

e impact ofλs,i on

bservation then

ll 12th order:

.duct in this case!


Spatially varying adaptive inflation alg

Have a distribution forλ at each time for each sta

Use prior correlation from ensemble to determinprior variance for given observation.

If γ is correlation between state variable i and o

.

Equation for finding mode of posterior is now fuAnalytic solution appears unlikely.

Can do Taylor expansion ofθ aroundλs,i .

Retaining linear term is normally quite accurateThere is an analytic solution to find mode of pro

θ 1 γ λs i, 1–( )+[ ]2σprior

2σobs

2+=

7/17/08

P

erature, moisture.

huge spread).

inds.

reanalysis.

day.


Adaptive Inflation in Global NW

Model: CAM 3.1 T85L26.

Several million model variables: winds, temp

Initialized from a climatological distribution (

Observations: Radiosondes, ACARS, Satellite W

Subset of observations used in NCAR/NCEP

Several hundred thousand observations per

7/17/08

ce Prior RMS, Spread


Adaptive Inflation in CAM; 500 hPa T Obs. Spa

7/17/08

; 266 hPa U


Adaptive Inflation in CAM after 1-Month

7/17/08

on in CAM

nsely observed regions.

embleto work well!


Adaptive Inflation for Numerical Weather Predicti

1. Largest inflation caused by model error in de

2. RMS reduced, spread increased.

3. Fewer observations rejected.

Combinedwith localization,allows80memberens

7/17/08

lters: Summary

tio to minimize error.

ter system.e of estimate.


Hierarchical Bayesian Methods for Adaptive Fi

1. Localization:Run an ensemble of ensembles.Use regression coefficient signal-to-noise ra

2. Inflation:Use each observation twice.

Once to adjust parameter (inflation) of filSecond time to adjust mean and varianc

7/17/08

ticians.


Conclusions

Lots of cool statistics being done poorly.

Work in small models can give insight.

Applications in large models are important.

Looking for collaborations with interested statis

Using Small Ensembles in High Dimensions: Hierarchical ...

Documents

Transcript of Using Small Ensembles in High Dimensions: Hierarchical ...