Peter Reichert Eawag Dübendorf, ETH Zürich and SAMSI

Eawag: Swiss Federal Institute of Aquatic Science and Technology

Analyzing input and structural uncertainty of deterministic models with stochastic, time-dependent parameters

Peter Reichert

Eawag Dübendorf, ETH Zürich and SAMSI

SAMSI meetingFeb. 1, 2007

Contents

Motivation

Approach

Implementation

Results for epidem. Model

Results for hydrol. model

Discussion

Motivation

Approach

Implementation[Tomassini, Reichert, Künsch and Borsuk, 2007, subm.]

Results for a simple epidemiological model[Cintron-Arias, Reichert, Lloyd, Banks, 2007, initiated]

Results for a simple hydrologic model[Reichert and Mieleitner, 2007, in progress]

Discussion


Motivation

Motivation

Motivation

Approach

Implementation



Discussion


Motivation

Problem: In most applications, there is a bias in results of

deterministic models. This bias is typically caused by input and model structure

errors, and not by correlated measurement errors. Bias modelling describes this bias statistically, but does

not directly support the identification of its causes and the formulation of improved model structures.

Objective: Search for improved model structures or stochastic

influence factors to model input or parameters that reduce/remove the bias.

Motivation

Approach

Implementation



Discussion


Approach

Approach

Motivation

Approach

Implementation



Discussion


Approach

The proposed approach consists of the following steps:

1) Replace selected (constant) parameters of the deterministic model by a stochastic process in time.

2) Estimate the states of this process jointly with the model parameters.

3) Analyze the degree of bias reduction achieved by doing this for different parameters.

4) Analyze the identified time dependence of the parameters for correlation with external influence factors and internal model variables.

5) Improve the deterministic model if such dependences were found.

6) Explain the remaining stochasticity (if any) of the model by including the appropriate stochastic parameter(s) into the model description.

Motivation

Approach

Implementation



Discussion


Implementation

Implementation

Motivation

Approach

Implementation



Discussion


Model

)(),(),( ODDMM yxyxy

O

DM

Deterministc model:

),( DD xy

Consideration of observation error:

),( MM xyf

x D

yD

O

y

x M

yM

Motivation

Approach

Implementation



Discussion


Model

)(,,, P,MMMMM

iti,-ixyxy (i)(i)

i

,-i(i)

P

M

M

Model with parameter i time dependent:

iti

tii fxyfxyf (i)(i) P,MP,M,MMMM

,,,

x M,i

yM

tM,-i

P

Motivation

Approach

Implementation



Discussion


Time Dependent Parameter

This has the advantage that we can use the analytical solution:

)(d)()(d tWtt W

The time dependent parameter is modelled by a mean-reverting Ornstein Uhlenbeck process:

)(2

2)( 1

2,)(N~ stWstsst ee

or, after reparameterization:

stst

sst ee2

2 1,)(N~

),,,(2

,1

iniP

22

tW

Motivation

Approach

Implementation



Discussion


Inference

We combine the estimation of

constant model parameters, , with

state estimation of the time-dependent parameter(s), , and with

the estimation of (constant) parameters of the Ornstein-Uhlenbeck process(es) of the time dependent parameter(s), .

,-iM

ti,M

iP

Motivation

Approach

Implementation



Discussion


Inference

Gibbs sampling for the three different types of parameters. Conditional distributions:

tiii

tii

itii θxyfθfyxθfyxθf i ,M,MM,M,M,MP,M,M ,,)(,,,,, )(

iti

iti

itii

i fffyxf P,MPP,MP,M,MP )(,,,

tii

iti

ii

ti xyffyθxf i ,M,MMP,MPP,M,M ,,,,, )(

Ornstein-Uhlenbeck process (cheap)

simulation model (expensive)

simulation model (expensive)

Ornstein-Uhlenbeck process (cheap)

x M,i

yM

tM,-i

P

Motivation

Approach

Implementation



Discussion


Inference

Metropolis-Hastings sampling for each type of parameter:

Multivariate normal jump distributions for the parameters and . This requires one simulation to be performed per suggested new value of .

The discretized Ornstein-Uhlenbeck parameter, , is split into subintervals for which OU-process realizations conditional on initial and end points are sampled. This requires the number of subintervals simulations per complete new time series of .

tiii

tii

itii θxyfθfyxθfyxθf i ,M,MM,M,M,MP,M,M ,,)(,,,,, )(

tii

iti

ii

ti xyffyθxf i ,M,MMP,MPP,M,M ,,,,, )(

iti

iti

itii

i fffyxf P,MPP,MP,M,MP )(,,,

ti,M

ti,M

Motivation

Approach

Implementation



Discussion


Estimation of Hyperparametersby Cross - Validation

Due to identifiability problems we select the two hyperparameters () by cross-validation:

),(max),,(log:),( i

ii yyfpsl

Motivation

Approach

Implementation



Discussion


Estimation of Hyperparametersby Cross - Validation

For a state-space model of the form

2

)(1

1)(

12/112/1

)(

)(

2)(

exp)(1

)2(

1

d),,(),(

11

),,(

ij

kjj

ii

ijkii

ii

kii

ii

ii

yyn

yfyyf

yyf

we can estimate the pseudo-likelihood from the sample:

),,(d

d:),,( DD xF

txt

)(),,( O,D iyii xty

),0(N y

Motivation

Approach

Implementation



Discussion


Results for Epidemiological Model

Results for a Simple Epidemiological Model

Model

Model Application

Preliminary Results

Motivation

Approach

Implementation



Discussion


Model

A Simple Epidemiological Model:

N

SIt

L

ISN

t

S)(

d

d

D

I

N

SIt

t

I )(

d

d

)(2cos(1)( 010 ttt

Dushoff et al. 2004

1

S: susceptibleI: infectedN: „total population“N-S-I: resistant

2

3

4 5 6

6 model parameters2 initial conditions1 standard dev. of

meas. err.

Motivation

Approach

Implementation



Discussion


Model Application

Influenza Outbreaks in the USA Motivation

Approach

Implementation



Discussion


Preliminary Results

Model predictions and data

1.0 1.5 2.0 2.5 3.0 3.5 4.0

01

02

03

04

0

constant parameters

time

I_n

ew

_in

t

1.0 1.5 2.0 2.5 3.0 3.5 4.0

01

02

03

04

0

beta0 time varying

time

I_n

ew

_in

t

Motivation

Approach

Implementation



Discussion


Preliminary Results

20 40 60 80 120

-3-1

01

23

Sequence of Residuals

Data Points

No

rma

lize

d R

es

idu

als

0 5 10 15 20 25 30 35

-3-1

01

23

Residuals vs. Observed

Observed

No

rma

lize

d R

es

idu

als

Hist. of Residuals

Normalized Residuals

De

ns

ity

-3 -2 -1 0 1 2 3

0.0

0.2

0.4

-3 -2 -1 0 1 2 3

-3-1

01

23

Sample vs. Normal Quant.

Normal Quantiles

Sa

mp

le Q

ua

nti

les

Residual analysis – constant parameters

Motivation

Approach

Implementation



Discussion


Preliminary Results

Residual analysis – beta0 time varying

20 40 60 80 120

-2-1

01

2

Sequence of Residuals

Data Points

No

rma

lize

d R

es

idu

als

0 5 10 15 20 25 30 35

-2-1

01

2

Residuals vs. Observed

Observed

No

rma

lize

d R

es

idu

als

Hist. of Residuals

Normalized Residuals

De

ns

ity

-2 -1 0 1 2

0.0

0.1

0.2

0.3

0.4

-3 -2 -1 0 1 2 3

-3-2

-10

12

3

Sample vs. Normal Quant.

Normal Quantiles

Sa

mp

le Q

ua

nti

les

Motivation

Approach

Implementation



Discussion


Preliminary Results

Time dependent parameter:

1 2 3 4

10

02

00

30

0

time series of beta_0

time

be

ta_

0

1 2 3 4

10

02

00

30

0

time series of beta_0

time

be

ta_

0

Motivation

Approach

Implementation



Discussion


Next Steps

More Fourier terms, more general periodicity.

Cross validation for choice of and . Interpretation of remaining stochasticity.

Inclusion of identified forcing in model equations.

Inclusion of stochastic parameter in model equations.

Different data set: measles.

Motivation

Approach

Implementation



Discussion


Preliminary for Hydrologic Model

Results for a Simple Hydrologic Model

Model

Model Application

Preliminary Results

Motivation

Approach

Implementation



Discussion


Model

A Simple Hydrologic Watershed Model (1):

gwlatetrunoffrains )(

d

dqqqqq

t

h

bfgwgw

d

dqq

t

h

rbflatrunoffr

d

dqqqq

t

h

Kuczera et al. 2006

Motivation

Approach

Implementation



Discussion


Model

A Simple Hydrologic Watershed Model (2):

)(rainrain trainfq

)(rainsatrunoff trainffq

)()exp(1 petet tpetfhkq set

maxlat,satlat qfq

gwbfbf hkq

maxgw,satgw qfq

rrr hkq

rwQr qAfQ

1

1

)exp(1

1

FssFsat

shks

f

Kuczera et al. 2006

1

2

3 4

5

6

7

A

B

7 model parameters3 initial conditions1 standard dev. of meas. err.3 „modification parameters“

C

Motivation

Approach

Implementation



Discussion


Model Application

Model application:

Data set of Abercrombie watershed, New South Wales, Australia (2770 km2), kindly provided by George Kuczera (Kuczera et al. 2006).

Box-Cox transformation applied to model and data to decrease heteroscedasticity of residuals.

Step function input to account for input data in the form of daily sums of precipitation and potential evapotranspiration.

Daily averaged output to account for output data in the form of daily average discharge.

Motivation

Approach

Implementation



Discussion


Model Application

Prior distribution:

Estimation of constant parameters:

Independent uniform distributions for the loga-rithms of all parameters (7+3+1=11), keeping correction factors (frain, fpet, fQ) equal to unity.

Estimation of time-dependent parameters:

Ornstein-Uhlenbeck process applied to log of the parameter.Hyper-parameters: = 5d, fixed, only estimation of initial value and mean (0 for log frain, fpet, fQ). Constant parameters as above.

Motivation

Approach

Implementation



Discussion


Preliminary Results

Posterior marginals:

0.2 0.4 0.6

13

5

h_s_ini

5 7 9 11

0.1

0.3

h_gw _ini

0.0 1.0 2.0

0.2

0.6

1.0

h_r_ini

0.015 0.025

20

60

10

0

k_s

100 300

0.0

01

0.0

05

s_F

0.010 0.016

50

15

0

k_et

1.0 2.0

0.5

1.5

q_lat_max

4 5 6 7

0.2

0.6

q_gw _max

0e+00 3e-04

10

00

30

00

50

00

k_bf

0.5 0.7 0.9

13

5

k_r

1.00 1.15

24

68

sd_Q_trans

Motivation

Approach

Implementation



Discussion


Preliminary Results

Max. post. simulation with constant parameters:

400 500 600 700 800 900 1000 1100

05

01

00

15

02

00

t [days since 31/12/1971]

Q [m

3/s

]

Motivation

Approach

Implementation



Discussion


Preliminary Results

Residuals of max. post. sim. with const. pars.:

500 600 700 800 900 1000

-20

00

20

0


resi

du

als

[m3

/s]

NS = 0.64se = 20

500 600 700 800 900 1000

-6-2

26


resi

d. o

f tr.

ou

tpu

t

NS = 0.64se = 1.1

Motivation

Approach

Implementation



Discussion


Preliminary Results

Residual analysis, max. post., constant parameters

Residual analysis, max. post., q_gw_max time-dependent

500 700 900

-6-2

26


resi

d. o

f tr.

ou

tpu

t

NS = 0.64se = 1.1

0 5 10 15 20 25

0.0

0.4

0.8

Lag

AC

F

500 700 900

-50

5


resi

d. o

f tr.

ou

tpu

t

NS = 0.69se = 0.99

0 5 10 15 20 25

0.0

0.4

0.8

Lag

AC

F

Motivation

Approach

Implementation



Discussion


Preliminary Results

Residual analysis, max. post., s_F time-dependent

Residual analysis, max. post., f_rain time-dependent

500 700 900

-3-1

13


resi

d. o

f tr.

ou

tpu

t

NS = 0.88se = 0.6

0 5 10 15 20 25

0.0

0.4

0.8

Lag

AC

F

500 700 900

-3-1

13


resi

d. o

f tr.

ou

tpu

t

NS = 0.9se = 0.67

0 5 10 15 20 25

0.0

0.4

0.8

Lag

AC

F

Motivation

Approach

Implementation



Discussion


Preliminary Results

Residual analysis, max. post., f_pet time-dependent

Residual analysis, max. post., f_Q time-dependent

500 700 900

-6-2

26


resi

d. o

f tr.

ou

tpu

tNS = 0.79se = 0.83

0 5 10 15 20 25

0.0

0.4

0.8

Lag

AC

F

500 700 900

-40

24


resi

d. o

f tr.

ou

tpu

t

NS = 0.83se = 0.72

0 5 10 15 20 25

0.0

0.4

0.8

Lag

AC

F

Motivation

Approach

Implementation



Discussion


Preliminary Results

Residual analysis, max. post., k_et time-dependent

Residual analysis, max. post., q_latmax time-dependent

500 700 900

-6-2

26


resi

d. o

f tr.

ou

tpu

tNS = 0.67se = 1.1

0 5 10 15 20 25

0.0

0.4

0.8

Lag

AC

F

500 700 900

-50

5


resi

d. o

f tr.

ou

tpu

t

NS = 0.67se = 1.0

0 5 10 15 20 25

0.0

0.4

0.8

Lag

AC

F

Motivation

Approach

Implementation



Discussion


Preliminary Results

Residual analysis, max. post., k_s time-dependent

500 700 900

-4-2

02

4


resi

d. o

f tr.

ou

tpu

t

NS = 0.87se = 0.64

0 5 10 15 20 25

0.0

0.4

0.8

Lag

AC

F

NS parameter NS parameter

0.90 f_rain 0.79 f_pet

0.88 s_F 0.69 q_gwmax

0.87 k_s 0.67 q_latmax

0.83 f_Q 0.67 k_et

const. parameter: 0.64

Motivation

Approach

Implementation



Discussion


Preliminary Results

Time-dependent parameters

400 500 600 700 800 900 1000 11000

.61

.01

.41

.8

time [days since 31/12/1971]

fra

in

400 500 600 700 800 900 1000 1100

20

08

00

14

00


sF

400 500 600 700 800 900 1000 1100

0.0

15

0.0

30


ks

Motivation

Approach

Implementation



Discussion


Preliminary Results

Time-dependent parameters

400 500 600 700 800 900 1000 11000

.81

.21

.62

.0


fQ

400 500 600 700 800 900 1000 1100

0.6

1.0

1.4

1.8


fpe

t

Motivation

Approach

Implementation



Discussion


Preliminary Results

Scatter plot of parameter vs model variables

Motivation

Approach

Implementation



Discussion

50 200 350

0.6

1.2

1.8

Q

frai

n

20 60

0.6

1.2

1.8

h_s

frai

n

100 300

0.6

1.2

1.8

h_gw

frai

n

0.5 1.5

0.6

1.2

1.8

h_r

frai

n

0.05 0.15

0.6

1.2

1.8

f_sat

frai

n

0 20 40 60

0.6

1.2

1.8

q_rain

frai

n

1 2 3 4

0.6

1.2

1.8

q_et

frai

n

0 1 2 3 4 5

0.6

1.2

1.8

q_runoff

frai

n

0.5 2.0 3.5

0.6

1.2

1.8

q_lat

frai

n

1 3 5 7

0.6

1.2

1.8

q_gw

frai

n

0.02 0.08 0.14

0.6

1.2

1.8

q_bf

frai

n

2 4 6 8

0.6

1.2

1.8

q_r

frai

n


Preliminary Results


Motivation

Approach

Implementation



Discussion

50 150 250

200

800

1400

Q

sF

50 150

200

800

1400

h_s

sF

50 150 250

200

800

1400

h_gw

sF

2 4 6 8

200

800

1400

h_r

sF

0.1 0.3

200

800

1400

f_sat

sF

0 20 40 60

200

800

1400

q_rain

sF

1 2 3 4 5

200

800

1400

q_et

sF

0 2 4 6 8

200

800

1400

q_runoff

sF

0.2 0.6

200

800

1400

q_lat

sF

0.5 1.5

200

800

1400

q_gw

sF

0.02 0.08

200

800

1400

q_bf

sF

2 4 6

200

800

1400

q_r

sF


Preliminary Results


Motivation

Approach

Implementation



Discussion

50 150

0.01

50.

035

Q

ks

50 150

0.01

50.

035

h_s

ks

50 150 250

0.01

50.

035

h_gw

ks

2 4 6 8

0.01

50.

035

h_r

ks

0.1 0.3 0.5

0.01

50.

035

f_sat

ks

0 20 40 60

0.01

50.

035

q_rain

ks

1 2 3 4 5

0.01

50.

035

q_et

ks

0 2 4 6 8

0.01

50.

035

q_runoff

ks

0.2 0.6

0.01

50.

035

q_lat

ks

0.5 1.5 2.5

0.01

50.

035

q_gw

ks

0.02 0.08

0.01

50.

035

q_bf

ks

1 3 5 7

0.01

50.

035

q_r

ks


Preliminary Results


Motivation

Approach

Implementation



Discussion

50 150

0.8

1.4

2.0

Q

fQ

50 150

0.8

1.4

2.0

h_s

fQ

100 300

0.8

1.4

2.0

h_gw

fQ

1 3 5 7

0.8

1.4

2.0

h_r

fQ

0.05 0.20

0.8

1.4

2.0

f_sat

fQ

0 20 40 60

0.8

1.4

2.0

q_rain

fQ

1 2 3 4

0.8

1.4

2.0

q_et

fQ

0 2 4 6

0.8

1.4

2.0

q_runoff

fQ

0.1 0.3 0.5

0.8

1.4

2.0

q_lat

fQ

0.5 1.5

0.8

1.4

2.0

q_gw

fQ

0.01 0.03 0.05

0.8

1.4

2.0

q_bf

fQ

1 2 3 4 5

0.8

1.4

2.0

q_r

fQ


Preliminary Results


Motivation

Approach

Implementation



Discussion

50 150

0.6

1.2

1.8

Q

fpet

20 60 120

0.6

1.2

1.8

h_s

fpet

50 150

0.6

1.2

1.8

h_gw

fpet

1 2 3 4 5

0.6

1.2

1.8

h_r

fpet

0.1 0.3

0.6

1.2

1.8

f_sat

fpet

0 20 40 60

0.6

1.2

1.8

q_rain

fpet

1 2 3 4 5

0.6

1.2

1.8

q_et

fpet

0 2 4 6 8

0.6

1.2

1.8

q_runoff

fpet

0.2 0.8 1.4

0.6

1.2

1.8

q_lat

fpet

0.5 1.5

0.6

1.2

1.8

q_gw

fpet

0.02 0.06 0.10

0.6

1.2

1.8

q_bf

fpet

1 3 5 7

0.6

1.2

1.8

q_r

fpet


Next Steps

Search for correlations between time dependent parameters and external influence factors and system variables.

Improve formulation of deterministic model and redo the analysis.

Cross validation for choice of and . Inclusion of stochastic parameter in model

equations to explain remaining stochasticity.

Motivation

Approach

Implementation



Discussion


Discussion

Discussion

Motivation

Approach

Implementation



Discussion


Discussion

Discussion

1. Other formulations of time-dependent parameters?

2. Dependence on other factors than time?

3. How to estimate hyperparameters? (Reduction in correlation time always improves the fit.)

4. How to avoid modelling physical processes with the bias term?

5. Learn from more applications.

6. Compare results with methodology by Bayarri et al. (2005). Combine/extend the two methodologies?

7. How to improve efficiency?

8. How to combine statistical with contextual knowledge?

9. ?

Motivation

Approach

Implementation



Discussion

Eawag: Swiss Federal Institute of Aquatic Science and Technology

Thank you for your attention

Peter Reichert Eawag Dübendorf, ETH Zürich and SAMSI

Documents

Transcript of Peter Reichert Eawag Dübendorf, ETH Zürich and SAMSI