Monte Carlo filtering and regional sensitivity analysis (RSA)

http://eemc.jrc.ec.europa.eu1

Monte Carlo filtering and

regional sensitivity analysis (RSA)

M. Ratto, IPSC-JRC, European Commission

DYNARE Workshop, CEF2011, San Francisco


Background

• environmental sciences, early 80’s;

• complex numerical and analytical models, based on first principles, conservation laws, … ;

• ill-defined parameters, competing model structures(different constitutive equations, different types of process considered, spatial/temporal resolution, … )

• need to establish magnitude and sources of prediction uncertainty;

• Monte Carlo simulation analyses.


MC filtering and RSA

One of the earliest landmark application of MC simulation: eutrophication in the Peel Inlet, SW Australia.

Regional Sensitivity Analysis developed and employed.

Hornberger G.M., Spear R.C., An approach to the preliminary analysis of environmental systems, J. Environm. Management, 12, 7-18, 1981.

Young, P.C., Parkinson, S. D. and Lees, M. Simplicity out of complexity: Occam’s razor revisited. Journal of Applied Statistics 23, 165-210, 1996.

Young, P.C., Data-based mechanistic modelling, generalised sensitivity and dominant mode analysis. Computer Physics Communication 117, 113-129, 1999.



Two tasks for RSA:

• qualitative definition of the system behaviour

[a set of constraints: thresholds, ceilings, time bounds based on available information on the system];

• binary classification of model outputs based on the specified behaviour definition.

[qualifies a simulation as behaviour (B) if the model output lies within constraints, non-behaviour ( ) otherwise]B



Define a range for k input factors xi [1 ≤ i ≤ k], reflecting uncertainty in parameters and model constituent hypotheses.

Each Monte Carlo simulation is associated to a vector of values of the input factors.

Classifying simulations as either B or , a set of binary elements is defined allowing to distinguish two sub-sets for each xi:

B

)|( Bxi

)|( Bxi


)( BXi

iX

Y

)( BXi

B

B




The Kolmogorov-Smirnov two-sample test (two-sided version) is performed for each factor independently

)|()|(:)|()|(:

1

0

BxfBxfH

BxfBxfH

inim

inim

≠=

Test statistic:

)|()|(sup)(,

BxFBxFxd inimyinm −=

whereF are marginal cumulative probability functions, f are probability density functions



At what significance level α does the computed value of dm,n determine the rejection of Ho?

α is the probability to reject Ho when it is true (i.e. to recognise a factor as important when it is not).

The importance of each parameter is inversely related to this significance level.



Derive the critical level Dα at which the computed value of dm,n determines the rejection of Ho (the smaller α, the higher Dα).

If dm,n > Dα, then Ho is rejected at significance level α.



0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

Xi

F(X

i)

PriorF

m(X

i|B)

Fn(X

i|B)

dm,n


RSA: comment

RSA has many ‘global’ properties:

The whole range of value of input factors is considered, all factors are varied at the same time.

CAVEAT:

RSA classification analyses univariate marginal distributions.

Lacks for interaction structure.


Mapping stability: example (1)

Phillips curve:

GDP y, inflation i, output gap c

titytttt

tcttt

tptt

ttt

actiEii

acAcAc

app

cpy

,11

,22

1

,1

)1(

)/2cos(2

++−+⋅=+−=

+=+=

+−

−−

−

ωωτπ



Stability is assured if

5.0>ω



0 0.5 10

5

10

15

20A

0 0.5 10

5

10

15

20ω

0 50 100 1500

5

10

15

20τ

0 0.5 10

0.2

0.4

0.6

0.8

1

x

F(x

)

A. K-S prob 1

0 0.5 10

0.2

0.4

0.6

0.8

1

x

F(x

)

ω. K-S prob 4.5031e-221

0 50 100 1500

0.2

0.4

0.6

0.8

1

x

F(x

)

τ. K-S prob 1



Phillips curve, extended version:

GDP y, inflation i, output gap c

tityttftbt

tcttt

tptt

ttt

actiEii

acAcAc

app

cpy

,11

,22

1

,1

)/2cos(2

+++⋅=+−=

+=+=

+−

−−

−

ωωτπ



Stability is assured if

1<+ fb ωω



0 0.5 10

5

10

15

20A

0 0.5 10

10

20

30

40

ωb

0 0.5 10

10

20

30

40

ωf

0 50 100 1500

5

10

15

20τ

0 0.5 10

0.2

0.4

0.6

0.8

1

x

F(x

)

A. K-S prob 1

0 0.5 10

0.2

0.4

0.6

0.8

1

x

F(x

)

ωb. K-S prob 1.0423e-054

0 0.5 10

0.2

0.4

0.6

0.8

1

x

F(x

)

ωf. K-S prob 2.2889e-057

0 50 100 1500

0.2

0.4

0.6

0.8

1

x

F(x

)

τ. K-S prob 0.99541



0 0.5 10

0.5

1

ωb

ω fcc = -0.50539

Bivariate analysis



• As the complexity of the model structure and its parameterization increases, it is not trivial to know a priory the set of model coefficients assuring stability;


Mapping stability : example (3)

• The Lubik Schorfheide (2005) model

• 12 parameters;


Lubik Schorfheide model (2005)


Toolbox documentation


Mapping the fit

How do parameters adjust to fit the multivariate set of observed covariates?

Are there trade-off’s?


Mapping the fit

- Sample structural coefficients prior distributions (or ranges);

OR

- Use Metropolis posterior sample;

Compute RMSE’s of the 1-step ahead model predictions for each of the N observed series.


Mapping the fit

N filtering rules: defining as B the

samples corresponding to the best 10% RMSE’s, -B otherwise.

For each parameter we get N ‘behavioural’ sets:

fj(Xi|B); j = 1,…, N


Mapping the fit

Trade off occurs when:

1) one structural parameter drives significantly the fit of more than one observed series;

AND

2) fj(Xi|B)≠ fk(Xi|B)


Mapping the fit: prior

0 100 200 300 4000

0.2

0.4

0.6

0.8

1e_R

0 200 400 600 8000

0.2

0.4

0.6

0.8

1e_q

0 100 200 300 4000

0.2

0.4

0.6

0.8

1e_A

0 100 200 300 4000

0.2

0.4

0.6

0.8

1e_ys

0 200 400 6000

0.2

0.4

0.6

0.8

1e_pies

0 2 4 6 80

0.2

0.4

0.6

0.8

1psi1

basey

obs

Robs

pieobs

dqde



0 0.5 1 1.5 20

0.2

0.4

0.6

0.8

1psi2

0 0.5 1 1.5 20

0.2

0.4

0.6

0.8

1psi3

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1rho_R

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1alpha

0 5 10 150

0.2

0.4

0.6

0.8

1rr

0 1 2 3 40

0.2

0.4

0.6

0.8

1k

basey

obs

Robs

pieobs

dqde



0 1 2 30

0.2

0.4

0.6

0.8

1tau

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1rho_q

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1rho_A

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1rho_ys

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1rho_pies

basey

obs

Robs

pieobs

dqde


Mapping the fit: posterior

0.2 0.3 0.4 0.5 0.6 0.70

0.2

0.4

0.6

0.8

1

σR

1 1.2 1.4 1.6 1.8 20

0.2

0.4

0.6

0.8

1

σq

0.4 0.6 0.8 1 1.2 1.40

0.2

0.4

0.6

0.8

1

σz

0 1 2 3 40

0.2

0.4

0.6

0.8

1

σy*

1.4 1.6 1.8 2 2.20

0.2

0.4

0.6

0.8

1

σπ*

1 2 3 40

0.2

0.4

0.6

0.8

1

ψ1

basey

R

π∆q

∆e

basey

R

π∆q

∆e

basey

R

π∆q

∆e

base

y

R

π∆q

∆e

base

y

R

π∆q

∆e

base

y

R

π∆q

∆e


0 0.2 0.4 0.6 0.80

0.2

0.4

0.6

0.8

1

ψ2

0 0.2 0.4 0.6 0.80

0.2

0.4

0.6

0.8

1

ψ3

0.5 0.6 0.7 0.8 0.90

0.2

0.4

0.6

0.8

1

ρR

0 0.1 0.2 0.3 0.40

0.2

0.4

0.6

0.8

1α

0 2 4 6 80

0.2

0.4

0.6

0.8

1r

0 0.5 1 1.50

0.2

0.4

0.6

0.8

1κ

basey

R

π∆q

∆e

basey

R

π∆q

∆e

basey

R

π∆q

∆e

base

y

R

π∆q

∆e

base

y

R

π∆q

∆e

base

y

R

π∆q

∆e



0 0.2 0.4 0.6 0.80

0.2

0.4

0.6

0.8

1τ

0 0.2 0.4 0.6 0.80

0.2

0.4

0.6

0.8

1

ρq

0.4 0.5 0.6 0.70

0.2

0.4

0.6

0.8

1

ρz

0.8 0.85 0.9 0.95 10

0.2

0.4

0.6

0.8

1

ρy*

0 0.2 0.4 0.6 0.80

0.2

0.4

0.6

0.8

1

ρπ*

basey

R

π∆q

∆e

basey

R

π∆q

∆e

base

y

R

π∆q

∆e

base

y

R

π∆q

∆e

base

y

R

π∆q

∆e




psi1-psi3 conflict: an interpretation?


RSA: problems

Correlation and interaction structures of the B subset [⇒ also Beck’s review, 1987]:

• Smirnov test is a sufficient test only if Hois rejected

(i.e. the factor is IMPORTANT);

• any covariance structure induced by the classification is not detected by the univariate dm,n statistic.

e.g. factors combined as products or quotients may compensate

• bivariate correlation analysis is not revealing, either.


RSA problems: example (1)

Example 1. Y = X1*X2, X1,X2 ∼U[-0.5,0.5]

Behavioural runs: Y>0.

-0.5 0 0.5-0.5

0

0.5

X1

X2 Smirnov test fails

Correlation analysis would help (e.g. PCA)

Scatter plot of the B subset in the X1,X2 plane



Example 1. Cumulative distributions of X1

(Behavioural runs: Y>0.)

-0.5 0 0.50

0.2

0.4

0.6

0.8

1

X1

F(X

1)

originalB sampleB- sample



In this case, a correlation analysis would be helpful.

the correlation coefficient of (X1, X2) for the B set is ρ=0.75.

This suggests performing a Principal Component Analysis on the B set, obtaining the two components (the eigenvectors of the correlation matrix):



PC1 = (0.7079, 0.7063); accounting for the 87.5% of the variation of the B set.

PC2 = (-0.7063, 0.7079); accounting for the 12.5% of the variation of the B set.

The direction of the principal component PC1 (associated with the highest eigenvalue) indicates the privileged orientation for acceptable runs.



-0.5 0 0.5-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

PC1

PC

2

B subsetB subset



-0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.80

0.5

1

PC1

cdf(P

C1)

-0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.80

0.5

1

PC2

cdf(P

C2)PriorB subsetB subset

PriorB subsetB subset



Now the level of significance for rejecting the

null hypothesis when it is true is very small

(α<0.1%), implying a very strong relevance of the linear combinations of the two input factors,

defined by the principal component analysis



Example 2. Y = X12 + X2

2, X1,X2 ∼U[-0.5,0.5]

Behavioural runs: [0.2 < Y < 0.25]

-0.5 0 0.5-0.5

0

0.5

X1

X2

Smirnov test fails

Correlation fails as well

(sample ρ≈-0.04)

Scatter plot of the B subset in the X1,X2 plane


RSA problems (ctd.)

To address the RSA limitations and to better understand the impact of uncertainty and interaction in the high-dimensional parameter spaces of models, Spear et al. (1994) developed the computer intensive tree-structured density estimation technique (TSDE),

Interesting applications of TSDE in environmental sciences can be found in Spear (1997), Grieb at al. (1999) and Osidele and Beck (2001).

Monte Carlo filtering and regional sensitivity analysis (RSA)

Documents

Transcript of Monte Carlo filtering and regional sensitivity analysis (RSA)