Computational statistics 2009 Sensitivity analysis (SA) - outline 1.Objectives and examples 2.Local...

Computational statistics 2009

Sensitivity analysis (SA)- outline

1. Objectives and examples

2. Local and global SA

3. Importance measures involving derivatives or regression coefficients

4. A general variance-based importance measure

5. First-order indices and total indices


Objectives of sensitivity analysis

SA is the study of how the variation in the output of a model can be apportioned to different sources of variation.

JRC web-site:http://sensitivity-analysis.jrc.cec.eu.int/


Examples of questions addressed in SA

Supply-demand models: How sensitive is the predicted petrol consumption to a major change in price?

Weather forecasting models:How sensitive are the weather forecasts to the grid size used in the calculations?

Watershed models of the turnover of nitrogen:How sensitive is the predicted discharge of nitrogen to the sea to the model parameters describing the turnover of nitrogen in agricultural land?


The INCA-N ( Integrated Nitrogen in Catchments )model of the flow of nitrogen and water through a river basin

Model parameters:

•Initial conditions

•Nitrogen transformation rates

•Hydrogeological parameters

Daily weather data

INCA - N

20

40

60

80

100

120

1 11 21

20

40

60

80

100

120

1 11 21

Daily estimates of water discharge and NO3 and NH4

concentrations in river water

Average annual riverine load of inorganic

nitrogen


Examples of response surfaces produced by theINCA-N model

Average annual nitrogen loss Average annual nitrogen loss

Denitrification rate

Denitrification rate

Max. nitrate uptake ratePlant nitrate

uptake rate


Local sensitivity analysis

A local SA is concerned with small changes about some central case of interest and their impact on the outcome of the calculations

Essentially, the objective of a local sensitivity analysis is to find the partial derivatives of the outcome with respect to the inputs at the point in question


Global sensitivity analysis

A global sensitivity analysis is concerned with the whole set of potential inputs and aims to give an overall indication of the way that the outcome varies


Global sensitivity analysis- questions addressed

Which of the uncertain input factors is more important in determining the uncertainty in the output of interest?

If we could eliminate the uncertainty in one of the input factors, which factor should we choose to reduce the most the variance of the output?


Sensitivity with respect to factors

A factor is anything in a model that can be changed prior to its execution

model input (driving force) model parameter (unknown constant in the model) one mesh size versus another one model structure versus another


A simple portfolio model having threeindependent components

Let

where Cs, Ct, and Cu are the quantities per item and Ps, Pt, and Pu are hedged (delta-neutral) portfolios.

Assume that Ps, Pt, and Pu are independent with mean zero and different standard deviations (volatilities)

uuttss PCPCPCY

1),;0(~

2),;0(~

4),;0(~

uuu

ttt

sss

NP

NP

NP


Sensitivity analysis using partial derivatives

Our analysis indicates the largest portfolio to be the most important factor regardless of its volatility

uuttss PCPCPCY

utsxCP

YS x

xx ,,,


Sensitivity analysis using partial derivativesof fractional change

gives the fractional increase in Y corresponding to a unit fractional increase in Px at a given point in the input space

uuttss PCPCPCY

xx

pyx

x

pyx

lx C

y

p

P

Y

y

p

P

YS

xx0

0

,

0

0

, 0000)ln(

)ln(


Sensitivity analysis using partial derivativesnormalised with respect to the standard deviation

of each factor

When using as a sensitivity measure, the relative importance of Ps, Pt, and Pu depends both on the volatility of these factors and on the weights Cs, Ct, and Cu

uuttss PCPCPCY

xy

x

xy

xx C

P

YS

xS


Sensitivity analysis using derivatives normalised with respect to the standard deviation of each factor

Let

Then

uuttss PCPCPCY

xy

x

xy

xx C

P

YS

22222

22

2

22

2

2

1 xu

xt

xsu

y

ut

y

ts

y

s SSSCCC

Fraction of variance attributable to each factor


A simple portfolio model having three independent factors- a Monte Carlo experiment

Let

Then, we can fit a regression model

)(

)2(

)1(

)()()(

)2()2()2(

)1()1()1(

NNu

Nt

Ns

uts

uts

y

y

y

ppp

ppp

ppp

M

y

)()()()(0

)( iiuu

itt

iss

i pbpbpbby

bx = Cx, x = s, t, u


Standardised regression coefficients (SRCs)

If we set

then we obtain

xutsx

x py ~~,,

yxxx b /

x

xxx

y

ppp

yyy

~;~

For linear models: xx S


Standardised regression coefficients (SRCs)for non-linear models

The R2-value obtained by a linear regression model represents the fraction of the model output variance accounted for by that regression

Provided that the factors are independent, the standardised regression coefficients tell us how this fraction of the output variance can be decomposed according to the input factors, leaving us ignorant about the non-linear parts of the model

The SRCs offer measures of sensitivity that are averaged over a set of possible values of other factors


A portfolio model having six independent factors

Let

where Cs, Ct, and Cu are the quantities per item and Ps, Pt, and Pu are hedged portfolios. Assume that all factors are independent and that

uuttss PCPCPCY

valuespositive totruncated)400;500(~

valuespositive totruncated)300;400(~

valuespositive to truncated)200;250(~

)1;0(~

)2;0(~

)4;0(~

NC

NC

NC

NP

NP

NP

u

t

s

u

t

s


A general importance measureapplicable to all models with random inputs

How does change if one can fix a factor Px at its mid-point? Can it decrease? Can it increase?

How does change if one can fix a factor Px at a point different from the midpoint? Can it decrease? Can it increase?

After averaging over all possible values of Px we obtain

The ratio

is called importance measure, sensitivity index, or first order effect

The sensitivity index of a factor shows how much the variance of the model output would decrease if you were told the exact value of that factor

2yyV

))|(( xPYVE

Y

x

Y

xyx V

PYEV

V

PYVEVS

))|(())|((

2yyV


A portfolio model having six independent factors- calculation of first order effects

Factor Sx

Ps 0.36

Pt 0.22

Pu 0.08

Cs 0.00

Ct 0.00

Cu 0.00

Sum 0.66


Sums of importance measuresfor models having independent factors

In general,

For additive models,

11

k

iiS

11

k

iiS


Second-order effects in the six-factor portfolio model

Calculations or simulations show that for this model we obtain:

jjttss PCPCPCY

y

t

y

s

y

ts

V

PYEV

V

PYEV

V

PPYEV ))|(())|(()),|((

y

s

y

s

y

ss

V

PYEV

V

CYEV

V

PCYEV ))|(())|(()),|((


Second-order effects in the six-factor portfolio model

jjttss PCPCPCY

0))|(())|(()),|((

tststs PP

cPP

y

t

y

s

y

tsPP SSS

V

PYEV

V

PYEV

V

PPYEVS

0))|(())|(()),|((

ssssss PC

cPC

y

s

y

s

y

ssPC SSS

V

PYEV

V

CYEV

V

PCYEVS


A portfolio model having six independent factors- estimates of second order effects

Factor Sx

Ps, Cs 0.18

Pt, Ct 0.22

Pj, Cj 0.08

Ps, Ct 0.00

. 0.00

. 0.00

Sum of second order terms0.34

Sum of first order terms0.66

Grand sum 1.00


General decomposition theoremfor the effects of independent factors

Total effect index (STx) of a factor:

The sum of all effects involving the factor under consideration

1...12 i

ki ij jl

ijlij

iji

i SSSS

First order effects

Second order effects

Third order effects


A portfolio model having six independent factors- main effects and total effect indices

Factor Sx STx

Ps 0.36 0.57

Pt 0.22 0.35

Pj 0.08 0.14

Cs 0.00 0.19

Ct 0.00 0.12

Cj 0.00 0.06

Sum 0.66 1.43


Computational aspects of sensitivity analysis

We need to estimate integrals representing variances of random variables

The estimation of integrals is performed using Monte-Carlo techniques involving pseudo- or quasi-random numbers

The computed indices depend on what assumptions that are made regarding the uncertainty of the input factors


Homework: Decomposition of V(Y) and relative value ofV(E(Y | Xi)) and E(V(Y | X-i)) for different model classes

1. Independent factors

For example: Y = X1 + X2 + X1X2

2. Additive models having dependent factors

For example: Y = X1X1 + X2


Estimation of first order sensitivity indices

Consider the sensitivity indices

where

These indices can be estimated by computing

where and are pseudo – or quasi -random vectors

)(

)(

)(

))|(( 2

YV

YEU

YV

xYEVS ii

i

iiiiii xdxpxxYEU ~)~()~|(2

2

1

2 )(1

)(ˆ

n

mmf

nYE x

2

11

2 )(1

)(1

)(ˆ

m

im

n

mm f

nf

nYV xx

),...,,,,...,,(),...,,...,,(1ˆ ''

)1('

)1('

2'

11

21 mpimmiimmm

n

mmpmimmi xxxxxxfxxxxf

nU

'mxmx


Estimation of total sensitivity indices

Consider the sensitivity indices

where the symbol –i indicates that all variables except xi are kept fixed

These indices can be estimated by computing

)(

)(1

)(

))|((1

2

YV

YEU

YV

xYEVS ii

Ti

),...,,,,...,,(),...,,...,,(1ˆ

)1('

)1(211

21 mpimmiimmm

n

mmpmimmi xxxxxxfxxxxf

nU

Computational statistics 2009 Sensitivity analysis (SA) - outline 1.Objectives and examples 2.Local...

Documents

Transcript of Computational statistics 2009 Sensitivity analysis (SA) - outline 1.Objectives and examples 2.Local...