Sensitivity analysis:An introduction

Stefano TarantolaEuropean Commission, Joint Research Centre,

Ispra (I)

Fifth CourseOn Impact Assessment

BrusselsJanuary 20-21, 2015

Stefano.tarantola@jrc.ec.europa.eu

Simulation (or computer) models are used in manydisciplinesto understand complex phenomena (natural or social) and consequently as tools to support decisionsand policy.

x y

Knowledge base is often flawed by uncertainties (partly irreducible, largely unquantifiable), imperfect understanding, subjective values.

A few examples … x y

Uncertainties in model parameters that govern surface and ground water transport, …

Courtesy of

Models in hydrology

Ex: biological model© 2008 Zi et al; licensee BioMed Central Ltd.

… Uncertainties of kinetic parameters in a chemical process…

Models in bio-chemistry

A B

C

D

E

F

• Parameters of the supply model are mostly uncertain (but kept fixed in the usual practice)

Models in traffic simul.

Uncertainty analysis the analyst can scrutinize uncertainties in model parameters, input data, subjective assumptions and alternative model structures how they propagate through the model effect on predictions identification of the best policy alternative.

Uncertainty analysis ‘forward process’

Sensitivity analysis ‘backward process’. “The study of how the uncertainty in the output of a model can be apportioned to different sources of uncertainty in the model input”. identify which inputs are most influential for the prediction

On those important inputs one should focus to see whether their uncertainty can be reduced

improve prediction accuracy.

Various types of uncertain inputs:

Data

Parameters

Assumptions

Scenarios

Alternative model specifications

x y

Sensitivity analysis: what for

1. Prioritising acquisition of information

If model prediction is too uncertain SA identify important factors reduce uncertainty of important factors increase robustness of results

Sensitivity analysis: what for

2. Model understanding

Is the model doing what we expect from it?Discover inputs interactions.

Sensitivity analysis: what for

3. Model simplification

Identify inputs with no effect on the prediction

Sensitivity analysis: what for

4. Model simplification

Identify critical regions in the space of inputsExample …

x y

y

P(y)

Sensitivity analysis: what for

5. Are policy options distinguishable given the uncertainties?

Example …

Traffic modelling:

Average Travel timeA BPolicy A: traffic lightsPolicy B: roundabouts

Deterministic assessment

Traffic modelling:

A B

Average Travel time

Probabilistic assessment

A B

Average Travel time

A better than B (given the otheruncertainties)

The other uncertainties obfuscatethe effect of the policies

- Identify the factors responsible for the overlap

-More knowledge on those factors could allow the decisionto be taken

Other uncertainties

A vs B A B

Average Travel Time

A B

Average Travel time

A B

Average Travel Time

Local, One at a Time

and Global Sensitivity Analysis

Local SA

xr

xr = nominal value

0 1x

2x

),( 21 xx = space of input

- evaluation of partial derivatives - works in the neighborhood of nominal point- use of Taylor-like formulas

0xxiXY

rr=

∂∂

x y

One at a time SA

xr = nominal value

0 1x

2x

),( 21 xx = space of input

xr

- SA performed by changing one input variable by one while keeping others at their baseline nominal values

- the other inputs are kept fixed

Global SA

xr = nominal value0 1x

2x

),( 21 xx = space of input

- full exploration of uncertainty- Monte Carlo methods to generate samples

Regression / correlationScreening techniques

Variance decompositionMoment- independent

Statistical testsGraphical tools

At large dimension of input space OAT exploresa negligible volume with respect to GSA

Limitations of OAT

Area circle / area square = 0.78Volume sphere / volume cube = 0.5

In 10 dimensions:Vol hyper-sphere / vol. hyper-cube = 0.0025

… the modeller is afraid his model willcrash in that region as global SA explores the boundary of the input space

OAT is still widely used because …

Model OutputInput

x1

x2

x3

…

x4

xk

y

xr )(xfy r= y

Monte Carlo approach to uncertainty analysis

Space of uncertainty

...X1

Positive Impactof policy

Policy 1  Policy 2NO policy

10

20

30

40

50

60

Monte Carlo approach to uncertainty analysis

X2 X3

XjXk

Specification of the model inputs

)( ii xp

x1

x2

x3

x4

Characterise the uncertainty of each input.

Assign a pdf using all available information

eg experiments, estimations, physical bounds

considerations, scientific knowledge and

expert opinion.

A very delicate step: it may require significant

resources.

Extended peer-review should be considered

to ensure quality in the treatment of

uncertainty

A major issue in global sensitivity analysis is the number of model runs required to conduct the analysis.

Our preferred methods for SA: variance-based

concise and easy to communicate

Variance-based method’s best formalization is based on the work of Ilya M. Sobol’(1990) who extended the work of R. I. Cukier (1973).

First-order sensitivity indices

x y

)()]|([

yVarxyEVarS i

i =

)]|([)]|([)( ii xyVarExyEVaryVar +=

Easy to prove using V(•)=E(•)2-E

2(•)

-60

-40

-20

0

20

40

60

-4 -3 -2 -1 0 1 2 3 4

-60

-40

-20

0

20

40

60

-4 -3 -2 -1 0 1 2 3 4

The ordinate axis is always Y

The abscissa are the various factors Xi in turn.

The points are always the same!

-60

-40

-20

0

20

40

60

-4 -3 -2 -1 0 1 2 3 4

-60

-40

-20

0

20

40

60

-4 -3 -2 -1 0 1 2 3 4

Which variable is the most important?

-60

-40

-20

0

20

40

60

-4 -3 -2 -1 0 1 2 3 4

-60

-40

-20

0

20

40

60

-4 -3 -2 -1 0 1 2 3 4

These are ~1,000 points

Divide them in 20 bins of ~ 50 points

-60

-40

-20

0

20

40

60

-4 -3 -2 -1 0 1 2 3 4

-60

-40

-20

0

20

40

60

-4 -3 -2 -1 0 1 2 3 4

Compute the bin’s average (pink dots)

( )iXYEi~XEach pink point is ~

-60

-40

-20

0

20

40

60

-4 -3 -2 -1 0 1 2 3 4

( )( )iX XYEVii ~X

Take the variance of the pinkies

-60

-40

-20

0

20

40

60

-4 -3 -2 -1 0 1 2 3 4

( )( )iX XYEVii ~X

First order effect =

= the expected reduction in variance that would be achieved if factor Xi could be fixed.

Why? 

( )( )( )( ) )(

~

~

YVXYVE

XYEV

iX

iX

ii

ii

=+

+

X

X

Because:

( )( )( )( ) )(

~

~

YVXYVE

XYEV

iX

iX

ii

ii

=+

+

X

X

Because:

The variance that would be left (on average) if Xi could be fixed.

Variance decomposition (ANOVA) 

( )

kiji

iji

i VVV

YV

...123,

...+++

=

∑∑>

)(YVVi

i ≈∑

For additive systems one can decompose the total variance as a sum of first order effects  

)],|([)],|([)( jiji xxyVarExxyEVaryVar +=

Joint effects

)()],|([

yVarxxyEVar

S jiij

tjoin =

45

The expected amount of variance that would remain unexplained (residual variance)

if xi, and only xi, were left free to vary over its uncertainty range.

Use: for model simplification, to identify unessential inputs in the model, which are not important neither singularly nor in combination with others.

An input with a small value of its total effect sensitivity index can be frozento any value within its range.

)(/)]|([ YVarxYVarES iTi −=

Total effects

)]|([)]|([)( ii xYVarExYEVarYVar −− +=

We cannot use Si  to fix a factor; Si =0 is a necessary but not sufficient condition for Xi to be non‐influential.

Xi could be influential at the second order.

Example …

Si ?

-60

-40

-20

0

20

40

60

-4 -3 -2 -1 0 1 2 3 4

Other (non variance-based) techniques

Screening techniques (Morris, 1991)

Graphical methods

Derivative-based techniques

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

∫ ⎟⎟⎠

⎞⎜⎜⎝

⎛∂∂

= dxxy

ii

2

ν

Global sensitivity analysis.

The Primer

A textbook of methods to evidencehow model-based inference depends

upon modelspecifications and assumptions,

John Wiley, 2008

Saltelli, A., Ratto M., Andres, T., Campolongo, F., Cariboni J., Gatelli

D., Ratto, M., Saisana, M., Tarantola, S.

List of References

Sobol’ and Kucherenko (2009) Derivative based global sensitivity measures and their link with global sensitivity indices, Mathematics and Computers in Simulation 79, 3009–3017

Bolado, Castaings and Tarantola (2009) Contribution to the sample mean plot for graphical and numerical sensitivity analysis, Reliability Engineering and System Safety 94, 1041–1049

Tarantola, S., V. Kopustinskas, R. Bolado-Lavin, A. Kaliatka, E. Uspuras, M. Vaisnoras (2012) Sensitivity analysis using contribution to sample variance plot: Application to a water hammer model, Reliability Engineering and System Safety 99, 62–73

Morris, M.D. (1991) Factorial Sampling Plans for Preliminary Computational Experiments, Technometrics 33: 161–174

Saltelli A., P. Annoni I. Azzini, F. Campolongo, M. Ratto and S. Tarantola (2010) Variance based sensitivity analysis of model output. Design and estimator for the total sensitivity index, Computer Physics Communications 181, 259–270

Kucherenko S., S. Tarantola, P. Annoni (2012) Estimation of global sensitivity indices for models with dependent variables, Computer Physics Communications 183, 937–946

Thank you for your attention!

Questions?