Post on 24-Dec-2015
1
Expert Judgment
EMSE 280 Techniques of Risk Analysis and Management
2
Expert Judgment
• Why Expert Judgement?
– Risk Analysis deals with events with low intrinsic rates of occurrence not much data available.
– Data sources not originally constructed with a Risk Analysis in mind can be in a form inadequate form for the analysis.
– Data sources can be fraught with problems e.g. poor entry, bad data definitions, dynamic data definitions
– Cost, time, or technical considerations
3
Expert Judgment
• Issues in the Use of Expert Judgement– Selection of Experts
• Wide enough to encompass all facets of scientific thought on the topic• Qualifications\criteria need to be specified
– Pitfalls in Elicitation – Biases• Mindsets – unstated assumptions that the expert uses• Structural Biases – from level of detail or choice of background scales for
quantification• Motivational Biases – expert has a stake in the study outcome • Cognitive Biases
– Overconfidence – manifested in uncertainty estimation– Anchoring – expert subconsciously bases his judgement on some
previously given estimate– Availability – when events that are easily (difficult) to recall are
likely to be overestimated (underestimated)
4
Expert Judgment
– Avoiding Pitfalls
• Be aware
• Carefully design elicitation process
• Perform a dry run elicitation with a group of experts not
participating in the study
• Strive for uniformity in elicitation sessions
• Never perform elicitation session without the presence of a qualified analyst
• Guaranteeing Anonymity of Experts
– Combination of Expert Judgements
• Technical and political issues
5
Basic Expert Judgment for Priors
• Method of Moments
– Expert provides most likely value at parameter, , say * and a range L,U
– for a distribution f() we equate
E[]=(L+4*+U)/6 Var[}= [(U-L)/6]2
– And solve for distribution parameters
• Method of Range
– Expert provides maximum possible range for say L,U
– for a distribution f() with CDF F() we equate
– F(U) = .95 F(L) = .05
– And solve for distribution parameters
6
Combining Expert Judgment: Paired Comparison
• Description
– Paired Comparison is general name for a technique used to combine several experts’ beliefs about the relative probabilities (or rates of occurrence) of certain events.
• Setup
– E # experts
– a1, …, an object to be compared
– v1, …, vn true value of the objects
– v1,r, …, vn,r internal value of object i for expert r
– Experts are asked a series (specifically a total of n taken 2 at a time) of paired comparisons ai, vs aj
– ai, >> aj by e e thinks P(ai) > P(aj)
7
• Statistical Tests– Significance of Expert e’s Preferences (Circular Triad)
Test H0 Expert e Answered Random
Ha Expert e Did Not Answered RandomlyA circular triad is a set of preferences
ai, >> aj , aj >> ak , ak >> ai
Define c # circular triads in a comparison of n objects and
Nr(i) the number of times that expert r prefers ai to another
object expert data Nr(1), …, Nr(n), r = 1, …, e. c(r) the number of circular triads in expert r’s preferences
David(1963)
n
ir niN
nnrc
1
22
12
1)(
2
1
24
)1()(
Combining Expert Judgment: Paired Comparison
8
– Significance of Expert e’s Preferences (Circular Triad)• Kendall (1962)
– tables of the Pr{c(r) >c*} under H0 that the expert answered in a random fashion for n = 2, …, 10
– developed the following statistic for comparing n items in a random fashion,
When n>7, this statistic has (approximately) a chi-
squared distribution with df =
– perform a standard one-tailed hypothesis test. If H0 for any expert cannot be rejected at the 5% level of significance i.e. Pr{2c’(e)}>.05, the expert is dropped
24
21
n
nnn
Combining Expert Judgment: Paired Comparison
2
1)(
34
1
4
8
)4(
)2)(1()('
2ec
n
nn
nnnec
9
• Statistical Tests– Agreement of Experts : coefficient of agreement
Test H0 Experts Agreement is Due to Chance
Ha Experts Agreement is not Due to Chance
Define
N(i,j) denote the number of times ai >> aj.
coefficient of agreement
attains a max of 1 for complete agreement
Combining Expert Judgment: Paired Comparison
1
22
2
),(2
1 ,1
ne
jiN
u
n
i
n
ijj
10
– Agreement of Experts : coefficient of agreement• tabulated distributions of
for small values of n and e under H0
• These are used to test hypothesis concerning u. For large values of n and e, Kendall (1962) developed the statistic
which under H0 has (approx.) a chi squared distribution with .
we want to reject at the 5% level and fail if Pr{2u’}>.05
Combining Expert Judgment: Paired Comparison
n
i
n
ijj
jiN
1 ,1 2
),(
2
23
2221
2
),(4
'1 ,1
e
eenejiN
u
n
i
n
ijj
22
12
e
een
df
11
• Statistical Tests– Agreement of Experts : coefficient of concordance Define
R (i,r) denote the rank of ai obtained expert r’s responses
coefficient of concordance
Again attains max at 1 for complete agreement
Combining Expert Judgment: Paired Comparison
n
i
e
r
n
j
e
r
n
rjR
riRS1
2
1
1 1
),(
),(
)(121 32 nne
Sw
12
– Agreement of Experts : coefficient of concordance
• Tables of critical values developed for distribution of S under H0 for 3n7 and 3n20 by Siegel (1956)
• For n>7, Siegel (1956) provides the the statistic
Which is (approx) Chi Squared with dfn1Again we should reject a the 5% level of significance
Combining Expert Judgment: Paired Comparison
)1(121
'
nen
Sw
13
• Assumptions
vi,r ~N(i, i2) with i = vi and i
2 = 2
Paired Comparison: Thurstone Model
Think of this as tournament play
Probability that 3 beats 2 or 3 is preferred to 2
14
• Assumptions
vi,r ~N(i, i2) with i = vi and i
2 = 2
• Implicationsvi,r vj,r ~N(i j, 22) ~N(i,j, 22) (experts assumed indep) ai is preferred to aj by expert r with probability
if pi,j is the % of experts that preferred ai to aj then
Paired Comparison: Thurstone Model
2}
2Pr{
}2
Pr{}0Pr{
,,
,,,
jiji
jirjri
Z
Zvv
22
2,
,1,
,jiji
jiji
ji porp
15
• Establishing Equations
Then we can establish a set of equations by choosing a scaling constant so that
as this is an over specified system for we solve for i such that
we get and
Mosteler (1951) provides a goodness of fit test based on an approx Chi-Squared Value
Paired Comparison: Thurstone Model
2 x Let ,
1ji,
jijip
ji ji,x
2,
ji,,..,
)(xmin1
jiji
n
n
est
n
ji
1
ji,x
)(
2
n
16
• Assumptions
Thus each paired comparison is the result of a Bernoulli rv for a single expert , a binomial rv for he set of experts
vi are determined up to a constant so we can assume
Define
then vi can be found as the solution to
Paired Comparison: Bradley-Terry Model
ji
ij vv
vaajir
} to prefered is Pr{),( i
n
iiv
1
1
e
rr iNiN
1
)()(
n
ijjji
i
hhe
iNh
,1
1
)(
17
Iterative solution Ford (1956)
Ford (1957) notes that the estimate obtained is the MLE and that the solution is unique and convergence under the conditions that it is not possible to separate the n objects into two sets where all experts deem that no object in the first set is more preferable to any object in the second set.
Bradley (1957) developed a goodness of fit test based on
(asymptotically) distributed as a chi-square distribution with df = (n1)(n2)/2
Paired Comparison: Bradley-Terry Model
1
1 1
1)()(1)1()(
)1( /)(i
j
n
ij
kj
ki
kj
ki
ki
hhhh
eiNh
n
i
n
ijj
n
i
n
i
n
ijjii vveviNjirjiNF
1 ,1 1 1 1
)ln()ln()(),(ln(),(2
18
• Motivation– If Ti~exp(i) then
– For a set of exponential random variables,we may ask experts which one will occur first
– We can use all of the Bradley-Terry machinery to estimate i – We need only have a separate estimate one particular anchor all the others
Paired Comparison: NEL Model
ji
iji TT
}Pr{
19
Combination of Expert Judgment:Bayesian Techniques
• Method of Winkler (1981) & Mosleh and Apostolakis (1986)
– Set Up
• X an unknown quantity of interest
• x1, …, xe estimates of X from experts 1, …, e
• p(x) DM’s prior density for X
• Then
– If the experts are independent
x)?|x,...,p(x
x)p(x)|x,...,p(x)x,..,x|p(x
e1
e1e1
x)?|p(x
x)|p(xx)|x,...,p(x
i
e
1iie1
20
Combination of Expert Judgment:Bayesian Techniques
• Method of Winkler (1981) & Mosleh and Apostolakis (1986)
– Approach
where the parameters μi and σi are selected by the DM to
reflect his\her opinion about the experts’ biases and accuracy
• Under the assumptions of the linear (multiplicative) model, the likelihood is simply the value of the normal (lognormal) density with parameters x+μi and σi .
• Then for the additive model we have
indep. and ),N(~ )Ln( Model tiveMultiplica
indep. and ),N(~ Model Additive2
i
2i
iiii
iiii
xx
xx
bygiven varianceandmean with normal is
),...,x|p(x then ,deviation standard and mean with
normal be )(prior smaker'decision Let the 10.1n Propositio
111e ee xx
xp
21
Combination of Expert Judgment:Bayesian Techniques
0 and with
1 ),...,x|VAR(X
and )( ),...,x|E(X
1e1e
1
2
2
1e
1
21
1e
1i1
jj
ii
ii
e
iiie
w
x
xwx
Note: i. the multiplicative model follows the same line of
reasoning but with the lognormal distribution
ii. the DM acts as the e+1st expert, (perhaps uncomfortable)
22
Combination of Expert Judgment:Bayesian TechniquesMOSLEH APOSTALAKAIS MODEL
FOR EXPERT JUDGMENT COMBINATION
Selct Model
Additive
Multiplicative x
Number of Experts
3
DM PRIOR SPECIFICATION
Mean X St Dev X Mean Ln X St Dev Ln X
4.63E-05 9.36E-06 -1.00E+01 2.00E-01
EXPERT INPUT
Estimate Expert EvaluationMult. Error Parameters Ln Error Parameters
Number Xi Mean St Dev Ln Xi Ln Mean Ln St Dev
1 2.00E-05 1.01E+00 1.01E-01 -1.08E+01 0.00E+00 1.00E-012 4.50E-06 8.00E+00 3.33E+00 -1.23E+01 2.00E+00 4.00E-013 1.00E-04 2.77E+00 5.60E-01 -9.21E+00 1.00E+00 2.00E-01
POSTERIOR INFERENCE
Mean 2.19E-05 Ln Mean -1.07E+01St Dev 1.76E-06 Ln St Dev 8.00E-02
Prob X in =
-5.00E +04
0.00E +00
5.00E +04
1.00E +05
1.50E +05
2.00E +05
2.50E +05
0.00E +00 1.00E -05 2.00E -05 3.00E -05 4.00E -05 5.00E -05 6.00E -05 7.00E -05 8.00E -05
Prior
Post.
23
Combination of Expert Judgment:The Classical Model
• Overview
– Experts are asked to assess their uncertainty distribution via specification of a 5%, 50% and 95%-ile values for unknown values and for a set of seed variables (whose actual realization is known to the analyst alone) and a set of variables of interest
– The analyst determines the Intrinsic Range or bounds for the variable distributions
– Expert weights are determined via a combination of calibration and information scores on the seed variable values
– These weights can be shown to satisfy an asymptotic strictly proper scoring rules, i.e., experts achieve their best maximum expected weight in the long run only by stating assessments corresponding to their actual beliefs
24Combination of Expert Judgment:The Classical Model
0
.5
1
ql q5 q50 q95 qu
0
.5
1
ql q5 q50 q95 qu
For a weighted combination of expert CDFs take the weighted combination at all break points (i.e. qi values for each expert) and then linearly interpolate
Expert 1
Expert 2
25Combination of Expert Judgment:The Classical Model
Var. 1
Expert 1 Expert 2 Expert 3
x| || | | | || |x x
Var. 2 x| || | | | || |x x
Var. 3 x| || | || || |x x
Var. 4 x| || | || || |x x
Var. 5 x| || | || || |x x
Expert 1 – Calibrated but not informative
Expert 2 – Informative but not calibrated
Expert 3 – Informative and calibrated
Expert Distribution Break Points
Realization
26Combination of Expert Judgment:The Classical Model
– Information• Informativeness is measured with respect to some background
measure, in this context usually the uniform distributionF(x) = [x-l]/[h-l] l < x < h
• or log-uniform distribution F(x) = [ln(x)-ln(l)]/[ln(h)-ln(l)] l < x < h
• Probability densities are associated with the assessments of each expert for each query variable by– the density agrees with the expert’s quantile assessment – the densities are nominally informative with respect to the
background measure– When the background measure is uniform, for example,
then the Expert’s distribution is uniform on it’s 0% to 5% quantile, 5% to 50% quantile, etc.
27Combination of Expert Judgment:The Classical Model
– Information
• The relative information for expert e on a variable is
• That is, r1 = F(q5(e)) F(ql(e)) , …, r4 = F(qh(e)) F(q95(e))
•
• The expert information score is the average information over all variables
i intervalfor measure background
theis r and 45,.05)(.05,.45,. is ),...,( where
)/()(
i41
4
1
pp
rpLnprIi
iii
min max| | | | |
1
0.5
Expert Distribution
Uniform Background Measure
28
Combination of Expert Judgment:The Classical Model
• Intrinsic Range for Each Seed Variable
• Let qi(e) denote expert e’s i% quantile for seed variable X
• Let seed variable X have realization (unknown to the experts ) of r
• Determine intrinsic range as (assuming m experts)
l=min{q5(1),…, q5(m),r} and h =max{q95(1),…, q95(m),r}
• then for k the overshoot percentage (usually k = 10%)
– ql(e)=l – k(h - l)
– qh(e)=l + k(h - l)
– Expert Distribution (CDF) for seed variable X is a linear interpolation between
• (ql(e),0), (q5(e),.05), (q50(e),.5), (q.95(e),.95), (qh(e),1)
29
Combination of Expert Judgment:The Classical Model
• Calibration
– By specifying the 5%, 50% and 95%-iles, the expert is specifying a 4-bin multinomial distribution with probabilities .05, .45, .45, and .05 for each seed variable response
– For each expert, the seed variable outcome (realization), r, is the result of a multinomial experiment, i.e.
• r [ql(e), (q5(e)), [interval 1], with probability 0.05
• r [q5(e), q50(e)), [interval 2], with probability 0.45
• r [q50(e), q95(e)), [interval 3], with probability 0.45
• r [q95(e), qh(e)], [interval 4], with probability 0.05
– Then if there are N seed variables and assuming independence
si= [# seed variable in interval i]/N is an empirical estimate of
(p1, p2, p3, p4) = (.05, .45, .45, .05)
30
Combination of Expert Judgment:The Classical Model
• Calibration
• We may test how well the expert is calibrated by testing the hypothesis that
H0 si = pi for all i vs Ha si pi for some i
• This can be performed using Relative Information
45,.05)(.05,.45,. is ),...,( and variablesseed the
fromdensity empirical theis ),...,(s where
)/(),(
41
41
4
1
pp
s
psLnspsIi
iii
31
Combination of Expert Judgment:The Classical Model
Note that this value is always nonnegative and only takes the value 0 when si=pi for all i.
• If N (the number of seed variables) is large enough
• Thus the calibration score for the expert is the probability of getting a relative information score worse (greater or equal to) than what was obtained
4
1
)/(),(i
iii psLnspsI
χ~ p)2NI(s, 2(3)
p)}2NI(s, Pr{χ p))(2NI(s,χ where
p))(2NI(s,χ1c(e)2(3)
2(3)
2(3)
32Combination of Expert Judgment:The Classical Model
– Weights
• Proportional to calibration score * information score
• Don’t forget to normalize
– Note
• as intrinsic range for a variable is dependent on expert quantiles, dropping experts may cause the intrinsic range
to be recalculated
• change in intrinsic range and background measure have negligible to modest affects on scores
33Combination of Expert Judgment:The Classical Model
• Optimal (DM)Weights – Choose minimum value such that if C(e) > , C(e) = 0 (some experts will get 0 weight)– is selected so that a fictitious expert with a distribution equal to that of the the weighted combination of expert distributions would be given the highest weight among experts
34Combination of Expert Judgment:The Classical Model
COMBINATION OF EXPERT JUDGMENT
Number of Experts 4
Select Type of Weights
Equal
Use Specified
Performance Based x
Intrinsic Range Adjustment 0.1
Expert Input Weights# l 5% 50% 95% h Equal User Perform
1 40 200.00 330.00 500.00 760 0.25 0.40 0.2662 40 270.00 300.00 400.00 760 0.25 0.10 0.4293 40 100.00 200.00 300.00 760 0.25 0.10 0.0004 40 200.00 300.00 700.00 760 0.25 0.40 0.306
35Combination of Expert Judgment:The Classical Model
SEED VARIABLE INPUT
Overshoot 0.1Number of Variables 5
VARIABLE 1 ql= 0
Realization 26 qh= 87.9
Experts 5% 50% 95% <5% 5%-50% 50%-95% >95% Information1 2.00E+01 5.50E+01 8.00E+01 0 1 0 0 1.56E-012 3.00E+01 5.00E+01 6.00E+01 1 0 0 0 7.37E-013 1.00E+00 1.00E+01 2.00E+01 0 0 0 1 1.22E+004 1.00E+01 5.00E+01 6.00E+01 0 1 0 0 4.80E-01
0 0 0 0 0.00E+000 0 0 0 0.00E+000 0 0 0 0.00E+000 0 0 0 0.00E+000 0 0 0 0.00E+000 0 0 0 0.00E+00
36Combination of Expert Judgment:The Classical Model
PERFORMANCE W EIGHTS
EMPITRICAL DISTRIBUTION INDIVIDUAL VARIABLE INFORMATIONEXPERT NORM WT WT CAL 0.05 0.45 0.45 0.05 INF VAR 1 VAR 2 VAR 3 VAR 4 VAR 5 VAR 6
1 0.274 0.187 0.395 0.000 0.200 0.800 0.000 0.4749 0.1564 0.3182 0.7481 0.6284 0.5236 0.00002 0.439 0.301 0.411 0.200 0.200 0.600 0.000 0.7323 0.7372 0.6301 0.5415 0.6391 1.1136 0.00003 0.000 0.000 0.000 0.000 0.000 0.200 0.800 1.2720 1.2222 0.7847 1.3326 0.7847 2.2356 0.00004 0.287 0.196 0.740 0.000 0.400 0.600 0.000 0.2656 0.4803 0.2260 0.2884 0.2260 0.1071 0.0000
37Combination of Expert Judgment:The Classical Model
Plot ComparisonPlot 1 DMPlot 2 1Plot 3 2Plot 4 3Plot 5 4
Expert CDF
0.0000.100
0.2000.300
0.4000.500
0.6000.700
0.8000.900
1.000
0 200 400 600 800 1000Varaible Value
CD
F
DM
1
2
3
4
Expert CDF
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1.000
0 200 400 600 800 1000Varaible Value
CD
F
DM
1
2
3
4
Expert CDF
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1.000
0 200 400 600 800 1000Varaible Value
CD
FDM
1
2
3
4
USER DEFINED WEIGHTS EQUAL WEIGHTS
PERFORMANCE BASED WEIGHTS