Stakeholder Preference Modeling with

Probabilistic Inversion

Roger M. CookeResources for the Future

Dept Math, TU DelftJune 16, 2011

• Foundations• Health states• Risks nano-enabled food

Expert Judgment for Uncertainty Quantification: PM2.5

Uncertainty in Mortality Response to Airborne Fine Particulate Matter: Combining European Air Pollution Experts Jouni T. Tuomisto, Andrew Wilson, John S. Evans, Marko Tainio (RESS 2008)

Fundamental Theorem of Decision TheoryFor Rational PreferenceUNIQUE probability P which represents degree of belief:

DegBel(France wins worldcup) > DegBel(Belgium wins worldcup) P(F) > P(B)

AND a Utility function, unique up to 0 and 1, that represents values:

($1000 if F, $0 else) > ($1000 if B, $0 else) Exp’d Utility (($1000 if F, $0 else)) > Exp’d Utility (($9000 if B, $0 else))

BUT….

UNLIKE Expert Judgment: There is no

• Updating utilities on observations

• Convergence of utilities via Observations

• Empirical control on Utilities

• Community of ‘Utility Experts’

• Rational consensus on Utilities

Why is Preference Modeling Impoverished?

AHPMAUTMCDMELECTRAREMBRANTOUTRANKINGTHURSTONE

BRADLEY TERRYPROBITLOGITNESTED LOGITPSYCH’L SCALING

Validation???

What means Validation?

Goal = find ‘true Utility values’ for alternatives?Fools’s Errand

Condorcet’s Paradox of Majority Preference

1/3 prefer Mozart > Hayden > Bach 1/3 prefer Hayden > Bach > Mozart1/3 prefer Bach > Mozart > Hayden

THEN 2/3’s prefer

Bach > MozartMozart > HaydenHayden > Bach

What can we do?

Random Utility Theory

Each (rational) stakeholder has a utility function over alternatives characterize population as distribution over utility functions

Probabilistic Inversion

Domain: utility functionsOf stakeholders

Range: choices of stakeholders

G maps utilities into choices

Invert G at this distribution

Observe Stakeholders Preferences

Used for stakeholder Preference Modeling:

• Risks of Nano enabled foods (Flari, WHO, CIS)• Valuing impaired health states (Flari, FDA)• Valuing fossil fuel policy options (RFF)• Prioritizing ecosystem threats (NCEAS)• Prioritizing zoonose threats (RIVM)• Modeling wiring failure (Mazzuchi)• Prioritizing vCVJ options (Aspinall Health Canada)• UK Research Council (Aspinall)• Aus. Univ. Fac Sci reviews (Aspinall).

steps

1. Get discrete choice data from stakeholders for choice alternatives A1,…An– “Which of (A,B) do you prefer”– “Rank your top 3 of (A, B, C, D, E, F)”

2. Find dist’n over utilities on [0,1]n which reproduces stakeholders preferences

3. If utility is function of covariates, validate out of sample.

Valuation of impaired Health statesFlari et al

17 health states6 criteriaEach criteria has 3 values, described in narrative19 Experts ranked 5 groups of 5 health states

First, find dist’n over utilities for the 17 Health States which recover Observed Frequencies of rankings (i.e. wo criteria)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.80

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

f(x) = 0.912874736742624 x + 0.0143452928596942R² = 0.977197926615134

Predicted vs Observed Frequencies of 105 rankings, scale values for the 17 Health States, no MAUT model

RecoveredLinear (Recovered)

Observed

Reco

vere

d

Build MAUT model for HS utilities, based on the 6 criteria

• Each stakeholder has a weight vector that determines his/her preference

• Population of stakeholders = population of weights

• Characterize population based on all rankings involving at least 7 (30%) experts (= 28 rankings).

• Validate on remaining rankings (= 77 rankings)

Average weights

12%

29%

8%27%

10%

15%

Mean Weights

MobilitySelf CareUsual ActivitiesPain DiscomfortAnxiety DepressionCognitive Functioning

Preference dependence emerges from fitting

Average weights per group of 5 health states

Predict out-of-sample rankings

0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.800

0.1

0.2

0.3

0.4

0.5

0.6

0.7

f(x) = 0.879511080111118 x + 0.0210766294292423R² = 0.5721455934275

Predicted vs Observed % rankings, 28 fitted, 77 Out-of-Sample

PredictedLinear (Predicted)

Observed Percentage Ranks

pred

icted

per

cent

ages

Ran

ks

First time in HISTORY that a multi attribute model has been WRONG!!!

Average of predictions vs Out-of-Sample observed rankings, Not SOOO bad

0.00 0.05 0.10 0.15 0.20 0.25 0.30-0.05

0.00

0.05

0.10

0.15

0.20

0.25

0.30

Out-of-sample:average predicted versus observed

observed percentages for out-of-sample rankings (less than 30% of experts)

pred

icte

d pe

rcen

tage

s fo

r ou

t-of

-sam

-pl

e ra

nkin

gs

Risks of Nano-Enabled Foods

Flari et al

Nano enabled food risks (VLARI) rank top and bottom 5

Correlation of criteria weights

Conclusion

• Stakeholder preference modeling is empirical science

• ‘preference for criteria’ inferred from data, not elicited

• (in) dependence in choices inferred from data, not assumed

THANK YOU