Quantifying Unobserved Attributes in Expert Elicitation

of Terrorist PreferencesVicki Bier, Chen Wang

University of Wisconsin-Madison

Research Goals• To construct a reasonable defender prior

distribution over possible terrorist preferences: – By explicitly modeling the defender’s uncertainty about

unobserved attributes– I.e., attributes that may be important to the terrorist, but are

un-quantified or unobserved by the defender

• To simplify the task of quantifying threat probabilities for subject-matter experts:– By using ordinal rather than cardinal estimates– To increase the acceptance of quantitative approaches in

the intelligence community

Indirect Expert Elicitation

• Allows experts to express their knowledge as rank orderings rather than numerical values:– Simplifies the process of bringing expert knowledge to bear

• Well suited to elicitation challenges commonly encountered in the intelligence community:– High uncertainty and sparse data– Reluctance on the part of experts to express their

knowledge in probabilistic form

Mathematical Approach• Experts provide rank orderings of selected targets or

attack strategies:– Reflecting their knowledge of adversary preferences

• Adversary preferences are assumed to follow an additive multi-attribute utility function:– With an “error term” representing the effect of any attributes

that have not been identified or observed by the defender

– Assumed to be independent and identically distributed!

• Probabilistic inversion is used to estimate the values of both attribute weights and unobserved attributes: – To yield the best fit to the stated rank orderings

– Taking into account expert consensus or disagreement

Urban AreaProperty Loss($ million),

X1

Fatalities,X2

Population,X3

Population Density

(per sq mile),X4

New York, NY 413 304 9,314,235 8,159 Chicago 115 54 8,272,768 1,634

San Francisco 57 24 1,731,183 1,705 Washington, DC-MD-VA-WV 36 29 4,923,153 756

Los Angeles-Long Beach 34 17 9,519,338 2,344 Philadelphia, PA-NJ 21 9 5,100,931 1,323

Boston, MA-NH 18 12 3,406,829 1,685 Houston 11 9 4,177,646 706

Newark, NJ 7.3 4 2,032,989 1,289 Seattle-Bellevue-Everett 6.7 4 2,414,616 546

Jersey City 4.4 2 608,975 13,044 Detroit 4.2 1.9 4,441,551 1,140

Las Vegas, NV-AZ 4.1 1 1,563,282 40 Oakland, CA 4 1 2,392,557 1,642

Orange County, CA 3.7 2 2,846,289 3,606 Cleveland-Lorain-Elyria 3 0.5 2,250,871 832

San Diego 2.8 1 2,813,833 670 Miami, FL 2.7 0.5 2,253,362 1,158

Minneapolis-St. Paul, MN-WI 2.7 0.4 2,968,806 490 Denver 2.5 1.1 2,109,282 561

Attribute Values for the Case Study

3 Hypothetical Expert Groups• Each group has 10 experts:

– Each of whom ranks the top 10 targets

• Group 1: – All find X1 (property loss) and X2 (fatalities) important,

but not X4 (population density)– Little or no weight on unobserved attributes

• Group 2: – All think X4 (population density) is important– Opinions reflect an unobserved attribute,

corresponding to presence of entertainment industry

• Group 3 (expert disagreement):– Five experts from Group 1, and five from Group 2

Modeling Unobserved Attributes• Attempt to fit the attribute weights to the stated rank

orderings• Use trial and error to find the weight for the unobserved

attribute that yields the lowest infeasibility• Resulting weights:

– Group 1: 0.02– Group 2: 0.09 (largest)– Group 3: 0.08

Results of Probabilistic Inversion

• Group 1 yields high weights on X1 and X2:– Low weight on X4

• Group 2 yields the highest weight on X4:– With Group 3 intermediate between Groups 1 and 2

• As expected, Group 2 has the largest infeasibility:– Since the experts in Group 2 take unobserved attributes into account,

even the best fit performs worse than the other two groups

E[X1](Property

Loss)E[X2]

(Fatalities)E[X3]

(Population)

E[X4] (Population

Density)

X5Unobserved Attributes

Relative Information (Infeasibility)

Group 1 0.367 0.552 0.023 0.038 0.02 0.272Group 2 0.210 0.265 0.090 0.345 0.09 1.067Group 3 0.325 0.366 0.113 0.117 0.08 0.016

Distributions of Attribute Weights

Group 1

Group 2

Group 3

Uniform prior

Probabilistic inversion

X1 and X2 increase

X4 increases

Inconsistent judgments –

higher variance

x_1

0.0 0.2 0.4 0.6 0.8 1.0

010

0020

0030

0040

00

x_2

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

050

015

0025

0035

00 x_3

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

010

0030

0050

00

x_4

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

050

0015

000

x_1

0.0 0.2 0.4 0.6 0.8 1.0

010

0030

0050

00

x_2

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

010

0030

0050

00

x_3

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

040

0080

0012

000

x_4

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

020

0040

0060

0080

00

x_1

0.0 0.2 0.4 0.6 0.8 1.0

010

0030

0050

00

x_2

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

010

0030

00

x_3

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

020

0060

00

x_4

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

020

0060

0010

000

Unobserved Attributes• Can look at the posterior distributions for the unobserved attribute:

– To identify candidate unobserved attribute(s)

Group 2

• Posterior correlations for unobserved attributes:

– Positive correlation between Los Angeles and Las Vegas suggests that some experts consider presence of an entertainment industry important

LA, Jersey City, and Las

Vegas increase

epsilon_NYC0

2000

4000

6000

epsilon_LA

Fre

quen

cy

010

0030

0050

00

epsilon_Jersey

Fre

quen

cy

010

0030

0050

00 epsilon_LV

Fre

quen

cy

010

0030

0050

00

Uniform prior

Probabilistic inversion

NYC DC LA Las VegasNYC 1 -0.201 0.034 0.045DC 1 -0.011 0.014LA 1 0.161

Las Vegas 1

Predicted Rankings With Unobserved Attributes

Rank Group 1 Group 2 Group 31 NYC NYC NYC2 Chicago Jersey City Chicago3 San Francisco Chicago LA4 DC LA DC5 LA Orange County San Francisco6 Boston San Francisco Jersey City7 Philadelphia DC Boston8 Houston Boston Philadelphia9 Jersey City Philadelphia Houston

10 Newark Detroit Orange County11 Seattle Houston Newark12 Orange County Oakland Oakland13 Detroit Newark Detroit14 Oakland Miami Miami15 San Diego Las Vegas Seattle16 Miami Seattle Minneapolis17 Denver Cleveland San Diego18 Minneapolis Minneapolis Las Vegas19 Cleveland San Diego Cleveland20 Las Vegas Denver Denver

Red –Increased

Blue –decreased

Changes by more than 1 place are colored.

Compared to case without unobserved attributes

Assessment of Results• Predicted rankings are more consistent with expert

judgments when unobserved attributes are included, especially for Group 2: – Las Vegas gets high rankings due to unobserved attribute

(presence of entertainment industry)– The model without unobserved attributes does not have the

flexibility to adequately reflect expert judgments

• Can also be used as a basis for inference about what unobserved attributes

• For example, if LA and Las Vegas are rated higher than their known attribute values would suggest: – That might indicate the need to include presence of a large

entertainment industry as a terrorist attribute

Assessment of Results• Results may be better than direct weight elicitation• For example, some experts may put high weight on

population density:– Without realizing this implies a high ranking for Jersey City

• Can deal with conflicting and/or inconsistent expert opinions:– By (possibly multi-modal) distributions of attribute weights

with high variance

Pre-Posterior Analysis

• How do the models perform with Bayesian updating:– Especially after an unexpected attack?

• Problem:– Some targets have zero probability of being attacked in the

model– Model would break in the event of an attack on such a target– Cannot condition on a set of measure zero!

• Model without unobserved attributes is especially poor in this respect

• May need to consider non-uniform (e.g., U-shaped) prior distributions for the unobserved attributes

Bayesian Updating• Consider a target with a positive probability of being

attacked:– Assume an (unexpected) attack on Jersey City

• Probability that the next attack is also on Jersey City becomes quite high (maybe unrealistically high)

• What happens to the attribute weights?

Group 1

Group 2

Group 3

Prior mean

Posterior mean

X1 and X2 decrease;

X4 increases

An Attack on Jersey City

x_1

0.0 0.2 0.4 0.6 0.8 1.0

020

060

010

00

x_2

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

020

060

010

00

x_3

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

020

040

060

0

x_4

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

020

040

060

0

x_1

0.0 0.2 0.4 0.6 0.8 1.0

050

100

150

200

x_2

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

050

100

150

200

x_3

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

050

100

150

200

x_4

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

050

100

150

x_1

0.0 0.2 0.4 0.6 0.8 1.0

050

010

0020

00

x_2

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

050

010

0020

00

x_3

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

020

060

010

00

x_4

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

020

060

010

00

epsilon_NYC

0.0 0.2 0.4 0.6 0.8 1.0

01

23

45

epsilon_Jersey

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

01

23

45

67

epsilon_Las Vegas

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

02

46

8

Posterior of Unobserved Attribute

Group 1

Group 2

Group 3

Jersey City has higher values on the unobserved

attributeepsilon_NYC

0.0 0.2 0.4 0.6 0.8 1.0

010

020

030

040

050

0 epsilon_Jersey

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

010

020

030

040

050

0 epsilon_Las Vegas

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

010

020

030

040

050

0

epsilon_NYC

0.0 0.2 0.4 0.6 0.8 1.0

0100

200

300

400

epsilon_Jersey

Fre

quenc

y

0.0 0.2 0.4 0.6 0.8 1.0

0100

200

300

400

epsilon_Las Vegas

Fre

quenc

y

0.0 0.2 0.4 0.6 0.8 1.0

0100

200

300

400

Prior mean

Posterior mean

Predicted Rankings after an Attack on Jersey City

Rank Group 1 Group 2 Group 31 NYC Jersey City NYC2 Jersey City NYC Jersey City3 Chicago Orange Orange4 LA LA LA5 San Francisco Chicago Chicago6 Orange Boston Boston7 DC San Francisco San Francisco8 Boston Philadelphia Philadelphia9 Philadelphia Oakland Oakland

10 Oakland Detroit Detroit11 Detroit DC DC12 Newark Newark Newark13 Houston Miami Miami14 Miami Houston Houston15 Cleveland Cleveland Cleveland16 San Diego San Diego San Diego17 Seattle Seattle Seattle18 Minneapollis Minneapollis Minneapollis19 Denver Denver Denver20 LV LV LV

Red –increased

Blue –decreased

Jersey City ranks higher, but not the

highest!

This seems reasonable

Groups 1 and 3 still consider

NYC more attractive

Changes by more than 2 places are colored.

Group 1

Group 2

Group 3

x_1

0.0 0.4 0.8

0500

1000

1500

x_2

Fre

quency

0.0 0.4 0.8

0500

1000

1500

2000 x_3

Fre

quency

0.0 0.4 0.8

0500

1500

2500

x_4

Fre

quency

0.0 0.4 0.8

01000

2000

3000

4000

x_1

0.0 0.4 0.8

0200

400

600

800

1000 x_2

Fre

quency

0.0 0.4 0.8

0200

600

1000

1400 x_3

Fre

quency

0.0 0.4 0.8

0200

600

1000

x_4

Fre

quency

0.0 0.4 0.8

02000

4000

6000

x_1

0.0 0.4 0.8

0500

1000

1500

x_2

Fre

quency

0.0 0.4 0.8

0500

1000

1500

x_3

Fre

quency

0.0 0.4 0.8

01000

2000

3000

x_4

Fre

quency

0.0 0.4 0.8

0500

1000

2000

Prior mean

Posterior mean No significant changes

An Attack on New York City

Future Directions

• An alternative approach for fitting expert opinions:– Bayesian density estimation

• Sensitivity analysis on the performance of the two approaches: – Computational behavior (convergence properties, run times)– Reasonableness of predicted rankings– Performance with Bayesian updating after expected and

unexpected attacks

• Obtain stakeholder feedback on applicability of methodology and realism of results