Capturing User Intent for Analytical Process

Eugene Santos Jr.John WilkinsonFei YuDeqing LiKeum Kim

Dartmouth College

Hien NguyenJacob RussellAdam Olson

University of Wisconsin - Whitewater

Presentation at AITP meeting, February 2nd 2010.

Outline

• Problem• Challenges• Our User Intent Model• Preliminary evaluation• Future work

Presentation at AITP meeting, February 2nd 2010.

Outline

• Problem• Challenges• Our User Intent Model• Preliminary evaluation• Future work

Presentation at AITP meeting, February 2nd 2010..

Problem

Presentation at AITP meeting, February 2nd 2010.

APEXB, Senior AnalystIran’s deskExpertise: Iran, Iraq, Middle EastNuclear weapon programs

APEXF, Junior Analyst

Iran’s desk

Expertise: assessment of Iranian

clerical support. APEX

L, Ad

visory

Analy

st

Iran’s

desk

Expe

rtise:

Iraq,

irania

n poli

tical

polic

ies

Problem

Presentation at AITP meeting, February 2nd 2010.

APEXB, Senior AnalystDoes not like to talk to anyone

APEXF, Junior Analyst

Face book, Second lifeAP

EXL,

Advis

ory An

alyst

Rush

es th

rough

every

thing

Problem

Presentation at AITP meeting, February 2nd 2010.

Modeling an analyst’s intent to improve the effectiveness of collaboration among intelligence

analysts• Capture the process instead of

topical interests• May help improve the diversity of

collaboration

Research Challenges

• Partiality and incompleteness of captured knowledge

• Dynamic information spaces, user foci, and preferences

• Uncertainty in analyst’s actions

Presentation at AITP meeting, February 2nd 2010.

Research Questions

1. What is user intent and how do we capture it

2. How do we evaluate the effectiveness of the intent model

Presentation at AITP meeting, February 2nd 2010.

Outline

• Problem• Challenges• Our User Intent Model• Preliminary evaluation• Future work

Presentation at AITP meeting, February 2nd 2010.

Our Intent Model• Analyst intent = Goals + Actions + Commitment• Our user model consists of 3 components that are

designed to capture intent: – Foci: What is the working space of the analyst

and what he is concentrating on? (Goal & Commitment)

– Rationale: Why does the analyst have these foci? (Goals)

– Action: How are the analyst’s goals accomplished? (Actions)

Presentation at AITP meeting, February 2nd 2010.

Our Intent Model

Presentation at AITP meeting, February 2nd 2010.

APEXFSearch

Retain

Event database

Foci Rationale Action

Example - Rationale

Presentation at AITP meeting, February 2nd 2010.

Search: Where does the Iranian clerical community stand on Ayatollah Khamenei and President Ahmadinejad's policies with regards to Iran's civilian and military nuclear program?

APEXF

12/11/2007 at 11:00

Iranian_cleric_commu

nity

community

Cleric_communit

y

iranian_communit

y

Nuclear_program

Program

isa isa

isa isa

Related_to

Iranian

Related_to

Ayatollah_khamenei

isa

Related_to

Ahmadinejad

Related_to

G0

…

Example - Foci

Presentation at AITP meeting, February 2nd 2010.

Search: Where does the Iranian clerical community stand on Ayatollah Khamenei and President Ahmadinejad's policies with regards to Iran's civilian and military nuclear program?

APEXF

12/11/2007 at 11:00

G0iranian_cleric_community

nuclear_programayatollah_khamenei

…Commitment level: 1.0

Example - Action

Presentation at AITP meeting, February 2nd 2010.

Search: Where does the Iranian clerical community stand on Ayatollah Khamenei and President Ahmadinejad's policies with regards to Iran's civilian and military nuclear program?

APEXF

12/11/2007 at 11:00 Bayesian networkHMM

1

2

3.88 .91

.82

.06

.06

.01

.06

.03

.17

Examining Evidence

Searching for Evidence

Going through Documents

Example

Presentation at AITP meeting, February 2nd 2010.

Search: Is iranian clerical community supporting Iran's nuclear program?

12/11/2007 at 11:10

Search: Are there clerics who oppose Iran's nuclear program?

APEXF12/11/2007 at 11:05

Example - Rationale

Presentation at AITP meeting, February 2nd 2010.

APEXF

Iranian_cleric_communit

y

community

Cleric_community

iranian_community

Nuclear_program

Program

isa isa

isa isa

Related_to

Iranian

Related_to

Ayatollah_khamenei

isa

Related_to

Ahmadinejad

Related_to

G0 G1 G2

G3Iran_nuclear_porogram

isa

….

Example - Foci

Presentation at AITP meeting, February 2nd 2010.

APEXF

G0iranian_cleric_community

nuclear_programayatollah_khamenei

…Commitment level: 0.33

G1

clericIran_nuclear_program

Iran…

Commitment level: 0.5

G2

clericIran_nuclear_program

Iran…

Commitment level: 0.66

G3 …

Example - Action

Presentation at AITP meeting, February 2nd 2010.

APEXF

1

2

3 .91

.82

.06

.06

.01

.06

.03

.17

Examining Evidence

Searching for Evidence

Going through Documents

HMMBayesian networks

.88

Example - Intent

• Goal:– Iranian clerics and nuclear weapons

program• Actions:

– Searches• Commitment:

– F(0.33, 0.5, 0.66)

Presentation at AITP meeting, February 2nd 2010.

APEXF

Outline

• Problem• Motivation• Our User Intent Model• Preliminary evaluation• Future work

Presentation at AITP meeting, February 2nd 2010..

Evaluation

• Do we capture user intent more precisely compared to a simple interest list approach?

• Do we capture a user’s intent earlier in the analytic process compared to the interest-based approach?

Presentation at AITP meeting, February 2nd 2010.

Evaluation

• Testbed:APEX 07, 8 intelligence analysts

• Tasks: • Where does the Iranian clerical

community stand on Ayatollah Khamenei ?• President Ahmadinejad's policies with

regards to Iran’s civilian and military nuclear program?

• Database: contain 5613 events, 8 types of actions.

Presentation at AITP meeting, February 2nd 2010.

Evaluation

Presentation at AITP meeting, February 2nd 2010.

Retain

Acces

s

Search

StartA

pp

Discard

Asses

s

MakeH

ypoth

esis

Assoc

iateE

vid...

0

0.1

0.2

0.3

0.4

0.5

0.6

Probability of Each Action in APEX Dataset

APEXBAPEXCAPEXEAPEXFAPEXHAPEXKAPEXLAPEXP

Do we capture user intent more precisely?

• Analysts: APEXL- APEXC, APEXE-APEXH APEXL- APEXK, APEXF-APEXB.

• Procedure: Create a user model for each user using 25 consecutive events on December 11, 2007.

Precision =

• Cgoals: a goal node that is found in both intent models.

Presentation at AITP meeting, February 2nd 2010.

CgoalsCgoalsrelevant _

Do we capture user intent more precisely?

Presentation at AITP meeting, February 2nd 2010.

APEXBF APEXEH APEXKL APEXLC0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

InterestIntent

P-value=0.0396<0.05

Whether we capture a user’s intent earlier?

Presentation at AITP meeting, February 2nd 2010..

• Analysts: APEXF – APEXB– APEXB: nuclear weapon program and Iran.– APEXF: grand Ayatollah

Intent (t=5)

Interest (t=8)

Future work

Presentation at AITP meeting, February 2nd 2010.

• Research and implementation:– Generation of goal descriptions– Integrating between content-based and

action-based information– Link between common intent and

collaboration effectiveness• Evaluation:

– Prediction of action sequences– Define measures of collaboration

effectiveness

Acknowledgement

Presentation at AITP meeting, February 2nd 2010.

• This research is funded by CASE program (IARPA) grant #FA8750-07-2-0219 DOD-AFRL (2007-2008). Thanks to the CASE technical advisory team for making the data available, giving constructive feedback and supporting this research.

Related work

Presentation at AITP meeting, February 2nd 2010.

• Eugene Santos Jr. and Hien Nguyen. “Modeling Users for Adaptive Information Retrieval by Capturing User Intent”. Book Chapter. In M. Chevalier, C. Julien, C. Souledupuy (Eds.). Collaborative and Social Information Retrieval and Access: Techniques for Improved User Modeling. 2009.

• Hien Nguyen and Eugene Santos Jr. (a) “An Evaluation of the Accuracy of Capturing User Intent for Information Retrieval”. In Proceedings of the 2007 International Conference on Artificial Intelligence. Las Vegas, NV. June 2007.

• Hien Nguyen and Eugene Santos Jr. (b) “Effects of prior knowledge on the effectiveness of a hybrid user model for information retrieval”. In Proceedings of the Homeland Security and Homeland Defense VI conference. Orlando, FL. March 2007. Vol. 6538.

• Hien Nguyen, Eugene Santos Jr., Aaron Schuet, and Nathan Smith. “Hybrid User Model for Information Retrieval”. In Technical Report of Modeling Others from Observations workshop at AAAI-2006 conference.

• Hien Nguyen, Eugene Santos Jr., Qunhua Zhao and Hua Wang. “Capturing User Intent for Information Retrieval”. In Proceedings of the 48th Annual Meeting for the Human Factors and Ergonomics Society (HFES-04), New Orleans, LA. 2004. Pages 371-375

Past undergraduate research project

Presentation at AITP meeting, February 2nd 2010.

• - Hybrid Approach to Word Sense Disambiguation. Jacob Russell, Hien Nguyen. Poster presented at National Undergraduate Research Conference. Lacrosse, WI. (nominated by UWW Undergraduate Research Committee to be presented at CUR on the Hill).

• Implementation of intent-based user model for business domain. Adam Olson, Nathan Lutterman, LQ McDonald III, Hien Nguyen. Poster presented at National Undergraduate Research Conference. Lacrosse, WI.

• Web-based user model services for analytic process. 2008. Nathan Lutterman, Adam Olson, Hien Nguyen. Poster presented at National Undergraduate Research Conference. Maryland, MD.

• Simulations of network defects. 2007. Russell Jacob, Nathan Smith, Nicholas Orcust, Hien Nguyen, Jim Harris, Benette Harris. Poster presented at Undergraduate research day, April 4, 2007 and National Conference for Undergraduate research (NCUR 2007), April 12-14. CA.

• Evaluation of Hybrid User Model for Information Retrieval. 2006. Aaron Schuett, Nathan Smith, and Hien Nguyen. Poster presented at National Conference of Undergraduate Research 2006, Raleigh, NC.

Current undergraduate research project

Presentation at AITP meeting, February 2nd 2010.

• Roulette game – Application of Modeling under Uncertainty. Richard Lilja, Hien Nguyen

• Intent model for supporting medical trauma treatment. LQ McDonald, Adam Olson, Hien Nguyen