2014, Learning Agents Center 2014, Learning Agents Center
George Mason University and IBM ASC Symposium Fairfax, VA, 4 November 2014
Gheorghe Tecuci, Dorin Marcu, Mihai Boicu, David Schum
Learning Agents Center, Volgenau School of Engineering George Mason University, Fairfax, VA 22030
[email protected], http://lac.gmu.edu
2014, Learning Agents Center 2014, Learning Agents Center 2
We present research performed in the Learning Agents Center to develop a computational theory of evidence-based reasoning and to implement it in intelligent analytical tools, such as Disciple-CD (Disciple cognitive assistant for Connecting the Dots) and COGENT (Cognitive Assistant for Cogent Analysis), addressing the complex task of “connecting the dots” to discover knowledge from masses of data of all kinds. “Connecting the Dots” is performed through a mixed-initiative process of ceaseless discovery of evidence, hypotheses and arguments in a non-stationary world, process integrating analyst’s imagination with agent’s knowledge and evidence-based reasoning, and involving abductive, deductive, and inductive inference. The analyst and the cognitive assistant marshal thoughts and evidence to generate or discover productive competing hypotheses, use the hypotheses to discover new evidence, and construct defensible and persuasive arguments on the hypotheses believed to be most favored by the evidence that has been gathered and evaluated. We illustrate our approach to “connecting the dots” in the area of intelligence analysis, and discuss its application to other areas, including cyber insider threat, forensics, medicine, law, and natural sciences.
Summary
2014 Learning Agents Center 3
Overview
Computational Theory of Evidence-based Reasoning: Application to Intelligence Analysis
From TIACRITIS to Disciple-CD and to COGENT
“Knowledge Engineering” Book and Disciple-EBR
“Intelligence Analysis” Book and Disciple-CD
Development of Cognitive Assistants
Computational Theory of Evidence-based Reasoning: Application to other Domains
2013, Learning Agents Center
• Learn the (explicit and tacit) knowledge of subject matter experts
• Assist their users in complex problem solving and decision making
• Train junior
professionals and students
COGNITIVE ASSISTANT
COGNITIVE ASSISTANT
COGNITIVE ASSISTANT
Theory, Methodology, and Tools for the Development of Cognitive Assistants
2014 Learning Agents Center 5
Computational Theory of Evidence-based Reasoning
Explanatory Hypotheses
Observations
Probability of each Hypothesis
What is the evidence-based
probability of each hypothesis?
What evidence is entailed
by each hypothesis?
What hypothesis would explain
these observations?
Hypotheses in search of evidence
Evidence in search of hypotheses
Evidentiary testing of hypotheses
Big Data
New Evidence
Induction Deduction Abduction
2014 Learning Agents Center
Astonishing Complexity of Intelligence Analysis
Often Stunningly Complex Arguments
Masses of Evidence
imaginative reasoning
critical reasoning
incomplete inconclusive
ambiguous dissonant
various degrees of credibility
COMP
LEXI
TY
TIME
2014 Learning Agents Center 7
Sample Problem: Analysis of Wide-Area Motion Imagery
From: Mita Desai, Multi-entity activity discovery over large space-time windows, DARPA, http://www.darpa.mil/ipto/solicit/baa/BAA-09-55_ID01.pdf
Real‐Time Analysis Discover impending threat events (e.g., ambush, rocket launch, IED, suicide bomber, false check-point, kidnapping, etc.) early enough to be able to interdict them.
Forensic Analysis Backtrack from a past event (e.g., an ambush) and discover participants, possible related locations and events, and movement patterns.
2014 Learning Agents Center
Big Data
Discovery of Hypotheses
Evidence in search of hypotheses
road work not
road work
road block
E1: Evidence of road work at Al Batha
junction at 1:17am
ambush preparation
ambush threat
road repair
traffic disruption
traffic deviation
deceptive threat
What hypotheses would explain this
observation? 8
Explanatory Hypotheses
Observations
Probability of each Hypothesis
What is the evidence-based
probability of each hypothesis?
What evidence is entailed
by each hypothesis?
What hypothesis would explain
these observations?
Hypotheses insearch of evidence
Evidence in searchof hypotheses
Evidentiary testingof hypotheses
Big DataNew Evidence
2014 Learning Agents Center
Discovery of Evidence
road block
ambush preparation
Ambush threat at Al Batha highway junction around 1:17am.
&
Hypotheses in search of evidence
good location
route used by U.S. forces
ambush cover
Search for evidence that the Al Batha highway junction is on a
route used by the U.S. forces.
Search for evidence that
there is ambush cover near the Al Batha highway
junction.
people move
to cover
deployment of terrorists
people descended from vehicle
vehicle departed from terrorist facility
Search for evidence that people descended
at the Al Batha highway junction from a vehicle short before
1:17am.
Search for evidence that the vehicle that drove the
people to the Al Batha highway junction short
before 1:17am, departed from a terrorist facility.
& &
&
Assuming that this hypothesis is true, what other things should be observable?
2014, Learning Agents Center 9
Explanatory Hypotheses
Observations
Probability of each Hypothesis
What is the evidence-based
probability of each hypothesis?
What evidence is entailed
by each hypothesis?
What hypothesis would explain
these observations?
Hypotheses insearch of evidence
Evidence in searchof hypotheses
Evidentiary testingof hypotheses
Big DataNew Evidence
2014 Learning Agents Center
ambush cover
E3: Brushes, trees, ruins at Al Batha junction
Evidence-based Hypothesis Assessment
not set
almost certain
certain
very likely
no support
likely
AC
C
VL
N
L
NS
almost certain What is the probability of the link?
How certain are we that if there are brushes, trees, and ruins, then there is ambush cover?
relevance
almost certain What is the probability that the hypothesis is true?
inferential force Automatically computed using the Minimum function
Simple, intuitive, rigorous, and less error-prone probability system and evidence-based assessment
What is the probability that the evidence is true?
credibility certain
10
2014 Learning Agents Center
Discovery of Arguments
road block
ambush preparation
ambush threat
&
good location
route used ambush cover
E2: AFC confirms route E3: brushes, trees, and ruins
people move to cover
deployment of terrorists
people descended from vehicle
vehicle departed from terrorist facility
E4: People get off pickup truck
E5: Pickup truck leaves abandoned rice farm
& &
&
Evidentiary testing of hypotheses
certain
certain
certain
almost certain
almost certain
almost certain
very likely
very likely
very likely
very likely
almost certain
certain very likely
almost certain
almost certain
almost certain certain
It is very likely that there is an ambush threat to the U.S. forces at the Al Batha junction.
certain Simple
Baconian/Fuzzy composition functions: Minimum, Maximum,
On Balance 11
Explanatory Hypotheses
Observations
Probability of each Hypothesis
What is the evidence-based
probability of each hypothesis?
What evidence is entailed
by each hypothesis?
What hypothesis would explain
these observations?
Hypotheses insearch of evidence
Evidence in searchof hypotheses
Evidentiary testingof hypotheses
Big DataNew Evidence
2014 Learning Agents Center 12
Advanced Tools for Intelligence Analysis: From TIACRITIS to Disciple-CD and to COGENT
COGENT
2011-2014
2012-2016
Version 1 (Summer 2014)
Teaching Intelligence Analysts Critical Thinking Skills
Disciple Assistant for Connecting the Dots
Cognitive Agent for Cogent Analysis
New Generation Tool Easy to use Enforcing cogent analyses Learning and reuse Collaborative analysis Enabling fast analyses Customizable scale
Improvements over TIACRITIS Probability system Argument development Evidence-based reasoning Knowledge base management Usability Scalability Reliability
Disciple-CD
TIACRITIS 2009-2011
2014 Learning Agents Center 13
Cogent: Cognitive Agent for Cogent Analysis
Multiple Intelligence Questions
Answers as Hypotheses
2014 Learning Agents Center 14
N VL L M H VH F
F F VH H M L VL N
VH VH H M L VL N N
H H M L VL N N N
M M L VL N N N N
L L VL N N N N N
VL VL N N N N N N
N N N N N N N N
Supp
ort f
or H
ypot
hesis
Support for negation of Hypothesis
Favo
ring
argu
men
ts
Disfavoring arguments
Strength of Hypothesis
Strength Probability Belief
Customizable assessment scale
On balance function
2014 Learning Agents Center 15
Attaching evidence to hypothesis
2014 Learning Agents Center 16
Credibility of evidence: How high is the credibility of E1 (i.e., that Aum has
indeed created two dummy chemical companies)?
1
Strength of link: How strong is the link between what the evidence states
and the hypothesis? That is, assuming that Aum has indeed created the two
dummy chemical companies, how strong is
the hypothesis that it has a legitimate business which is
justified to acquire sarin?
2
Strength of favoring argument: What is the strength of the favoring
argument for the “legitimate business” hypothesis, based
only on E1?
3
Strength of hypothesis (based on both favoring and
disfavoring arguments)
4
Strength of upper-level hypotheses
5
Cogent assessments
Analyst assessments
2014 Learning Agents Center 17
Reuse of learned patterns
Learned patterns
2014 Learning Agents Center 18
KNOWLEDGE ENGINEERING: Building Personal Learning Assistants for Evidence-based Reasoning Introduction Evidence-based Reasoning: Connecting the Dots Methodologies and Tools for System Design and Development Modeling the Problem Solving Process Ontologies Ontology Design and Development Reasoning with Ontology and Rules Learning for Knowledge-based Systems Rule Learning Rule Refinement Abstraction of Reasoning Disciple Agents (Disciple-WA, Disciple-COA, Disciple-COG, and Disciple-VPT)
Knowledge Engineering Book (with Disciple-EBR)
Theory of knowledge engineering and evidence-based reasoning
Practice with Disciple-EBR to build learning assistants such as Disciple-CD
Examples and exercises at each chapter
2014 Learning Agents Center 19
Intelligence Analysis: “Connecting the Dots” Marshaling Thoughts and Evidence for Imaginative Analysis Disciple-CD: A Cognitive Assistant for Intelligence Analysis Evidence Divide and Conquer: A Necessary Approach to Complex Analyses Assessing the Believability of Evidence Chains of Custody Recurrent Substance-blind Combinations of Evidence Major Sources of Uncertainty in Masses of Evidence Assessing and Reporting Uncertainty: Some Alternative Methods Analytic Bias Appendices
Intelligence Analysis Book (with Disciple-CD)
Intelligence Analysis as Discovery of Evidence, Hypotheses, and Arguments: Finding and Connecting the Dots
Theory of intelligence analysis and evidence-based
reasoning
Examples and exercises at each chapter
Basic and advanced practice with Disciple-CD to assess hypotheses based on evidence
2014 Learning Agents Center
Hypotheses in search of evidence
Evidence in search of hypotheses
Evidentiary testing of hypotheses
Observation of Actions
New Observable Actions
Possible Insider Missions
Probability of Insider Missions
Cyber Insider Threat Discovery and Analysis
20
(e.g., log record of denied service access)
(e.g., covert reconnaissance, collection, and exfiltration)
2014 Learning Agents Center
Hypotheses in search of evidence
Evidence in search of hypotheses
Evidentiary testing of hypotheses
Observations of Events in Nature
New Observable Phenomena
Possible Hypotheses or Explanations
Probability of New or Revised
Theories
Natural Sciences
21
2014 Learning Agents Center
Hypotheses in search of evidence
Evidence in search of hypotheses
Evidentiary testing of hypotheses
Observations at the Site of Incident
New Potential Evidence
Possible Causes
Probability of Causes
Forensics
22
2014 Learning Agents Center
Hypotheses in search of evidence
Evidence in search of hypotheses
Evidentiary testing of hypotheses
Patient’s Complaints New Tests
Possible Illnesses
Probability of Illnesses
Personalized Medicine
23
2014 Learning Agents Center
Hypotheses in search of evidence
Evidence in search of hypotheses
Evidentiary testing of hypotheses
Observations during Fact Investigation
New Potential Evidence
Possible Charges or Complaints
Probability of Charges
Law
24
2014 Learning Agents Center 25
Key Elements of the Computational Theory of EBR
Developed in the framework of the scientific method.
Systematic approach to evidence-based reasoning through a synergistic integration of abductive, deductive, and inductive reasoning.
Computational models for essential analytical tasks ( e.g., evidence marshaling, hypothesis-driven evidence collection, multi-INT fusion, detection and mitigation of bias).
General analysis structure with favoring and disfavoring arguments for competing hypotheses.
Intuitive system of Baconian probabilities with Fuzzy qualifiers, allowing customizable assessment scales.
Substance-blind ontology of evidence.
General procedures for credibility assessment.
Context-based rules learned from expert analysis examples.
2014 Learning Agents Center 26
Questions
Contact information Gheorghe Tecuci Professor of Computer Science and Director of the Learning Agents Center MSN 6B3, Learning Agents Center, George Mason Univ., Fairfax, VA 22030 [email protected] tel 703 993 1722 http://lac.gmu.edu/
Top Related