PUBLIC RELEASE – DISTRIBUTION UNLIMITED SPIE 2015 The International Technology Alliance in Network...

PUBLIC RELEASE – DISTRIBUTION UNLIMITED

SPIE 2015

The International Technology Alliancein

Network and Information Sciences

Collaborative human-machine analysis using a

Controlled Natural Language

David Mott, Dave Braines, IBM UKDon Shemanski, PSU

Cheryl Giammanco, ARL

v4

2

Our Objective: Supporting the AnalystOur Objective: Supporting the Analyst

doc27doc27

doc27

Facts

Inference Rationale

Hypotheses

Query

Analysts’s Conceptual

Model

Assumptions

UncertaintyReasoning Tools

NLP

Requirements

Information

Otherdata

Referencedata

Sensor Data

Structured

Facts

Unstructured

We aim to provide techniques to support the analyst in problem solving

PUBLIC RELEASE – DISTRIBUTION UNLIMITED

Reasoning tools based on ITA Controlled EnglishReasoning tools based on ITA Controlled English

Controlled English (CE) is a Controlled Natural language, being a subset of English that is both human readable and machine interpretable

Represents the Conceptual Models defined by the users:– types of things, and the relationships between them–written language used to express facts about things and relations– logical rules allowing query and inference from facts

CE is just one component to support reasoning–Support for different problem solving strategies–Conversion to other digitised semantic languages, eg OWL–Use of rationale to express flow of reasoning–Visualisation techniques for showing CE facts Human

Reasoning

Visualisation

CNL Rationale

Digitised Semantics

PUBLIC RELEASE – DISTRIBUTION UNLIMITED 3

Controlled English: Models, Facts, AssumptionsControlled English: Models, Facts, Assumptions

there is a girl named Fiona that lives in the town Oxford and is married to the man John.

for what X is it true that the person Jane suffers from the illness X?

if ( the group A is the enemy of the group B ) and ( the group B is the enemy of the group C ) then ( the group A is the friend of the group C ).

it is assumed that the attack a1 involves the group Browngroup. PUBLIC RELEASE – DISTRIBUTION UNLIMITED 4

Analytic PitfallsAnalytic Pitfalls

Mirroring– assuming that the rest of the world shares the same motivations, goals and

concepts of rationality (including the meaning of words) as those that the analyst possesses

Layering – using assumptions made in one analysis as the factual basis for judgments in a

subsequent analysis without factoring in the uncertainties of the initial analysis

Logical Failures– failing to have the correct concepts to describe or reason about the situation

These are normal human characteristics but can sometimes lead to incorrect reasoning


The “Analysis Game” and Analytic PitfallsThe “Analysis Game” and Analytic Pitfalls

Used in a analyst’s training course to highlight analytic pitfalls:

A problem that requires logical reasoning, and interpretation of sentence meaning

Mirroring:– ambiguous terms are given meaning according to the analyst’s frame of

reference and background: “cuddly pet”, “far end of the block”, “two doors down”

Layering– not accounting for uncertainties by losing track of what assumptions have

been made: “which way round is left and right”

Seven girls live on the same street. Each of the girls has a pet and lives in a different colored house. … Using the following clues, determine who lives in each house and the pet that lives with them.

… The monkey lives between the dog and the lizard. Cassie does not own a cuddly pet but she lives next door to someone who does and that person lives in a green house.…


Analysis Game: Basic CE FormulationAnalysis Game: Basic CE Formulation

“types”, “properties”, “relations”, “rules” and “assumptions”


Problem Solving StrategyProblem Solving Strategy

Problem is specified by constraints:– Brenda and Grace live next to each other – Two people cannot live in the same house

Reasoning by elimination– “If only one house left for the fish to live in, then that is where it must live”– if ( the pet P is a cuddly pet ) then ( the pet P cannot be the pet cassies_pet ).

Represented by negative concepts– the pet fish cannot live in the house red_house.– the pet ashleys_pet cannot be the pet fionas_pet.

Use of descriptive terms to handle unknown things:– the person Brenda owns the pet brendas_pet and lives in the house brendas_house.

Solution requires identification of unknowns and propagation of their properties– the pet brendas_pet is the same as the pet monkey.– the pet brendas_pet lives in the house orange_house.

=> the pet monkey lives in the house orange_house.

Constraint Satisfaction Variables


Collaborative Problem Solving InterfaceCollaborative Problem Solving Interface

Analyst and Machine collaborates via Microsoft® Word documents and macros

Reasoning may be executed by communication between Word and CE

reasoning systemPUBLIC RELEASE – DISTRIBUTION UNLIMITED 9

Visualisation and RationaleVisualisation and Rationale

Note that Cassie’s pet must be the

lizard

So Cassie (and the lizard) must live in

the blue house!

Geographic informationRationale


Interpretation of termsInterpretation of terms

“Cuddly” pets–Is a bunny cuddly? A dog? A lizard?–We decided to categorise only those that are “definitely” cuddly

“Two doors down”–One door in between or two doors in between?–Wrong definition led to an inconsistency and needed reformulation

No sentences mention “left” or “right”–Two possible solutions, which are mirror images–Reasoning can only limit the end houses to two possible locations

Making users define their terms carefully, and analyse their logic more deeply, may help to bring out possible preconceptions and thus reduce

mirroring?


Rationale for “the girl Cassie is located on the middle plot p2”Rationale for “the girl Cassie is located on the middle plot p2”

• “Random choice” assumption made for location of the green end house

• But in reality might need to know which house Cassie actually lives in?

• By recording assumptions, we do not forget that the choice was random, if the real location is important Recording and tracking assumptions may help to retain knowledge of sources of uncertainty lying behind high value conclusions, and thus avoid

layering?PUBLIC RELEASE – DISTRIBUTION UNLIMITED 12

Mis-formulationMis-formulation

Initially “houses” were used to define the locations of girls and pets:– the house h1 is next to the house h2.– the girl Ashley lives in the house h2.

But this was VERY confusing as the location of the houses are initially unknown!

Thus we defined a “plot” to represent the fixed locations:– the plot p1 is a left end plot.– the plot p2 is immediately right of the plot p1.

Things could then be located on plots:– the girl Brenda is located in the plot p3.– the house orange_house is located on the plot p3.

This gave a more stable mental model of space!

Creating new concepts DURING

problem solving

Choosing the right conceptualisations during problem solving makes the reasoning easier to understand thus helping to reduce logical

errors


Journey to the solution – Iterative FormulationJourney to the solution – Iterative Formulation

Important discovery was in the way that the man and machine could collaborate to gain an understanding of the problem

– Trying to make the machine “just provide the answer” is difficult to understand

– More natural, human, way is to engage in an iterative “journey of discovery”:• Partial formulation of concepts• Perform the reasoning• Review of results so far• Formulate new ways to conceptualise and represent the problem space

We must model the Problem Solving process itself

Reasoning

Initial Concepts

Revised Concepts

Problem Statement

Conclusions, Inconsistencies

& RationaleAssumptions

Solution

Domain ModelFormulation


Maintaining consistency of rulesMaintaining consistency of rules

During the exploration process, new rules were added, BUT– Some were missed, some duplicated– Difficult to maintain consistency and ensure all inferences are drawn

Many relationships have a particular semantic nature which can be used to generate rules

Helps to ensure consistency and completeness of rules

the relation 'lives in' is a one-to-one-relationship.if ( the girl S lives in the house O1 ) and ( the girl S lives in the house O2 ) and ( the thing O1 # the thing O2 )then ( the house O1 is the same as the house O2 ).

if ( the girl S1 lives in the house O ) and ( the girl S2 lives in the house O ) and ( the thing S1 # the thing S2 )then ( the girl S1 is the same as the girl S2 ).

if ( the girl S1 lives in the house O1 ) and ( the girl S1 cannot be the girl S2 ) and ( the girl S2 lives in the house O2 )then ( the house O1 cannot be the house O2 ).

if ( the girl S1 lives in the house O1 ) and ( the house O1 cannot be the house O2 ) and ( the girl S2 lives in the house O2 )then ( the girl S1 cannot be the girl S2 ).

User defines the nature of a relationship

System uses meta-reasoning to construct rules automatically


Conclusion and Future WorkConclusion and Future Work

Solved some fairly complex problems by collaborative human and machine reasoning using CE

– Iterative formulation seems a useful way to structure the collaboration– Microsoft® Word is useful for creating, organising, and executing the information– New ways to perform problem solving using a CNL have been developed

Analytic pitfalls– Formulating and solving the Analysis Game encountered analytic pitfalls– Exploring how CE-based problem solving could potentially reduce pitfalls

Experimenting on how humans could use CE for problem solving, modelling, and explanation

Augmenting CE reasoning capabilities and expressiveness


Potential to reduce cognitive biases and improve critical thinking through iterative formulation and evaluation of conclusions and rationale

AcknowledgementAcknowledgement

This research was sponsored by the U.S. Army Research Laboratory and the U.K. Ministry of Defence and was accomplished under Agreement

Number W911NF-06-3-0001. The views and conclusions contained in this document are those of the author(s) and should not be interpreted as

representing the official policies, either expressed or implied, of the U.S. Army Research Laboratory, the U.S. Government, the U.K. Ministry of Defence or the U.K. Government. The U.S. and U.K. Governments are

authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation hereon.


PUBLIC RELEASE – DISTRIBUTION UNLIMITED SPIE 2015 The International Technology Alliance in Network...

Documents

Transcript of PUBLIC RELEASE – DISTRIBUTION UNLIMITED SPIE 2015 The International Technology Alliance in Network...