InteractiveStoryAuthoringforKnowledge‐BasedSupportforInvestigativeAnalysisASecondSTABatMakingSenseofVASTData

AshokKGoelAvikSinharoySummerAdamsampAdityDokania

DesignampIntelligenceLaboratorySchoolofInteractiveComputingGeorgiaInstituteofTechnology

AtlantaGA30332USA

Abstract The sensemaking task in investigative analysis generates stories that connect entitiesandevents inan inputstreamofdata TheStabsystemrepresentscrimestoriesashierarchicalscriptswithgoalsandstatesItgeneratesmultiplestoriesasexplanatoryhypothesesforaninputdata stream containing interleaved sequences of events recognizes intent in a specific eventsequence and calculates confidence values for the generated hypotheses In this report wedescribeStab2anewinteractiveversionoftheknowledge‐basedStabsystemStab2containsastoryeditorthatenablesuserstoenterandeditcrimestoriesWeillustrateStab2withexamplesfromtheIEEEVASTcontestdatasets

Keywords Investigative Analysis Sense Making Story Understanding Plan RecognitionHierarchicalScriptsIntelligenceAnalysis

ThisisGeorgiaTechGVUTechnicalReportGIT‐GVU‐10‐02Foradditionalinformationpleasecontactthefirstauthoratgoelccgatechedu

Goel A K Sinharoy A Adams S amp Dokania A (2010) Interactive Story Authoring for Knowledge-Based Support for Investigative Analysis A Second STAB at Making Sense of VAST Data GVU Technical Report GIT-GVU-10-02

InteractiveStoryAuthoringforKnowledge‐BasedSupportforInvestigativeAnalysisASecondSTABatMakingSenseofVASTData

AshokKGoelAvikSinharoySummerAdamsampAdityDokania

Section1Introduction

Intelligence analysis investigative analysis and other related forms of information analysis share manycommoncharacteristicsandcomponentsOneunifyingelementinthevarioustypesofinformationanalysisisthetaskofsensemaking(Bodnar2005Heuer1999KleinMoonampHoffman2006Krizan1999PirolliampCard2005ThomasampCook2005)generationofamodelofasituationthatconnectsentitiesandeventsinaninputstreamofdataaboutthesituation(sometimescolloquiallycalledtheldquoconnectthedotsrdquoproblem)Theinputtothe sensemaking task in different types of information analysis is characterized by the same kinds ofcharacteristics the amount of data in the input stream is huge data comes frommultiple sources and inmultipleformsdatafromvarioussourcesmaybeunreliableandconflictingdataarrivesincrementallyandisconstantlyevolvingdatamaypertaintomultipleactorswheretheactionsofthevariousactorsneednotbecoordinatedtheactorsmaytrytohidedataabouttheiractionsandmayevenintroducespuriousdatatohidetheir actions datamay pertain to novel actors aswell as rare or novel actions and the amount of usefulevidence typically is a small fraction of the vast amount of data (the colloquial ldquoneedle in the haystackrdquoproblem)Thedesiredoutputof the sensemaking task indifferent typesof informationanalysis toohas thesamekindsofcharacteristicsmodelsthatexplaintheconnectionsamongtheentitiesandeventsthatspecifythe intentofthevariousactorsthatmakeverifiablepredictionsandthathaveconfidencevaluesassociatedwiththemPsychological studiesof sensemaking in intelligenceanalysis (Heuer1999) indicate thatcognitive limitationsandbiasesofhumananalystsresultinseveralkindsoferrorsThethreemainerrorsmadebyhumananalystsinhypothesisgenerationare(Heuer1999)(1)Duetolimitationsofhumanmemoryanalystsmayhavedifficultykeeping track of multiple explanatory hypotheses for a set of data over a long period of time (2) Due tocognitivefixationanalystsmayquicklydecideonasinglehypothesisforthedatasetandsticktoitevenasnewdata arrives (cognitive fixation) (3) Due to confirmation bias analystsmay look for data that supports thehypothesis onwhich they are fixated and not necessarily the data thatmay refute the hypothesis Thus ascientific and technological challenge for cognitive science artificial intelligence and human‐centeredcomputing is to develop interactive computational tools that can help human analysts overcome thesecognitivelimitations

Overthelastfewyearsseveralresearchershaveexploredtheuseofknowledge‐basedtechniquestosupporthumandecision‐makinginintelligenceandinvestigativeanalysis(Birnbaumetal2005Chenatal2003JarvisLuntampMyers2004MurdockAhaampBreslow2003PARC2004Sanfilippoetal2007Tecucietal2008Weltyetal2005Whitakeretal2004)Inourpreviousworkwehavedevelopedanautomatedknowledge‐basedsystemcalledStab(AdamsampGoel2007a2007b2008GoelAdamsCutshawampSugandh2009)forIEEEVAST2006and2007datasets(IEEEVAST2006IEEEVAST2007Plaisantetal2008)BrieflyStabcontainsalibraryofhierarchicallyorganizedabstractstoriesthatcapturepatternsofcriminalactivityintheIEEEVAST2006and2007datasetsThusinStabthemodelsthatconnecttheentitiesandeventsintheinputdataarecrimestoriesThe crime stories are represented in the TMKL knowledge representation language (MurdockampGoel 20032008) Stab takesas inputa setofeventsextracted from the IEEEVAST2006and2007datasets It givesasoutput a set of multiple competing instantiated stories as explanatory hypotheses that connect the inputeventsandascribegoalstoactorsalongwithconfidencevaluesforeachhypothesisaswellasasummaryofevidenceforeachhypothesisStabrsquosuserinterfaceallowstheusertoviewthelogicalstructureofagenerated

hypothesis inagraphicalrepresentation Italsoenablesselectionofportionsoftheevidence insupportofageneratedhypothesis for further analysisof theexplanatory relationshipbetween thegeneratedhypothesisandthesupportingevidenceAhumananalystmayuseStabasanexternalmemoryofdataandexplanatoryhypothesesInadditiontheanalystmayuseStabrsquosresultstoinformadditionalsearchforfurtherevidencefororagainstageneratedhypothesis

OfcourseStabalsohasseverallimitationsOneofthemajorlimitationsofStabisthatcrimestoriesinitsstorylibraryhavetobehandcoded inTMKLThismakes itdifficult forausertoenternewstoriesoreditexistingstoriesinStabrsquoslibraryToenableausertomoreflexiblyinteractwithStabwehavedevelopedStab2anewinteractiveversionofStabStab2containsastoryeditorthatenablesuserstoenterandeditcrimestoriesinagraphical notation Stab2 automatically converts a new (or modified) story into its internal knowledgerepresentation language(TMKL)Thisenablesusers to interactwithStabrsquosstoriesldquoonlinerdquo InthispaperwefirstbrieflydescribethebasicsofStabandthenpresentStab2rsquosstoryeditor

Section2Knowledge‐BasedSupportforSensemaking

InthissectionwebrieflydescribetheoriginalStabsystemforsensemakingoftheIEEEVASTcontestdatasetsAlthoughStabrsquosreasoningknowledgeandrepresentationsareintendedtobegeneralStabrsquoslibraryatpresentcontainsonlycrimestoriesrelevanttotheVAST‐2006andVAST‐2007datasets

VAST Datasets The VAST datasets are synthetic datasets generated by the Pacific Northwest NationalLaboratories forsupportingresearchanddevelopment intheemergingfieldofvisualanalytics (ThompsonampCook 2005) The datasets pertain to fictitious illegal and unethical activities as well as normal and typicalactivities in a fictitious town in the United States The VAST‐2006 dataset contains over a thousand newsstorieswritteninEnglishandascoreoftablesmapsandphotographsFigure1 illustratesanexamplenewsstoryfromtheVAST2006datasetTheVAST‐2007dataset isa little largerandslightlymorecomplicatedbutsimilarinnature

Inputs to StabWemanually screened the dataset for stories that indicated an illegal or unethical activitywhich leftaboutahundrednewsstoriesoutofthemorethanathousandoriginally inthedatasetWethenmanuallyextractedeventsandentitiespertainingtoillegalunethicalactivitiesTheseeventsentitiesformtheinput to STAB We also hand crafted representations for each event in terms of the knowledge states itproducesInadditionweexaminedthemapsphotosandtablesthatarepartoftheVASTdatasetandsimilarlyextracted and represented the relevant information about various entities Table 1 illustrates a sample ofinputstoSTABalongwiththeresultingknowledgestatecreatedbyaninputevent

Figure1ExamplenewsstoryfromtheVAST‐2006dataset

(Adaptedfrom(httpwwwcsumdeduhcilVASTcontest06)Table1AsampleofinputstoStab

SampleSTABInputs ResultingState

stolen(money$40Highway‐Tire‐Store) Has‐object

cured‐disease(Boynton‐LabsPhilip‐Boyntonprion‐disease) Is‐rich‐and‐famous

named‐after(labPhilip‐BoyntonDean‐USC) Expert‐involved

was‐founded(Boynton‐Labs) Is‐open

have‐developed(Boynton‐Labsprion‐disease) Exists‐new‐disease

announced‐investigation(USFDABoynton‐Labs) Is‐investigating

Injected‐cow(Boynton‐Labsprion‐disease) Cow‐is‐infected

treatment‐cow(Boynton‐Labsprion‐disease) Cow‐is‐cured

Stabrsquos Library of Crime Stories STAB contains a small library of hierarchically organized abstract stories orhierarchicalscriptsthatcapturethepatternsofcrimesthatoccurintheVAST‐2006andVAST‐2007datasetsFigure2illustratesasimplescriptinSTABrsquoslibraryRobaStoreThisscripthasthegoalofachievingthestateofHaveMoneygiventheinitialstateofNotHaveMoney(topoffigure)Thisgoal(ortask)accordingtotheRobaStorescriptisachievedbyaplan(ormethod)thathasseveralactions(orevents)initGotoStoreBreakintoStore TakeMoney (middle of the figure) Each of these events becomes a task at the next lower level ofabstraction in the hierarchical script and each task can be (potentially) achieved bymultiplemethods ForexampleaccordingtotheRobaStorescriptthetaskofBreakintoStorecanbeachievedbyEnteringthroughaWindow or Entering through a Door (bottom of figure) Each of thesemethods in turn specifies a processconsistingofmultipleeventsandsoon

STABrsquos hierarchical scripts explicitly represent both goal and state at multiple levels of abstraction Whilerepresentation of the state caused by an event is useful for inferring causality representation of goals ofsequences of events is useful for inferring intention Figure 3 illustrates a more complex script of politicalconspiracyinwhichapoliticalfiguremaygetanopponentoutofanelectoralraceeitherbyexposingdirtonhim(politicalblackmail)orhavinghimassassinatedNotethisscriptiscomposedofseveralsmallerscripts

We found that sevenhierarchical scripts appear to cover all the illegalunethical activities in theVAST‐2006datasetandthatanotherfourhierarchicalscriptsapparentlyaresufficienttocovertheVAST‐2007aswellWehandcraftedthislibraryofscriptsintoSTAB

Figure2ThecontentandstructureofastoryinSTAB1(AdaptedfromGoeletal2009)

(

Stabrsquos Knowledge Representation STABrsquos scripts are represented in the TMKL language A task in TMKL isdefinedbyinputknowledgeelementsoutputknowledgeelementsrequiredinputconditions(pre‐conditions)desiredoutputconditions (post‐conditions)andmethods for implementing the taskAmethod isdefined intermsofsubtasksofthetaskandorderingofthesubtasksandisrepresentedbyafinitestatemachineFinallyknowledge in TMKL is defined in terms of domain concepts and relationships among them the input andoutputknowledgeelements ina taskspecificationrefer tothesedomainconceptsandrelationsMurdockampGoel2008providesdetailsofTMKL

Figure3ThestoryforapoliticalconspiracyintendedtoremoveanopponentfromanelectoralraceActivatednodesaredenotedbyathickoutlinearoundyellowboxes(AdaptedfromGoelet

al2009)

Figure4High‐LevelArchitectureofSTAB(AdaptedfromGoeletal2009)

t ba SV Ts t t SA D t tsD u sDr a H snnr boao b o Tb l ampcr oo sl I n a eso or a u saboC oe ys I l nn ol a l nn ob oe sI cr o y I ob sI I ys I sn sI eal I l nl Ds n l a au SoC oe ol ah o e a og b l I sI c r o y I o o osamp I r b b sob a br nosI D gI l n D boo l oe y I o soe oe ob g I l b sI oe 13 a ca b I oo sl I b l oe b ascob bol a sI oe ol ah 13 s aa hu e ol ah o e a oD b oe amp o esI D ob gb I c b b boe amp o esI D b ascob ol l agsI D amp l ahu esb o DDsI D l amp o esI D ob gb sI o sI boI oso b oe ob gu e ol ah o e a oe I sI bc ob oe I So sI cr o y I o sI oe ys I sn I a c o b oe l y cal bbu oe I sI cr o y I o a br nob sI oe a oas yn l I b ascoC oe I oe b asco sb bsampsna nh bol a sI oe l agsI D amp l ahu oe I nh a oas y b asco sb n a h sI oe l agsI D amp l ahC oe I sosl I n ob g I l b oeo amp o e oe I sI cr o a n bl oD D r o l I nh l I b asco sI boI sb g cou

t ba s tTs tau o l r ocr ob l r a gsI b l sI l aamp o sl I p n n sI boI oso b ascob sI oe l agsI D amp amp l ahC n n I l bsI e br e b asco oeo a amp o e soe I sI cr o y I oC n n ys I l a e br e b ascoC I l I s I yn r l a e br e b ascou e h nnl l S b sI sDr a H snnr boao oe ob gb sI oe c l nsos n l I bcsa h b asco oeo a amp o e soe I sI cr o y I o sI l I ar I I sI D l o u e l I s I yn r l a b asco sb b l I oe I r amp a l ob gb sI b asco oeo a amp o e soe sI cr o y I obu l n o n u dPVV cal ys os nb l o Tba b l I sI DC gI l n D C a ca b I oo sl I b I l r ocr obu

c t n py nc r c v ct ro c n n c s

e sI o a osy o P bhbo amp sb I r Damp I oo sl I l oe l asDsI n o bhbo amp soe casamp a h l r b l I sampcal ysI Dr b a sI o a osl I u e bc ss Dl n b l o P I eI amp I ob sI nr p ds nnl l I nsI r oel asI D I sosI D l o Tb asamp bol as b sI ysbr n l aampC I dss I n r b a ol l I oal n n n b c ob l o Tb a b l I sI D oeal r De Dac es n sI o a u

i r Ty t I ei c c t n r c ci r t c c sua ct ro i c t r n t t

o s h

sDr a O snnr boao b oe l ampcr oo sl I n a eso or a l o PTb bol ah r oel asI D ol l nu e ol ah sol a sb r b sI a o sI D I amp I D sI D oe o PTb asamp bol as bu e b bol as b a sa o b I I o l oe es aa es n b ascob r b sI o u l y aC esn o PTb bol as b bosnn r b 13 b oe gI l n D a ca b I oo sl I nI Dr D C e I D oe sI o aI n o amp l n sI o P ol I n r ol amp o s oaI bno sl I l bol ah r oel a h r b a sI ol oe 13 gI l n D a ca b I oo sl I u

e I l n D sol a ol l sb r sno sI ol o PTb sI o a C I eI n b r b a bb ol oe gI l n D n amp I ob sI oe 13 a ca b I oo sl I l bol ahu o cal ys b oe r b a l I oal n l y a sI sI D oe I osos b sI bol ah b nn b oe a no sl I b amp l I D oe I osos bu e b l I a sI sDr a O a ab ol o b as sI oe ca ysl r b b osl I u

c t n pAsy ai onc S t ct ro i c t r n c s

sDr a ( snnr boao b oe ysbr n bhI oS r b h o PTb bol ah sol au e ob g sb a ca b I o b a oI Dn u b oe r b a amp l r b b l y a ob gC oe ol ah sol a sbcnh b oe 13 bc ss o sl I l oe ob gu e amp oel sb a ca b I o b b L r I l br ob gb oeo bc ss b oe l a asI D l oe br ob gb sI nr sI D so a osl I u ebr ob g sI or aI sb bc ss b a oI Dn soe sob l I amp oel b b oe I So nl a n y n l boa osl I u

i r My ai onc S t c ct ro ct r n c s

o o 13 h

sDr a B snnr boao b o PTb ol ah sol a sI o a u e amp s I c a o l oe sDr a l I os I b oe bol ah sI D so C sI oesb b oe l ol a b asco snnr boao sI sDr a Pu e c n oo l I oe ol c asDeo l sDr a B cal ys b oe s l I b l a sI D I ob g I amp oel n amp I ob ol oe bol ahu e sI l l I oe l ool amp n o c s ob oe I l n D sol au e sI l l I oe l ool amp asDeo sI s o b l oe a bol as b sI oe ol ah 13 s aa h I I n b oe r b a ol al b oe ns aa hu e Dac e sI oe l ool amp amps n I n b oe r b a ol I y sDo oe b asco l bol ahu

irkycs

uactroctrncr

TheStoryEditor inStab2gives theusermorecontrol thanwaspossible inStab In theoriginalStabsystemhierarchical scripts were hard‐coded in TMKL that itself is coded in the Lisp programming knowledge Thisresulted inseveraldifficulties regarding theusabilityof thesystemFirstly thestoredscripts inStabarenotavailableforreferenceorreviewexceptbyperusingtheactualLispcodeThislackoftransparencycanleadtoinaccurateandimpreciseunderstandingofthestoriesthataregeneratedbyStabasexplainingtheinputdataStab2rsquosStoryEditorprovidesvisualaccesstoboththelibraryofstoredscriptsaswellasthestoriesinstantiatedwhenStab2isrunforaninputdataset

SecondlysinceStabrsquosstoredscriptsarecodedinTMKLitisverydifficultforausertoauthoranewscriptorstoryevaluatethecorrectnessofastoredscriptandmakemodificationstoitStab2StoryEditorwithitsvisualinterface makes these tasks much easier though editing the knowledge elements in a story through theKnowledgeEditorstillisnon‐trivial

FinallytheinitialversionStabdidnotallowmodificationstothestructureofexistingscriptsatruntimeThisimpliesthat it isusefultorunStabjustonceonaninputdatasetHowever ingeneralahumananalystmaywanttorunStabseveraltimesndasheachwithdifferingvariationsofthescripts‐astheanalystrsquosinsightintothecriminal patterns evolves This is perhaps themost important side effect of the use of an interactive StoryEditor in Stab2 Stab2 enables an analyst to conduct ldquowhat ifrdquo simulations Using Stab2rsquos Story Editor andKnowledge Editor illustrated in Figure 7 an analystmay inspectmodify activate or deactivate the scriptsstored in Stab2rsquos Story Library the task and method elements in an individual script or the knowledgeelements likedomainconceptsand relationsTheanalystmay then runStab2 to simulate theeffectsof thepreciseconditionsshecreatedinStab2rsquosvirtualworld

Section34StoryAuthoringinStab2

Whenananalystwants tocreateanewstoryheneeds toopentheeditorandclickonaddstory iconThisopensawhiteareawithapaletteontherightWestartwithpickingtheldquoMainstoryldquoiconfromthepaletteanddropitintothewhiteboardThisrepresentsthenameofthestoryandisdepictedinFigure8(a)AfterthiswedragthebluerectanglethatreadsldquoIntermediaryGoalsrdquoandconnectthistothemainstoryusingaSub‐Goalconnector all ofwhich can be found in the palette The result is shown in Figure 8(b) Once themethod iscreatedwefillitinwithactionsperformedtoachievethatgoalThisisdoneusingtheldquoGoalsActionrdquoiconinthepaletteandthisisshowninFigure8(c)ThedraganddropcapabilitymakesitveryeasyforananalysttoconstructanewstoryplotThetreecangodownasmanylevelsdependingonthecomplexityofthestoryTherelationshipsbetweeneachofthegoalsarecapturedusingthelinksTwokindsoflinkagesarepossiblendashSub‐Goals that suggests that one goal is a child of another and Causal Connection that indicates relationshipsbetweendifferentactionsinsideeachintermediarygoal

Figure8(a)1

stSna

psho

tofStoryAutho

ringinStab2

ndashHerethehu

mananalystdragsth

eicon

mainstoryfrom

thepalette

andplacesitinth

eed

itorareaand

nam

esitasldquostoryplotrdquo

Figure8(b)A

2ndSna

psho

tofStoryAutho

ringin

Stab2

ndashHerethehu

mananalystdragsth

eInterm

ediaryGoalsicon

and

placesitbe

lowth

etopleveltaskAlsotheanalystdragsthesub‐goalslinkagefromthe

paletteand

con

nectsthetw

oblocksand

nam

esthe

bluerectangleas

ldquoMetho

d1rdquo

Figure8(c)ASna

psho

tofStoryAutho

ringinStab2

‐Heretheanalystd

ragsth

eGoalsA

ctionsicon

inside

thealreadycreatedmetho

dHe

then

con

nectstheseactio

nswith

thecausalcon

nectionslinkageinth

epaletteandnameseachactio

n

BehindtheScenes

Figure9(a)A

Sna

psho

tofaXMLfileforeachStoryPlot

Figure9(b)A

Sna

psho

tofaXMLfileforeachStoryPlotndash

Thisim

agehighlightsindividu

alta

gsThe

imageon

the

righth

ighlightsthechildTaskta

gandtheon

ethelefthighlightstheallM

etho

dsta

g

EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)

RelatedWork

ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors

StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)

Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo

Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)

CurrentWork

ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)

Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities

Acknowledgements

Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories

References

AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007

AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8

Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008

BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005

BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005

Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991

ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17

CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985

Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34

CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum

Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)

FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991

Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995

Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009

GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia

Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13

HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)

HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07

JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863

Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009

JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994

Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92

KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999

Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity

LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007

Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180

Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey

MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey

MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691

Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340

Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003

Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81

Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36

PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004

PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005

Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21

RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007

RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318

RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262

RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006

SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924

SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress

SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum

StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132

TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)

ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety

Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623

Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA

WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370

Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics

InteractiveStoryAuthoringforKnowledge‐BasedSupportforInvestigativeAnalysisASecondSTABatMakingSenseofVASTData

AshokKGoelAvikSinharoySummerAdamsampAdityDokania

Section1Introduction

Intelligence analysis investigative analysis and other related forms of information analysis share manycommoncharacteristicsandcomponentsOneunifyingelementinthevarioustypesofinformationanalysisisthetaskofsensemaking(Bodnar2005Heuer1999KleinMoonampHoffman2006Krizan1999PirolliampCard2005ThomasampCook2005)generationofamodelofasituationthatconnectsentitiesandeventsinaninputstreamofdataaboutthesituation(sometimescolloquiallycalledtheldquoconnectthedotsrdquoproblem)Theinputtothe sensemaking task in different types of information analysis is characterized by the same kinds ofcharacteristics the amount of data in the input stream is huge data comes frommultiple sources and inmultipleformsdatafromvarioussourcesmaybeunreliableandconflictingdataarrivesincrementallyandisconstantlyevolvingdatamaypertaintomultipleactorswheretheactionsofthevariousactorsneednotbecoordinatedtheactorsmaytrytohidedataabouttheiractionsandmayevenintroducespuriousdatatohidetheir actions datamay pertain to novel actors aswell as rare or novel actions and the amount of usefulevidence typically is a small fraction of the vast amount of data (the colloquial ldquoneedle in the haystackrdquoproblem)Thedesiredoutputof the sensemaking task indifferent typesof informationanalysis toohas thesamekindsofcharacteristicsmodelsthatexplaintheconnectionsamongtheentitiesandeventsthatspecifythe intentofthevariousactorsthatmakeverifiablepredictionsandthathaveconfidencevaluesassociatedwiththemPsychological studiesof sensemaking in intelligenceanalysis (Heuer1999) indicate thatcognitive limitationsandbiasesofhumananalystsresultinseveralkindsoferrorsThethreemainerrorsmadebyhumananalystsinhypothesisgenerationare(Heuer1999)(1)Duetolimitationsofhumanmemoryanalystsmayhavedifficultykeeping track of multiple explanatory hypotheses for a set of data over a long period of time (2) Due tocognitivefixationanalystsmayquicklydecideonasinglehypothesisforthedatasetandsticktoitevenasnewdata arrives (cognitive fixation) (3) Due to confirmation bias analystsmay look for data that supports thehypothesis onwhich they are fixated and not necessarily the data thatmay refute the hypothesis Thus ascientific and technological challenge for cognitive science artificial intelligence and human‐centeredcomputing is to develop interactive computational tools that can help human analysts overcome thesecognitivelimitations

Overthelastfewyearsseveralresearchershaveexploredtheuseofknowledge‐basedtechniquestosupporthumandecision‐makinginintelligenceandinvestigativeanalysis(Birnbaumetal2005Chenatal2003JarvisLuntampMyers2004MurdockAhaampBreslow2003PARC2004Sanfilippoetal2007Tecucietal2008Weltyetal2005Whitakeretal2004)Inourpreviousworkwehavedevelopedanautomatedknowledge‐basedsystemcalledStab(AdamsampGoel2007a2007b2008GoelAdamsCutshawampSugandh2009)forIEEEVAST2006and2007datasets(IEEEVAST2006IEEEVAST2007Plaisantetal2008)BrieflyStabcontainsalibraryofhierarchicallyorganizedabstractstoriesthatcapturepatternsofcriminalactivityintheIEEEVAST2006and2007datasetsThusinStabthemodelsthatconnecttheentitiesandeventsintheinputdataarecrimestoriesThe crime stories are represented in the TMKL knowledge representation language (MurdockampGoel 20032008) Stab takesas inputa setofeventsextracted from the IEEEVAST2006and2007datasets It givesasoutput a set of multiple competing instantiated stories as explanatory hypotheses that connect the inputeventsandascribegoalstoactorsalongwithconfidencevaluesforeachhypothesisaswellasasummaryofevidenceforeachhypothesisStabrsquosuserinterfaceallowstheusertoviewthelogicalstructureofagenerated

hypothesis inagraphicalrepresentation Italsoenablesselectionofportionsoftheevidence insupportofageneratedhypothesis for further analysisof theexplanatory relationshipbetween thegeneratedhypothesisandthesupportingevidenceAhumananalystmayuseStabasanexternalmemoryofdataandexplanatoryhypothesesInadditiontheanalystmayuseStabrsquosresultstoinformadditionalsearchforfurtherevidencefororagainstageneratedhypothesis

OfcourseStabalsohasseverallimitationsOneofthemajorlimitationsofStabisthatcrimestoriesinitsstorylibraryhavetobehandcoded inTMKLThismakes itdifficult forausertoenternewstoriesoreditexistingstoriesinStabrsquoslibraryToenableausertomoreflexiblyinteractwithStabwehavedevelopedStab2anewinteractiveversionofStabStab2containsastoryeditorthatenablesuserstoenterandeditcrimestoriesinagraphical notation Stab2 automatically converts a new (or modified) story into its internal knowledgerepresentation language(TMKL)Thisenablesusers to interactwithStabrsquosstoriesldquoonlinerdquo InthispaperwefirstbrieflydescribethebasicsofStabandthenpresentStab2rsquosstoryeditor

Section2Knowledge‐BasedSupportforSensemaking

InthissectionwebrieflydescribetheoriginalStabsystemforsensemakingoftheIEEEVASTcontestdatasetsAlthoughStabrsquosreasoningknowledgeandrepresentationsareintendedtobegeneralStabrsquoslibraryatpresentcontainsonlycrimestoriesrelevanttotheVAST‐2006andVAST‐2007datasets

VAST Datasets The VAST datasets are synthetic datasets generated by the Pacific Northwest NationalLaboratories forsupportingresearchanddevelopment intheemergingfieldofvisualanalytics (ThompsonampCook 2005) The datasets pertain to fictitious illegal and unethical activities as well as normal and typicalactivities in a fictitious town in the United States The VAST‐2006 dataset contains over a thousand newsstorieswritteninEnglishandascoreoftablesmapsandphotographsFigure1 illustratesanexamplenewsstoryfromtheVAST2006datasetTheVAST‐2007dataset isa little largerandslightlymorecomplicatedbutsimilarinnature

Inputs to StabWemanually screened the dataset for stories that indicated an illegal or unethical activitywhich leftaboutahundrednewsstoriesoutofthemorethanathousandoriginally inthedatasetWethenmanuallyextractedeventsandentitiespertainingtoillegalunethicalactivitiesTheseeventsentitiesformtheinput to STAB We also hand crafted representations for each event in terms of the knowledge states itproducesInadditionweexaminedthemapsphotosandtablesthatarepartoftheVASTdatasetandsimilarlyextracted and represented the relevant information about various entities Table 1 illustrates a sample ofinputstoSTABalongwiththeresultingknowledgestatecreatedbyaninputevent

Figure1ExamplenewsstoryfromtheVAST‐2006dataset

(Adaptedfrom(httpwwwcsumdeduhcilVASTcontest06)Table1AsampleofinputstoStab

SampleSTABInputs ResultingState

stolen(money$40Highway‐Tire‐Store) Has‐object

cured‐disease(Boynton‐LabsPhilip‐Boyntonprion‐disease) Is‐rich‐and‐famous

named‐after(labPhilip‐BoyntonDean‐USC) Expert‐involved

was‐founded(Boynton‐Labs) Is‐open

have‐developed(Boynton‐Labsprion‐disease) Exists‐new‐disease

announced‐investigation(USFDABoynton‐Labs) Is‐investigating

Injected‐cow(Boynton‐Labsprion‐disease) Cow‐is‐infected

treatment‐cow(Boynton‐Labsprion‐disease) Cow‐is‐cured

Stabrsquos Library of Crime Stories STAB contains a small library of hierarchically organized abstract stories orhierarchicalscriptsthatcapturethepatternsofcrimesthatoccurintheVAST‐2006andVAST‐2007datasetsFigure2illustratesasimplescriptinSTABrsquoslibraryRobaStoreThisscripthasthegoalofachievingthestateofHaveMoneygiventheinitialstateofNotHaveMoney(topoffigure)Thisgoal(ortask)accordingtotheRobaStorescriptisachievedbyaplan(ormethod)thathasseveralactions(orevents)initGotoStoreBreakintoStore TakeMoney (middle of the figure) Each of these events becomes a task at the next lower level ofabstraction in the hierarchical script and each task can be (potentially) achieved bymultiplemethods ForexampleaccordingtotheRobaStorescriptthetaskofBreakintoStorecanbeachievedbyEnteringthroughaWindow or Entering through a Door (bottom of figure) Each of thesemethods in turn specifies a processconsistingofmultipleeventsandsoon

STABrsquos hierarchical scripts explicitly represent both goal and state at multiple levels of abstraction Whilerepresentation of the state caused by an event is useful for inferring causality representation of goals ofsequences of events is useful for inferring intention Figure 3 illustrates a more complex script of politicalconspiracyinwhichapoliticalfiguremaygetanopponentoutofanelectoralraceeitherbyexposingdirtonhim(politicalblackmail)orhavinghimassassinatedNotethisscriptiscomposedofseveralsmallerscripts

We found that sevenhierarchical scripts appear to cover all the illegalunethical activities in theVAST‐2006datasetandthatanotherfourhierarchicalscriptsapparentlyaresufficienttocovertheVAST‐2007aswellWehandcraftedthislibraryofscriptsintoSTAB

Figure2ThecontentandstructureofastoryinSTAB1(AdaptedfromGoeletal2009)

(

Stabrsquos Knowledge Representation STABrsquos scripts are represented in the TMKL language A task in TMKL isdefinedbyinputknowledgeelementsoutputknowledgeelementsrequiredinputconditions(pre‐conditions)desiredoutputconditions (post‐conditions)andmethods for implementing the taskAmethod isdefined intermsofsubtasksofthetaskandorderingofthesubtasksandisrepresentedbyafinitestatemachineFinallyknowledge in TMKL is defined in terms of domain concepts and relationships among them the input andoutputknowledgeelements ina taskspecificationrefer tothesedomainconceptsandrelationsMurdockampGoel2008providesdetailsofTMKL

Figure3ThestoryforapoliticalconspiracyintendedtoremoveanopponentfromanelectoralraceActivatednodesaredenotedbyathickoutlinearoundyellowboxes(AdaptedfromGoelet

al2009)

Figure4High‐LevelArchitectureofSTAB(AdaptedfromGoeletal2009)

t ba SV Ts t t SA D t tsD u sDr a H snnr boao b o Tb l ampcr oo sl I n a eso or a u saboC oe ys I l nn ol a l nn ob oe sI cr o y I ob sI I ys I sn sI eal I l nl Ds n l a au SoC oe ol ah o e a og b l I sI c r o y I o o osamp I r b b sob a br nosI D gI l n D boo l oe y I o soe oe ob g I l b sI oe 13 a ca b I oo sl I b l oe b ascob bol a sI oe ol ah 13 s aa hu e ol ah o e a oD b oe amp o esI D ob gb I c b b boe amp o esI D b ascob ol l agsI D amp l ahu esb o DDsI D l amp o esI D ob gb sI o sI boI oso b oe ob gu e ol ah o e a oe I sI bc ob oe I So sI cr o y I o sI oe ys I sn I a c o b oe l y cal bbu oe I sI cr o y I o a br nob sI oe a oas yn l I b ascoC oe I oe b asco sb bsampsna nh bol a sI oe l agsI D amp l ahu oe I nh a oas y b asco sb n a h sI oe l agsI D amp l ahC oe I sosl I n ob g I l b oeo amp o e oe I sI cr o a n bl oD D r o l I nh l I b asco sI boI sb g cou

t ba s tTs tau o l r ocr ob l r a gsI b l sI l aamp o sl I p n n sI boI oso b ascob sI oe l agsI D amp amp l ahC n n I l bsI e br e b asco oeo a amp o e soe I sI cr o y I oC n n ys I l a e br e b ascoC I l I s I yn r l a e br e b ascou e h nnl l S b sI sDr a H snnr boao oe ob gb sI oe c l nsos n l I bcsa h b asco oeo a amp o e soe I sI cr o y I o sI l I ar I I sI D l o u e l I s I yn r l a b asco sb b l I oe I r amp a l ob gb sI b asco oeo a amp o e soe sI cr o y I obu l n o n u dPVV cal ys os nb l o Tba b l I sI DC gI l n D C a ca b I oo sl I b I l r ocr obu

c t n py nc r c v ct ro c n n c s

e sI o a osy o P bhbo amp sb I r Damp I oo sl I l oe l asDsI n o bhbo amp soe casamp a h l r b l I sampcal ysI Dr b a sI o a osl I u e bc ss Dl n b l o P I eI amp I ob sI nr p ds nnl l I nsI r oel asI D I sosI D l o Tb asamp bol as b sI ysbr n l aampC I dss I n r b a ol l I oal n n n b c ob l o Tb a b l I sI D oeal r De Dac es n sI o a u

i r Ty t I ei c c t n r c ci r t c c sua ct ro i c t r n t t

o s h

sDr a O snnr boao b oe l ampcr oo sl I n a eso or a l o PTb bol ah r oel asI D ol l nu e ol ah sol a sb r b sI a o sI D I amp I D sI D oe o PTb asamp bol as bu e b bol as b a sa o b I I o l oe es aa es n b ascob r b sI o u l y aC esn o PTb bol as b bosnn r b 13 b oe gI l n D a ca b I oo sl I nI Dr D C e I D oe sI o aI n o amp l n sI o P ol I n r ol amp o s oaI bno sl I l bol ah r oel a h r b a sI ol oe 13 gI l n D a ca b I oo sl I u

e I l n D sol a ol l sb r sno sI ol o PTb sI o a C I eI n b r b a bb ol oe gI l n D n amp I ob sI oe 13 a ca b I oo sl I l bol ahu o cal ys b oe r b a l I oal n l y a sI sI D oe I osos b sI bol ah b nn b oe a no sl I b amp l I D oe I osos bu e b l I a sI sDr a O a ab ol o b as sI oe ca ysl r b b osl I u

c t n pAsy ai onc S t ct ro i c t r n c s

sDr a ( snnr boao b oe ysbr n bhI oS r b h o PTb bol ah sol au e ob g sb a ca b I o b a oI Dn u b oe r b a amp l r b b l y a ob gC oe ol ah sol a sbcnh b oe 13 bc ss o sl I l oe ob gu e amp oel sb a ca b I o b b L r I l br ob gb oeo bc ss b oe l a asI D l oe br ob gb sI nr sI D so a osl I u ebr ob g sI or aI sb bc ss b a oI Dn soe sob l I amp oel b b oe I So nl a n y n l boa osl I u

i r My ai onc S t c ct ro ct r n c s

o o 13 h

sDr a B snnr boao b o PTb ol ah sol a sI o a u e amp s I c a o l oe sDr a l I os I b oe bol ah sI D so C sI oesb b oe l ol a b asco snnr boao sI sDr a Pu e c n oo l I oe ol c asDeo l sDr a B cal ys b oe s l I b l a sI D I ob g I amp oel n amp I ob ol oe bol ahu e sI l l I oe l ool amp n o c s ob oe I l n D sol au e sI l l I oe l ool amp asDeo sI s o b l oe a bol as b sI oe ol ah 13 s aa h I I n b oe r b a ol al b oe ns aa hu e Dac e sI oe l ool amp amps n I n b oe r b a ol I y sDo oe b asco l bol ahu

irkycs

uactroctrncr

TheStoryEditor inStab2gives theusermorecontrol thanwaspossible inStab In theoriginalStabsystemhierarchical scripts were hard‐coded in TMKL that itself is coded in the Lisp programming knowledge Thisresulted inseveraldifficulties regarding theusabilityof thesystemFirstly thestoredscripts inStabarenotavailableforreferenceorreviewexceptbyperusingtheactualLispcodeThislackoftransparencycanleadtoinaccurateandimpreciseunderstandingofthestoriesthataregeneratedbyStabasexplainingtheinputdataStab2rsquosStoryEditorprovidesvisualaccesstoboththelibraryofstoredscriptsaswellasthestoriesinstantiatedwhenStab2isrunforaninputdataset

SecondlysinceStabrsquosstoredscriptsarecodedinTMKLitisverydifficultforausertoauthoranewscriptorstoryevaluatethecorrectnessofastoredscriptandmakemodificationstoitStab2StoryEditorwithitsvisualinterface makes these tasks much easier though editing the knowledge elements in a story through theKnowledgeEditorstillisnon‐trivial

FinallytheinitialversionStabdidnotallowmodificationstothestructureofexistingscriptsatruntimeThisimpliesthat it isusefultorunStabjustonceonaninputdatasetHowever ingeneralahumananalystmaywanttorunStabseveraltimesndasheachwithdifferingvariationsofthescripts‐astheanalystrsquosinsightintothecriminal patterns evolves This is perhaps themost important side effect of the use of an interactive StoryEditor in Stab2 Stab2 enables an analyst to conduct ldquowhat ifrdquo simulations Using Stab2rsquos Story Editor andKnowledge Editor illustrated in Figure 7 an analystmay inspectmodify activate or deactivate the scriptsstored in Stab2rsquos Story Library the task and method elements in an individual script or the knowledgeelements likedomainconceptsand relationsTheanalystmay then runStab2 to simulate theeffectsof thepreciseconditionsshecreatedinStab2rsquosvirtualworld

Section34StoryAuthoringinStab2

Whenananalystwants tocreateanewstoryheneeds toopentheeditorandclickonaddstory iconThisopensawhiteareawithapaletteontherightWestartwithpickingtheldquoMainstoryldquoiconfromthepaletteanddropitintothewhiteboardThisrepresentsthenameofthestoryandisdepictedinFigure8(a)AfterthiswedragthebluerectanglethatreadsldquoIntermediaryGoalsrdquoandconnectthistothemainstoryusingaSub‐Goalconnector all ofwhich can be found in the palette The result is shown in Figure 8(b) Once themethod iscreatedwefillitinwithactionsperformedtoachievethatgoalThisisdoneusingtheldquoGoalsActionrdquoiconinthepaletteandthisisshowninFigure8(c)ThedraganddropcapabilitymakesitveryeasyforananalysttoconstructanewstoryplotThetreecangodownasmanylevelsdependingonthecomplexityofthestoryTherelationshipsbetweeneachofthegoalsarecapturedusingthelinksTwokindsoflinkagesarepossiblendashSub‐Goals that suggests that one goal is a child of another and Causal Connection that indicates relationshipsbetweendifferentactionsinsideeachintermediarygoal

Figure8(a)1

stSna

psho

tofStoryAutho

ringinStab2

ndashHerethehu

mananalystdragsth

eicon

mainstoryfrom

thepalette

andplacesitinth

eed

itorareaand

nam

esitasldquostoryplotrdquo

Figure8(b)A

2ndSna

psho

tofStoryAutho

ringin

Stab2

ndashHerethehu

mananalystdragsth

eInterm

ediaryGoalsicon

and

placesitbe

lowth

etopleveltaskAlsotheanalystdragsthesub‐goalslinkagefromthe

paletteand

con

nectsthetw

oblocksand

nam

esthe

bluerectangleas

ldquoMetho

d1rdquo

Figure8(c)ASna

psho

tofStoryAutho

ringinStab2

‐Heretheanalystd

ragsth

eGoalsA

ctionsicon

inside

thealreadycreatedmetho

dHe

then

con

nectstheseactio

nswith

thecausalcon

nectionslinkageinth

epaletteandnameseachactio

n

BehindtheScenes

Figure9(a)A

Sna

psho

tofaXMLfileforeachStoryPlot

Figure9(b)A

Sna

psho

tofaXMLfileforeachStoryPlotndash

Thisim

agehighlightsindividu

alta

gsThe

imageon

the

righth

ighlightsthechildTaskta

gandtheon

ethelefthighlightstheallM

etho

dsta

g

EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)

RelatedWork

ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors

StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)

Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo

Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)

CurrentWork

ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)

Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities

Acknowledgements

Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories

References

AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007

AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8

Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008

BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005

BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005

Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991

ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17

CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985

Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34

CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum

Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)

FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991

Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995

Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009

GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia

Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13

HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)

HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07

JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863

Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009

JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994

Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92

KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999

Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity

LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007

Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180

Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey

MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey

MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691

Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340

Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003

Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81

Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36

PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004

PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005

Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21

RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007

RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318

RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262

RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006

SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924

SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress

SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum

StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132

TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)

ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety

Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623

Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA

WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370

Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics

hypothesis inagraphicalrepresentation Italsoenablesselectionofportionsoftheevidence insupportofageneratedhypothesis for further analysisof theexplanatory relationshipbetween thegeneratedhypothesisandthesupportingevidenceAhumananalystmayuseStabasanexternalmemoryofdataandexplanatoryhypothesesInadditiontheanalystmayuseStabrsquosresultstoinformadditionalsearchforfurtherevidencefororagainstageneratedhypothesis

OfcourseStabalsohasseverallimitationsOneofthemajorlimitationsofStabisthatcrimestoriesinitsstorylibraryhavetobehandcoded inTMKLThismakes itdifficult forausertoenternewstoriesoreditexistingstoriesinStabrsquoslibraryToenableausertomoreflexiblyinteractwithStabwehavedevelopedStab2anewinteractiveversionofStabStab2containsastoryeditorthatenablesuserstoenterandeditcrimestoriesinagraphical notation Stab2 automatically converts a new (or modified) story into its internal knowledgerepresentation language(TMKL)Thisenablesusers to interactwithStabrsquosstoriesldquoonlinerdquo InthispaperwefirstbrieflydescribethebasicsofStabandthenpresentStab2rsquosstoryeditor

Section2Knowledge‐BasedSupportforSensemaking

InthissectionwebrieflydescribetheoriginalStabsystemforsensemakingoftheIEEEVASTcontestdatasetsAlthoughStabrsquosreasoningknowledgeandrepresentationsareintendedtobegeneralStabrsquoslibraryatpresentcontainsonlycrimestoriesrelevanttotheVAST‐2006andVAST‐2007datasets

VAST Datasets The VAST datasets are synthetic datasets generated by the Pacific Northwest NationalLaboratories forsupportingresearchanddevelopment intheemergingfieldofvisualanalytics (ThompsonampCook 2005) The datasets pertain to fictitious illegal and unethical activities as well as normal and typicalactivities in a fictitious town in the United States The VAST‐2006 dataset contains over a thousand newsstorieswritteninEnglishandascoreoftablesmapsandphotographsFigure1 illustratesanexamplenewsstoryfromtheVAST2006datasetTheVAST‐2007dataset isa little largerandslightlymorecomplicatedbutsimilarinnature

Inputs to StabWemanually screened the dataset for stories that indicated an illegal or unethical activitywhich leftaboutahundrednewsstoriesoutofthemorethanathousandoriginally inthedatasetWethenmanuallyextractedeventsandentitiespertainingtoillegalunethicalactivitiesTheseeventsentitiesformtheinput to STAB We also hand crafted representations for each event in terms of the knowledge states itproducesInadditionweexaminedthemapsphotosandtablesthatarepartoftheVASTdatasetandsimilarlyextracted and represented the relevant information about various entities Table 1 illustrates a sample ofinputstoSTABalongwiththeresultingknowledgestatecreatedbyaninputevent

Figure1ExamplenewsstoryfromtheVAST‐2006dataset

(Adaptedfrom(httpwwwcsumdeduhcilVASTcontest06)Table1AsampleofinputstoStab

SampleSTABInputs ResultingState

stolen(money$40Highway‐Tire‐Store) Has‐object

cured‐disease(Boynton‐LabsPhilip‐Boyntonprion‐disease) Is‐rich‐and‐famous

named‐after(labPhilip‐BoyntonDean‐USC) Expert‐involved

was‐founded(Boynton‐Labs) Is‐open

have‐developed(Boynton‐Labsprion‐disease) Exists‐new‐disease

announced‐investigation(USFDABoynton‐Labs) Is‐investigating

Injected‐cow(Boynton‐Labsprion‐disease) Cow‐is‐infected

treatment‐cow(Boynton‐Labsprion‐disease) Cow‐is‐cured

Stabrsquos Library of Crime Stories STAB contains a small library of hierarchically organized abstract stories orhierarchicalscriptsthatcapturethepatternsofcrimesthatoccurintheVAST‐2006andVAST‐2007datasetsFigure2illustratesasimplescriptinSTABrsquoslibraryRobaStoreThisscripthasthegoalofachievingthestateofHaveMoneygiventheinitialstateofNotHaveMoney(topoffigure)Thisgoal(ortask)accordingtotheRobaStorescriptisachievedbyaplan(ormethod)thathasseveralactions(orevents)initGotoStoreBreakintoStore TakeMoney (middle of the figure) Each of these events becomes a task at the next lower level ofabstraction in the hierarchical script and each task can be (potentially) achieved bymultiplemethods ForexampleaccordingtotheRobaStorescriptthetaskofBreakintoStorecanbeachievedbyEnteringthroughaWindow or Entering through a Door (bottom of figure) Each of thesemethods in turn specifies a processconsistingofmultipleeventsandsoon

STABrsquos hierarchical scripts explicitly represent both goal and state at multiple levels of abstraction Whilerepresentation of the state caused by an event is useful for inferring causality representation of goals ofsequences of events is useful for inferring intention Figure 3 illustrates a more complex script of politicalconspiracyinwhichapoliticalfiguremaygetanopponentoutofanelectoralraceeitherbyexposingdirtonhim(politicalblackmail)orhavinghimassassinatedNotethisscriptiscomposedofseveralsmallerscripts

We found that sevenhierarchical scripts appear to cover all the illegalunethical activities in theVAST‐2006datasetandthatanotherfourhierarchicalscriptsapparentlyaresufficienttocovertheVAST‐2007aswellWehandcraftedthislibraryofscriptsintoSTAB

Figure2ThecontentandstructureofastoryinSTAB1(AdaptedfromGoeletal2009)

(

Stabrsquos Knowledge Representation STABrsquos scripts are represented in the TMKL language A task in TMKL isdefinedbyinputknowledgeelementsoutputknowledgeelementsrequiredinputconditions(pre‐conditions)desiredoutputconditions (post‐conditions)andmethods for implementing the taskAmethod isdefined intermsofsubtasksofthetaskandorderingofthesubtasksandisrepresentedbyafinitestatemachineFinallyknowledge in TMKL is defined in terms of domain concepts and relationships among them the input andoutputknowledgeelements ina taskspecificationrefer tothesedomainconceptsandrelationsMurdockampGoel2008providesdetailsofTMKL

Figure3ThestoryforapoliticalconspiracyintendedtoremoveanopponentfromanelectoralraceActivatednodesaredenotedbyathickoutlinearoundyellowboxes(AdaptedfromGoelet

al2009)

Figure4High‐LevelArchitectureofSTAB(AdaptedfromGoeletal2009)

t ba SV Ts t t SA D t tsD u sDr a H snnr boao b o Tb l ampcr oo sl I n a eso or a u saboC oe ys I l nn ol a l nn ob oe sI cr o y I ob sI I ys I sn sI eal I l nl Ds n l a au SoC oe ol ah o e a og b l I sI c r o y I o o osamp I r b b sob a br nosI D gI l n D boo l oe y I o soe oe ob g I l b sI oe 13 a ca b I oo sl I b l oe b ascob bol a sI oe ol ah 13 s aa hu e ol ah o e a oD b oe amp o esI D ob gb I c b b boe amp o esI D b ascob ol l agsI D amp l ahu esb o DDsI D l amp o esI D ob gb sI o sI boI oso b oe ob gu e ol ah o e a oe I sI bc ob oe I So sI cr o y I o sI oe ys I sn I a c o b oe l y cal bbu oe I sI cr o y I o a br nob sI oe a oas yn l I b ascoC oe I oe b asco sb bsampsna nh bol a sI oe l agsI D amp l ahu oe I nh a oas y b asco sb n a h sI oe l agsI D amp l ahC oe I sosl I n ob g I l b oeo amp o e oe I sI cr o a n bl oD D r o l I nh l I b asco sI boI sb g cou

t ba s tTs tau o l r ocr ob l r a gsI b l sI l aamp o sl I p n n sI boI oso b ascob sI oe l agsI D amp amp l ahC n n I l bsI e br e b asco oeo a amp o e soe I sI cr o y I oC n n ys I l a e br e b ascoC I l I s I yn r l a e br e b ascou e h nnl l S b sI sDr a H snnr boao oe ob gb sI oe c l nsos n l I bcsa h b asco oeo a amp o e soe I sI cr o y I o sI l I ar I I sI D l o u e l I s I yn r l a b asco sb b l I oe I r amp a l ob gb sI b asco oeo a amp o e soe sI cr o y I obu l n o n u dPVV cal ys os nb l o Tba b l I sI DC gI l n D C a ca b I oo sl I b I l r ocr obu

c t n py nc r c v ct ro c n n c s

e sI o a osy o P bhbo amp sb I r Damp I oo sl I l oe l asDsI n o bhbo amp soe casamp a h l r b l I sampcal ysI Dr b a sI o a osl I u e bc ss Dl n b l o P I eI amp I ob sI nr p ds nnl l I nsI r oel asI D I sosI D l o Tb asamp bol as b sI ysbr n l aampC I dss I n r b a ol l I oal n n n b c ob l o Tb a b l I sI D oeal r De Dac es n sI o a u

i r Ty t I ei c c t n r c ci r t c c sua ct ro i c t r n t t

o s h

sDr a O snnr boao b oe l ampcr oo sl I n a eso or a l o PTb bol ah r oel asI D ol l nu e ol ah sol a sb r b sI a o sI D I amp I D sI D oe o PTb asamp bol as bu e b bol as b a sa o b I I o l oe es aa es n b ascob r b sI o u l y aC esn o PTb bol as b bosnn r b 13 b oe gI l n D a ca b I oo sl I nI Dr D C e I D oe sI o aI n o amp l n sI o P ol I n r ol amp o s oaI bno sl I l bol ah r oel a h r b a sI ol oe 13 gI l n D a ca b I oo sl I u

e I l n D sol a ol l sb r sno sI ol o PTb sI o a C I eI n b r b a bb ol oe gI l n D n amp I ob sI oe 13 a ca b I oo sl I l bol ahu o cal ys b oe r b a l I oal n l y a sI sI D oe I osos b sI bol ah b nn b oe a no sl I b amp l I D oe I osos bu e b l I a sI sDr a O a ab ol o b as sI oe ca ysl r b b osl I u

c t n pAsy ai onc S t ct ro i c t r n c s

sDr a ( snnr boao b oe ysbr n bhI oS r b h o PTb bol ah sol au e ob g sb a ca b I o b a oI Dn u b oe r b a amp l r b b l y a ob gC oe ol ah sol a sbcnh b oe 13 bc ss o sl I l oe ob gu e amp oel sb a ca b I o b b L r I l br ob gb oeo bc ss b oe l a asI D l oe br ob gb sI nr sI D so a osl I u ebr ob g sI or aI sb bc ss b a oI Dn soe sob l I amp oel b b oe I So nl a n y n l boa osl I u

i r My ai onc S t c ct ro ct r n c s

o o 13 h

sDr a B snnr boao b o PTb ol ah sol a sI o a u e amp s I c a o l oe sDr a l I os I b oe bol ah sI D so C sI oesb b oe l ol a b asco snnr boao sI sDr a Pu e c n oo l I oe ol c asDeo l sDr a B cal ys b oe s l I b l a sI D I ob g I amp oel n amp I ob ol oe bol ahu e sI l l I oe l ool amp n o c s ob oe I l n D sol au e sI l l I oe l ool amp asDeo sI s o b l oe a bol as b sI oe ol ah 13 s aa h I I n b oe r b a ol al b oe ns aa hu e Dac e sI oe l ool amp amps n I n b oe r b a ol I y sDo oe b asco l bol ahu

irkycs

uactroctrncr

TheStoryEditor inStab2gives theusermorecontrol thanwaspossible inStab In theoriginalStabsystemhierarchical scripts were hard‐coded in TMKL that itself is coded in the Lisp programming knowledge Thisresulted inseveraldifficulties regarding theusabilityof thesystemFirstly thestoredscripts inStabarenotavailableforreferenceorreviewexceptbyperusingtheactualLispcodeThislackoftransparencycanleadtoinaccurateandimpreciseunderstandingofthestoriesthataregeneratedbyStabasexplainingtheinputdataStab2rsquosStoryEditorprovidesvisualaccesstoboththelibraryofstoredscriptsaswellasthestoriesinstantiatedwhenStab2isrunforaninputdataset

SecondlysinceStabrsquosstoredscriptsarecodedinTMKLitisverydifficultforausertoauthoranewscriptorstoryevaluatethecorrectnessofastoredscriptandmakemodificationstoitStab2StoryEditorwithitsvisualinterface makes these tasks much easier though editing the knowledge elements in a story through theKnowledgeEditorstillisnon‐trivial

FinallytheinitialversionStabdidnotallowmodificationstothestructureofexistingscriptsatruntimeThisimpliesthat it isusefultorunStabjustonceonaninputdatasetHowever ingeneralahumananalystmaywanttorunStabseveraltimesndasheachwithdifferingvariationsofthescripts‐astheanalystrsquosinsightintothecriminal patterns evolves This is perhaps themost important side effect of the use of an interactive StoryEditor in Stab2 Stab2 enables an analyst to conduct ldquowhat ifrdquo simulations Using Stab2rsquos Story Editor andKnowledge Editor illustrated in Figure 7 an analystmay inspectmodify activate or deactivate the scriptsstored in Stab2rsquos Story Library the task and method elements in an individual script or the knowledgeelements likedomainconceptsand relationsTheanalystmay then runStab2 to simulate theeffectsof thepreciseconditionsshecreatedinStab2rsquosvirtualworld

Section34StoryAuthoringinStab2

Whenananalystwants tocreateanewstoryheneeds toopentheeditorandclickonaddstory iconThisopensawhiteareawithapaletteontherightWestartwithpickingtheldquoMainstoryldquoiconfromthepaletteanddropitintothewhiteboardThisrepresentsthenameofthestoryandisdepictedinFigure8(a)AfterthiswedragthebluerectanglethatreadsldquoIntermediaryGoalsrdquoandconnectthistothemainstoryusingaSub‐Goalconnector all ofwhich can be found in the palette The result is shown in Figure 8(b) Once themethod iscreatedwefillitinwithactionsperformedtoachievethatgoalThisisdoneusingtheldquoGoalsActionrdquoiconinthepaletteandthisisshowninFigure8(c)ThedraganddropcapabilitymakesitveryeasyforananalysttoconstructanewstoryplotThetreecangodownasmanylevelsdependingonthecomplexityofthestoryTherelationshipsbetweeneachofthegoalsarecapturedusingthelinksTwokindsoflinkagesarepossiblendashSub‐Goals that suggests that one goal is a child of another and Causal Connection that indicates relationshipsbetweendifferentactionsinsideeachintermediarygoal

Figure8(a)1

stSna

psho

tofStoryAutho

ringinStab2

ndashHerethehu

mananalystdragsth

eicon

mainstoryfrom

thepalette

andplacesitinth

eed

itorareaand

nam

esitasldquostoryplotrdquo

Figure8(b)A

2ndSna

psho

tofStoryAutho

ringin

Stab2

ndashHerethehu

mananalystdragsth

eInterm

ediaryGoalsicon

and

placesitbe

lowth

etopleveltaskAlsotheanalystdragsthesub‐goalslinkagefromthe

paletteand

con

nectsthetw

oblocksand

nam

esthe

bluerectangleas

ldquoMetho

d1rdquo

Figure8(c)ASna

psho

tofStoryAutho

ringinStab2

‐Heretheanalystd

ragsth

eGoalsA

ctionsicon

inside

thealreadycreatedmetho

dHe

then

con

nectstheseactio

nswith

thecausalcon

nectionslinkageinth

epaletteandnameseachactio

n

BehindtheScenes

Figure9(a)A

Sna

psho

tofaXMLfileforeachStoryPlot

Figure9(b)A

Sna

psho

tofaXMLfileforeachStoryPlotndash

Thisim

agehighlightsindividu

alta

gsThe

imageon

the

righth

ighlightsthechildTaskta

gandtheon

ethelefthighlightstheallM

etho

dsta

g

EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)

RelatedWork

ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors

StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)

Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo

Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)

CurrentWork

ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)

Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities

Acknowledgements

Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories

References

AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007

AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8

Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008

BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005

BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005

Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991

ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17

CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985

Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34

CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum

Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)

FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991

Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995

Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009

GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia

Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13

HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)

HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07

JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863

Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009

JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994

Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92

KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999

Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity

LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007

Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180

Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey

MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey

MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691

Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340

Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003

Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81

Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36

PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004

PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005

Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21

RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007

RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318

RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262

RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006

SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924

SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress

SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum

StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132

TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)

ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety

Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623

Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA

WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370

Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics

Figure1ExamplenewsstoryfromtheVAST‐2006dataset

(Adaptedfrom(httpwwwcsumdeduhcilVASTcontest06)Table1AsampleofinputstoStab

SampleSTABInputs ResultingState

stolen(money$40Highway‐Tire‐Store) Has‐object

cured‐disease(Boynton‐LabsPhilip‐Boyntonprion‐disease) Is‐rich‐and‐famous

named‐after(labPhilip‐BoyntonDean‐USC) Expert‐involved

was‐founded(Boynton‐Labs) Is‐open

have‐developed(Boynton‐Labsprion‐disease) Exists‐new‐disease

announced‐investigation(USFDABoynton‐Labs) Is‐investigating

Injected‐cow(Boynton‐Labsprion‐disease) Cow‐is‐infected

treatment‐cow(Boynton‐Labsprion‐disease) Cow‐is‐cured

Stabrsquos Library of Crime Stories STAB contains a small library of hierarchically organized abstract stories orhierarchicalscriptsthatcapturethepatternsofcrimesthatoccurintheVAST‐2006andVAST‐2007datasetsFigure2illustratesasimplescriptinSTABrsquoslibraryRobaStoreThisscripthasthegoalofachievingthestateofHaveMoneygiventheinitialstateofNotHaveMoney(topoffigure)Thisgoal(ortask)accordingtotheRobaStorescriptisachievedbyaplan(ormethod)thathasseveralactions(orevents)initGotoStoreBreakintoStore TakeMoney (middle of the figure) Each of these events becomes a task at the next lower level ofabstraction in the hierarchical script and each task can be (potentially) achieved bymultiplemethods ForexampleaccordingtotheRobaStorescriptthetaskofBreakintoStorecanbeachievedbyEnteringthroughaWindow or Entering through a Door (bottom of figure) Each of thesemethods in turn specifies a processconsistingofmultipleeventsandsoon

STABrsquos hierarchical scripts explicitly represent both goal and state at multiple levels of abstraction Whilerepresentation of the state caused by an event is useful for inferring causality representation of goals ofsequences of events is useful for inferring intention Figure 3 illustrates a more complex script of politicalconspiracyinwhichapoliticalfiguremaygetanopponentoutofanelectoralraceeitherbyexposingdirtonhim(politicalblackmail)orhavinghimassassinatedNotethisscriptiscomposedofseveralsmallerscripts

We found that sevenhierarchical scripts appear to cover all the illegalunethical activities in theVAST‐2006datasetandthatanotherfourhierarchicalscriptsapparentlyaresufficienttocovertheVAST‐2007aswellWehandcraftedthislibraryofscriptsintoSTAB

Figure2ThecontentandstructureofastoryinSTAB1(AdaptedfromGoeletal2009)

(

Stabrsquos Knowledge Representation STABrsquos scripts are represented in the TMKL language A task in TMKL isdefinedbyinputknowledgeelementsoutputknowledgeelementsrequiredinputconditions(pre‐conditions)desiredoutputconditions (post‐conditions)andmethods for implementing the taskAmethod isdefined intermsofsubtasksofthetaskandorderingofthesubtasksandisrepresentedbyafinitestatemachineFinallyknowledge in TMKL is defined in terms of domain concepts and relationships among them the input andoutputknowledgeelements ina taskspecificationrefer tothesedomainconceptsandrelationsMurdockampGoel2008providesdetailsofTMKL

Figure3ThestoryforapoliticalconspiracyintendedtoremoveanopponentfromanelectoralraceActivatednodesaredenotedbyathickoutlinearoundyellowboxes(AdaptedfromGoelet

al2009)

Figure4High‐LevelArchitectureofSTAB(AdaptedfromGoeletal2009)

t ba SV Ts t t SA D t tsD u sDr a H snnr boao b o Tb l ampcr oo sl I n a eso or a u saboC oe ys I l nn ol a l nn ob oe sI cr o y I ob sI I ys I sn sI eal I l nl Ds n l a au SoC oe ol ah o e a og b l I sI c r o y I o o osamp I r b b sob a br nosI D gI l n D boo l oe y I o soe oe ob g I l b sI oe 13 a ca b I oo sl I b l oe b ascob bol a sI oe ol ah 13 s aa hu e ol ah o e a oD b oe amp o esI D ob gb I c b b boe amp o esI D b ascob ol l agsI D amp l ahu esb o DDsI D l amp o esI D ob gb sI o sI boI oso b oe ob gu e ol ah o e a oe I sI bc ob oe I So sI cr o y I o sI oe ys I sn I a c o b oe l y cal bbu oe I sI cr o y I o a br nob sI oe a oas yn l I b ascoC oe I oe b asco sb bsampsna nh bol a sI oe l agsI D amp l ahu oe I nh a oas y b asco sb n a h sI oe l agsI D amp l ahC oe I sosl I n ob g I l b oeo amp o e oe I sI cr o a n bl oD D r o l I nh l I b asco sI boI sb g cou

t ba s tTs tau o l r ocr ob l r a gsI b l sI l aamp o sl I p n n sI boI oso b ascob sI oe l agsI D amp amp l ahC n n I l bsI e br e b asco oeo a amp o e soe I sI cr o y I oC n n ys I l a e br e b ascoC I l I s I yn r l a e br e b ascou e h nnl l S b sI sDr a H snnr boao oe ob gb sI oe c l nsos n l I bcsa h b asco oeo a amp o e soe I sI cr o y I o sI l I ar I I sI D l o u e l I s I yn r l a b asco sb b l I oe I r amp a l ob gb sI b asco oeo a amp o e soe sI cr o y I obu l n o n u dPVV cal ys os nb l o Tba b l I sI DC gI l n D C a ca b I oo sl I b I l r ocr obu

c t n py nc r c v ct ro c n n c s

e sI o a osy o P bhbo amp sb I r Damp I oo sl I l oe l asDsI n o bhbo amp soe casamp a h l r b l I sampcal ysI Dr b a sI o a osl I u e bc ss Dl n b l o P I eI amp I ob sI nr p ds nnl l I nsI r oel asI D I sosI D l o Tb asamp bol as b sI ysbr n l aampC I dss I n r b a ol l I oal n n n b c ob l o Tb a b l I sI D oeal r De Dac es n sI o a u

i r Ty t I ei c c t n r c ci r t c c sua ct ro i c t r n t t

o s h

sDr a O snnr boao b oe l ampcr oo sl I n a eso or a l o PTb bol ah r oel asI D ol l nu e ol ah sol a sb r b sI a o sI D I amp I D sI D oe o PTb asamp bol as bu e b bol as b a sa o b I I o l oe es aa es n b ascob r b sI o u l y aC esn o PTb bol as b bosnn r b 13 b oe gI l n D a ca b I oo sl I nI Dr D C e I D oe sI o aI n o amp l n sI o P ol I n r ol amp o s oaI bno sl I l bol ah r oel a h r b a sI ol oe 13 gI l n D a ca b I oo sl I u

e I l n D sol a ol l sb r sno sI ol o PTb sI o a C I eI n b r b a bb ol oe gI l n D n amp I ob sI oe 13 a ca b I oo sl I l bol ahu o cal ys b oe r b a l I oal n l y a sI sI D oe I osos b sI bol ah b nn b oe a no sl I b amp l I D oe I osos bu e b l I a sI sDr a O a ab ol o b as sI oe ca ysl r b b osl I u

c t n pAsy ai onc S t ct ro i c t r n c s

sDr a ( snnr boao b oe ysbr n bhI oS r b h o PTb bol ah sol au e ob g sb a ca b I o b a oI Dn u b oe r b a amp l r b b l y a ob gC oe ol ah sol a sbcnh b oe 13 bc ss o sl I l oe ob gu e amp oel sb a ca b I o b b L r I l br ob gb oeo bc ss b oe l a asI D l oe br ob gb sI nr sI D so a osl I u ebr ob g sI or aI sb bc ss b a oI Dn soe sob l I amp oel b b oe I So nl a n y n l boa osl I u

i r My ai onc S t c ct ro ct r n c s

o o 13 h

sDr a B snnr boao b o PTb ol ah sol a sI o a u e amp s I c a o l oe sDr a l I os I b oe bol ah sI D so C sI oesb b oe l ol a b asco snnr boao sI sDr a Pu e c n oo l I oe ol c asDeo l sDr a B cal ys b oe s l I b l a sI D I ob g I amp oel n amp I ob ol oe bol ahu e sI l l I oe l ool amp n o c s ob oe I l n D sol au e sI l l I oe l ool amp asDeo sI s o b l oe a bol as b sI oe ol ah 13 s aa h I I n b oe r b a ol al b oe ns aa hu e Dac e sI oe l ool amp amps n I n b oe r b a ol I y sDo oe b asco l bol ahu

irkycs

uactroctrncr

TheStoryEditor inStab2gives theusermorecontrol thanwaspossible inStab In theoriginalStabsystemhierarchical scripts were hard‐coded in TMKL that itself is coded in the Lisp programming knowledge Thisresulted inseveraldifficulties regarding theusabilityof thesystemFirstly thestoredscripts inStabarenotavailableforreferenceorreviewexceptbyperusingtheactualLispcodeThislackoftransparencycanleadtoinaccurateandimpreciseunderstandingofthestoriesthataregeneratedbyStabasexplainingtheinputdataStab2rsquosStoryEditorprovidesvisualaccesstoboththelibraryofstoredscriptsaswellasthestoriesinstantiatedwhenStab2isrunforaninputdataset

SecondlysinceStabrsquosstoredscriptsarecodedinTMKLitisverydifficultforausertoauthoranewscriptorstoryevaluatethecorrectnessofastoredscriptandmakemodificationstoitStab2StoryEditorwithitsvisualinterface makes these tasks much easier though editing the knowledge elements in a story through theKnowledgeEditorstillisnon‐trivial

FinallytheinitialversionStabdidnotallowmodificationstothestructureofexistingscriptsatruntimeThisimpliesthat it isusefultorunStabjustonceonaninputdatasetHowever ingeneralahumananalystmaywanttorunStabseveraltimesndasheachwithdifferingvariationsofthescripts‐astheanalystrsquosinsightintothecriminal patterns evolves This is perhaps themost important side effect of the use of an interactive StoryEditor in Stab2 Stab2 enables an analyst to conduct ldquowhat ifrdquo simulations Using Stab2rsquos Story Editor andKnowledge Editor illustrated in Figure 7 an analystmay inspectmodify activate or deactivate the scriptsstored in Stab2rsquos Story Library the task and method elements in an individual script or the knowledgeelements likedomainconceptsand relationsTheanalystmay then runStab2 to simulate theeffectsof thepreciseconditionsshecreatedinStab2rsquosvirtualworld

Section34StoryAuthoringinStab2

Whenananalystwants tocreateanewstoryheneeds toopentheeditorandclickonaddstory iconThisopensawhiteareawithapaletteontherightWestartwithpickingtheldquoMainstoryldquoiconfromthepaletteanddropitintothewhiteboardThisrepresentsthenameofthestoryandisdepictedinFigure8(a)AfterthiswedragthebluerectanglethatreadsldquoIntermediaryGoalsrdquoandconnectthistothemainstoryusingaSub‐Goalconnector all ofwhich can be found in the palette The result is shown in Figure 8(b) Once themethod iscreatedwefillitinwithactionsperformedtoachievethatgoalThisisdoneusingtheldquoGoalsActionrdquoiconinthepaletteandthisisshowninFigure8(c)ThedraganddropcapabilitymakesitveryeasyforananalysttoconstructanewstoryplotThetreecangodownasmanylevelsdependingonthecomplexityofthestoryTherelationshipsbetweeneachofthegoalsarecapturedusingthelinksTwokindsoflinkagesarepossiblendashSub‐Goals that suggests that one goal is a child of another and Causal Connection that indicates relationshipsbetweendifferentactionsinsideeachintermediarygoal

Figure8(a)1

stSna

psho

tofStoryAutho

ringinStab2

ndashHerethehu

mananalystdragsth

eicon

mainstoryfrom

thepalette

andplacesitinth

eed

itorareaand

nam

esitasldquostoryplotrdquo

Figure8(b)A

2ndSna

psho

tofStoryAutho

ringin

Stab2

ndashHerethehu

mananalystdragsth

eInterm

ediaryGoalsicon

and

placesitbe

lowth

etopleveltaskAlsotheanalystdragsthesub‐goalslinkagefromthe

paletteand

con

nectsthetw

oblocksand

nam

esthe

bluerectangleas

ldquoMetho

d1rdquo

Figure8(c)ASna

psho

tofStoryAutho

ringinStab2

‐Heretheanalystd

ragsth

eGoalsA

ctionsicon

inside

thealreadycreatedmetho

dHe

then

con

nectstheseactio

nswith

thecausalcon

nectionslinkageinth

epaletteandnameseachactio

n

BehindtheScenes

Figure9(a)A

Sna

psho

tofaXMLfileforeachStoryPlot

Figure9(b)A

Sna

psho

tofaXMLfileforeachStoryPlotndash

Thisim

agehighlightsindividu

alta

gsThe

imageon

the

righth

ighlightsthechildTaskta

gandtheon

ethelefthighlightstheallM

etho

dsta

g

EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)

RelatedWork

ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors

StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)

Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo

Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)

CurrentWork

ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)

Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities

Acknowledgements

Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories

References

AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007

AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8

Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008

BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005

BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005

Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991

ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17

CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985

Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34

CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum

Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)

FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991

Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995

Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009

GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia

Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13

HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)

HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07

JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863

Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009

JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994

Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92

KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999

Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity

LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007

Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180

Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey

MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey

MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691

Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340

Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003

Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81

Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36

PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004

PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005

Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21

RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007

RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318

RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262

RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006

SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924

SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress

SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum

StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132

TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)

ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety

Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623

Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA

WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370

Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics

Stabrsquos Library of Crime Stories STAB contains a small library of hierarchically organized abstract stories orhierarchicalscriptsthatcapturethepatternsofcrimesthatoccurintheVAST‐2006andVAST‐2007datasetsFigure2illustratesasimplescriptinSTABrsquoslibraryRobaStoreThisscripthasthegoalofachievingthestateofHaveMoneygiventheinitialstateofNotHaveMoney(topoffigure)Thisgoal(ortask)accordingtotheRobaStorescriptisachievedbyaplan(ormethod)thathasseveralactions(orevents)initGotoStoreBreakintoStore TakeMoney (middle of the figure) Each of these events becomes a task at the next lower level ofabstraction in the hierarchical script and each task can be (potentially) achieved bymultiplemethods ForexampleaccordingtotheRobaStorescriptthetaskofBreakintoStorecanbeachievedbyEnteringthroughaWindow or Entering through a Door (bottom of figure) Each of thesemethods in turn specifies a processconsistingofmultipleeventsandsoon

STABrsquos hierarchical scripts explicitly represent both goal and state at multiple levels of abstraction Whilerepresentation of the state caused by an event is useful for inferring causality representation of goals ofsequences of events is useful for inferring intention Figure 3 illustrates a more complex script of politicalconspiracyinwhichapoliticalfiguremaygetanopponentoutofanelectoralraceeitherbyexposingdirtonhim(politicalblackmail)orhavinghimassassinatedNotethisscriptiscomposedofseveralsmallerscripts

We found that sevenhierarchical scripts appear to cover all the illegalunethical activities in theVAST‐2006datasetandthatanotherfourhierarchicalscriptsapparentlyaresufficienttocovertheVAST‐2007aswellWehandcraftedthislibraryofscriptsintoSTAB

Figure2ThecontentandstructureofastoryinSTAB1(AdaptedfromGoeletal2009)

(

Stabrsquos Knowledge Representation STABrsquos scripts are represented in the TMKL language A task in TMKL isdefinedbyinputknowledgeelementsoutputknowledgeelementsrequiredinputconditions(pre‐conditions)desiredoutputconditions (post‐conditions)andmethods for implementing the taskAmethod isdefined intermsofsubtasksofthetaskandorderingofthesubtasksandisrepresentedbyafinitestatemachineFinallyknowledge in TMKL is defined in terms of domain concepts and relationships among them the input andoutputknowledgeelements ina taskspecificationrefer tothesedomainconceptsandrelationsMurdockampGoel2008providesdetailsofTMKL

Figure3ThestoryforapoliticalconspiracyintendedtoremoveanopponentfromanelectoralraceActivatednodesaredenotedbyathickoutlinearoundyellowboxes(AdaptedfromGoelet

al2009)

Figure4High‐LevelArchitectureofSTAB(AdaptedfromGoeletal2009)

t ba SV Ts t t SA D t tsD u sDr a H snnr boao b o Tb l ampcr oo sl I n a eso or a u saboC oe ys I l nn ol a l nn ob oe sI cr o y I ob sI I ys I sn sI eal I l nl Ds n l a au SoC oe ol ah o e a og b l I sI c r o y I o o osamp I r b b sob a br nosI D gI l n D boo l oe y I o soe oe ob g I l b sI oe 13 a ca b I oo sl I b l oe b ascob bol a sI oe ol ah 13 s aa hu e ol ah o e a oD b oe amp o esI D ob gb I c b b boe amp o esI D b ascob ol l agsI D amp l ahu esb o DDsI D l amp o esI D ob gb sI o sI boI oso b oe ob gu e ol ah o e a oe I sI bc ob oe I So sI cr o y I o sI oe ys I sn I a c o b oe l y cal bbu oe I sI cr o y I o a br nob sI oe a oas yn l I b ascoC oe I oe b asco sb bsampsna nh bol a sI oe l agsI D amp l ahu oe I nh a oas y b asco sb n a h sI oe l agsI D amp l ahC oe I sosl I n ob g I l b oeo amp o e oe I sI cr o a n bl oD D r o l I nh l I b asco sI boI sb g cou

t ba s tTs tau o l r ocr ob l r a gsI b l sI l aamp o sl I p n n sI boI oso b ascob sI oe l agsI D amp amp l ahC n n I l bsI e br e b asco oeo a amp o e soe I sI cr o y I oC n n ys I l a e br e b ascoC I l I s I yn r l a e br e b ascou e h nnl l S b sI sDr a H snnr boao oe ob gb sI oe c l nsos n l I bcsa h b asco oeo a amp o e soe I sI cr o y I o sI l I ar I I sI D l o u e l I s I yn r l a b asco sb b l I oe I r amp a l ob gb sI b asco oeo a amp o e soe sI cr o y I obu l n o n u dPVV cal ys os nb l o Tba b l I sI DC gI l n D C a ca b I oo sl I b I l r ocr obu

c t n py nc r c v ct ro c n n c s

e sI o a osy o P bhbo amp sb I r Damp I oo sl I l oe l asDsI n o bhbo amp soe casamp a h l r b l I sampcal ysI Dr b a sI o a osl I u e bc ss Dl n b l o P I eI amp I ob sI nr p ds nnl l I nsI r oel asI D I sosI D l o Tb asamp bol as b sI ysbr n l aampC I dss I n r b a ol l I oal n n n b c ob l o Tb a b l I sI D oeal r De Dac es n sI o a u

i r Ty t I ei c c t n r c ci r t c c sua ct ro i c t r n t t

o s h

sDr a O snnr boao b oe l ampcr oo sl I n a eso or a l o PTb bol ah r oel asI D ol l nu e ol ah sol a sb r b sI a o sI D I amp I D sI D oe o PTb asamp bol as bu e b bol as b a sa o b I I o l oe es aa es n b ascob r b sI o u l y aC esn o PTb bol as b bosnn r b 13 b oe gI l n D a ca b I oo sl I nI Dr D C e I D oe sI o aI n o amp l n sI o P ol I n r ol amp o s oaI bno sl I l bol ah r oel a h r b a sI ol oe 13 gI l n D a ca b I oo sl I u

e I l n D sol a ol l sb r sno sI ol o PTb sI o a C I eI n b r b a bb ol oe gI l n D n amp I ob sI oe 13 a ca b I oo sl I l bol ahu o cal ys b oe r b a l I oal n l y a sI sI D oe I osos b sI bol ah b nn b oe a no sl I b amp l I D oe I osos bu e b l I a sI sDr a O a ab ol o b as sI oe ca ysl r b b osl I u

c t n pAsy ai onc S t ct ro i c t r n c s

sDr a ( snnr boao b oe ysbr n bhI oS r b h o PTb bol ah sol au e ob g sb a ca b I o b a oI Dn u b oe r b a amp l r b b l y a ob gC oe ol ah sol a sbcnh b oe 13 bc ss o sl I l oe ob gu e amp oel sb a ca b I o b b L r I l br ob gb oeo bc ss b oe l a asI D l oe br ob gb sI nr sI D so a osl I u ebr ob g sI or aI sb bc ss b a oI Dn soe sob l I amp oel b b oe I So nl a n y n l boa osl I u

i r My ai onc S t c ct ro ct r n c s

o o 13 h

sDr a B snnr boao b o PTb ol ah sol a sI o a u e amp s I c a o l oe sDr a l I os I b oe bol ah sI D so C sI oesb b oe l ol a b asco snnr boao sI sDr a Pu e c n oo l I oe ol c asDeo l sDr a B cal ys b oe s l I b l a sI D I ob g I amp oel n amp I ob ol oe bol ahu e sI l l I oe l ool amp n o c s ob oe I l n D sol au e sI l l I oe l ool amp asDeo sI s o b l oe a bol as b sI oe ol ah 13 s aa h I I n b oe r b a ol al b oe ns aa hu e Dac e sI oe l ool amp amps n I n b oe r b a ol I y sDo oe b asco l bol ahu

irkycs

uactroctrncr

TheStoryEditor inStab2gives theusermorecontrol thanwaspossible inStab In theoriginalStabsystemhierarchical scripts were hard‐coded in TMKL that itself is coded in the Lisp programming knowledge Thisresulted inseveraldifficulties regarding theusabilityof thesystemFirstly thestoredscripts inStabarenotavailableforreferenceorreviewexceptbyperusingtheactualLispcodeThislackoftransparencycanleadtoinaccurateandimpreciseunderstandingofthestoriesthataregeneratedbyStabasexplainingtheinputdataStab2rsquosStoryEditorprovidesvisualaccesstoboththelibraryofstoredscriptsaswellasthestoriesinstantiatedwhenStab2isrunforaninputdataset

SecondlysinceStabrsquosstoredscriptsarecodedinTMKLitisverydifficultforausertoauthoranewscriptorstoryevaluatethecorrectnessofastoredscriptandmakemodificationstoitStab2StoryEditorwithitsvisualinterface makes these tasks much easier though editing the knowledge elements in a story through theKnowledgeEditorstillisnon‐trivial

FinallytheinitialversionStabdidnotallowmodificationstothestructureofexistingscriptsatruntimeThisimpliesthat it isusefultorunStabjustonceonaninputdatasetHowever ingeneralahumananalystmaywanttorunStabseveraltimesndasheachwithdifferingvariationsofthescripts‐astheanalystrsquosinsightintothecriminal patterns evolves This is perhaps themost important side effect of the use of an interactive StoryEditor in Stab2 Stab2 enables an analyst to conduct ldquowhat ifrdquo simulations Using Stab2rsquos Story Editor andKnowledge Editor illustrated in Figure 7 an analystmay inspectmodify activate or deactivate the scriptsstored in Stab2rsquos Story Library the task and method elements in an individual script or the knowledgeelements likedomainconceptsand relationsTheanalystmay then runStab2 to simulate theeffectsof thepreciseconditionsshecreatedinStab2rsquosvirtualworld

Section34StoryAuthoringinStab2

Whenananalystwants tocreateanewstoryheneeds toopentheeditorandclickonaddstory iconThisopensawhiteareawithapaletteontherightWestartwithpickingtheldquoMainstoryldquoiconfromthepaletteanddropitintothewhiteboardThisrepresentsthenameofthestoryandisdepictedinFigure8(a)AfterthiswedragthebluerectanglethatreadsldquoIntermediaryGoalsrdquoandconnectthistothemainstoryusingaSub‐Goalconnector all ofwhich can be found in the palette The result is shown in Figure 8(b) Once themethod iscreatedwefillitinwithactionsperformedtoachievethatgoalThisisdoneusingtheldquoGoalsActionrdquoiconinthepaletteandthisisshowninFigure8(c)ThedraganddropcapabilitymakesitveryeasyforananalysttoconstructanewstoryplotThetreecangodownasmanylevelsdependingonthecomplexityofthestoryTherelationshipsbetweeneachofthegoalsarecapturedusingthelinksTwokindsoflinkagesarepossiblendashSub‐Goals that suggests that one goal is a child of another and Causal Connection that indicates relationshipsbetweendifferentactionsinsideeachintermediarygoal

Figure8(a)1

stSna

psho

tofStoryAutho

ringinStab2

ndashHerethehu

mananalystdragsth

eicon

mainstoryfrom

thepalette

andplacesitinth

eed

itorareaand

nam

esitasldquostoryplotrdquo

Figure8(b)A

2ndSna

psho

tofStoryAutho

ringin

Stab2

ndashHerethehu

mananalystdragsth

eInterm

ediaryGoalsicon

and

placesitbe

lowth

etopleveltaskAlsotheanalystdragsthesub‐goalslinkagefromthe

paletteand

con

nectsthetw

oblocksand

nam

esthe

bluerectangleas

ldquoMetho

d1rdquo

Figure8(c)ASna

psho

tofStoryAutho

ringinStab2

‐Heretheanalystd

ragsth

eGoalsA

ctionsicon

inside

thealreadycreatedmetho

dHe

then

con

nectstheseactio

nswith

thecausalcon

nectionslinkageinth

epaletteandnameseachactio

n

BehindtheScenes

Figure9(a)A

Sna

psho

tofaXMLfileforeachStoryPlot

Figure9(b)A

Sna

psho

tofaXMLfileforeachStoryPlotndash

Thisim

agehighlightsindividu

alta

gsThe

imageon

the

righth

ighlightsthechildTaskta

gandtheon

ethelefthighlightstheallM

etho

dsta

g

EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)

RelatedWork

ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors

StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)

Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo

Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)

CurrentWork

ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)

Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities

Acknowledgements

Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories

References

AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007

AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8

Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008

BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005

BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005

Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991

ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17

CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985

Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34

CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum

Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)

FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991

Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995

Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009

GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia

Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13

HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)

HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07

JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863

Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009

JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994

Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92

KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999

Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity

LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007

Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180

Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey

MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey

MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691

Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340

Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003

Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81

Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36

PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004

PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005

Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21

RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007

RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318

RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262

RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006

SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924

SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress

SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum

StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132

TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)

ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety

Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623

Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA

WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370

Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics

Stabrsquos Knowledge Representation STABrsquos scripts are represented in the TMKL language A task in TMKL isdefinedbyinputknowledgeelementsoutputknowledgeelementsrequiredinputconditions(pre‐conditions)desiredoutputconditions (post‐conditions)andmethods for implementing the taskAmethod isdefined intermsofsubtasksofthetaskandorderingofthesubtasksandisrepresentedbyafinitestatemachineFinallyknowledge in TMKL is defined in terms of domain concepts and relationships among them the input andoutputknowledgeelements ina taskspecificationrefer tothesedomainconceptsandrelationsMurdockampGoel2008providesdetailsofTMKL

Figure3ThestoryforapoliticalconspiracyintendedtoremoveanopponentfromanelectoralraceActivatednodesaredenotedbyathickoutlinearoundyellowboxes(AdaptedfromGoelet

al2009)

Figure4High‐LevelArchitectureofSTAB(AdaptedfromGoeletal2009)

t ba SV Ts t t SA D t tsD u sDr a H snnr boao b o Tb l ampcr oo sl I n a eso or a u saboC oe ys I l nn ol a l nn ob oe sI cr o y I ob sI I ys I sn sI eal I l nl Ds n l a au SoC oe ol ah o e a og b l I sI c r o y I o o osamp I r b b sob a br nosI D gI l n D boo l oe y I o soe oe ob g I l b sI oe 13 a ca b I oo sl I b l oe b ascob bol a sI oe ol ah 13 s aa hu e ol ah o e a oD b oe amp o esI D ob gb I c b b boe amp o esI D b ascob ol l agsI D amp l ahu esb o DDsI D l amp o esI D ob gb sI o sI boI oso b oe ob gu e ol ah o e a oe I sI bc ob oe I So sI cr o y I o sI oe ys I sn I a c o b oe l y cal bbu oe I sI cr o y I o a br nob sI oe a oas yn l I b ascoC oe I oe b asco sb bsampsna nh bol a sI oe l agsI D amp l ahu oe I nh a oas y b asco sb n a h sI oe l agsI D amp l ahC oe I sosl I n ob g I l b oeo amp o e oe I sI cr o a n bl oD D r o l I nh l I b asco sI boI sb g cou

t ba s tTs tau o l r ocr ob l r a gsI b l sI l aamp o sl I p n n sI boI oso b ascob sI oe l agsI D amp amp l ahC n n I l bsI e br e b asco oeo a amp o e soe I sI cr o y I oC n n ys I l a e br e b ascoC I l I s I yn r l a e br e b ascou e h nnl l S b sI sDr a H snnr boao oe ob gb sI oe c l nsos n l I bcsa h b asco oeo a amp o e soe I sI cr o y I o sI l I ar I I sI D l o u e l I s I yn r l a b asco sb b l I oe I r amp a l ob gb sI b asco oeo a amp o e soe sI cr o y I obu l n o n u dPVV cal ys os nb l o Tba b l I sI DC gI l n D C a ca b I oo sl I b I l r ocr obu

c t n py nc r c v ct ro c n n c s

e sI o a osy o P bhbo amp sb I r Damp I oo sl I l oe l asDsI n o bhbo amp soe casamp a h l r b l I sampcal ysI Dr b a sI o a osl I u e bc ss Dl n b l o P I eI amp I ob sI nr p ds nnl l I nsI r oel asI D I sosI D l o Tb asamp bol as b sI ysbr n l aampC I dss I n r b a ol l I oal n n n b c ob l o Tb a b l I sI D oeal r De Dac es n sI o a u

i r Ty t I ei c c t n r c ci r t c c sua ct ro i c t r n t t

o s h

sDr a O snnr boao b oe l ampcr oo sl I n a eso or a l o PTb bol ah r oel asI D ol l nu e ol ah sol a sb r b sI a o sI D I amp I D sI D oe o PTb asamp bol as bu e b bol as b a sa o b I I o l oe es aa es n b ascob r b sI o u l y aC esn o PTb bol as b bosnn r b 13 b oe gI l n D a ca b I oo sl I nI Dr D C e I D oe sI o aI n o amp l n sI o P ol I n r ol amp o s oaI bno sl I l bol ah r oel a h r b a sI ol oe 13 gI l n D a ca b I oo sl I u

e I l n D sol a ol l sb r sno sI ol o PTb sI o a C I eI n b r b a bb ol oe gI l n D n amp I ob sI oe 13 a ca b I oo sl I l bol ahu o cal ys b oe r b a l I oal n l y a sI sI D oe I osos b sI bol ah b nn b oe a no sl I b amp l I D oe I osos bu e b l I a sI sDr a O a ab ol o b as sI oe ca ysl r b b osl I u

c t n pAsy ai onc S t ct ro i c t r n c s

sDr a ( snnr boao b oe ysbr n bhI oS r b h o PTb bol ah sol au e ob g sb a ca b I o b a oI Dn u b oe r b a amp l r b b l y a ob gC oe ol ah sol a sbcnh b oe 13 bc ss o sl I l oe ob gu e amp oel sb a ca b I o b b L r I l br ob gb oeo bc ss b oe l a asI D l oe br ob gb sI nr sI D so a osl I u ebr ob g sI or aI sb bc ss b a oI Dn soe sob l I amp oel b b oe I So nl a n y n l boa osl I u

i r My ai onc S t c ct ro ct r n c s

o o 13 h

sDr a B snnr boao b o PTb ol ah sol a sI o a u e amp s I c a o l oe sDr a l I os I b oe bol ah sI D so C sI oesb b oe l ol a b asco snnr boao sI sDr a Pu e c n oo l I oe ol c asDeo l sDr a B cal ys b oe s l I b l a sI D I ob g I amp oel n amp I ob ol oe bol ahu e sI l l I oe l ool amp n o c s ob oe I l n D sol au e sI l l I oe l ool amp asDeo sI s o b l oe a bol as b sI oe ol ah 13 s aa h I I n b oe r b a ol al b oe ns aa hu e Dac e sI oe l ool amp amps n I n b oe r b a ol I y sDo oe b asco l bol ahu

irkycs

uactroctrncr

TheStoryEditor inStab2gives theusermorecontrol thanwaspossible inStab In theoriginalStabsystemhierarchical scripts were hard‐coded in TMKL that itself is coded in the Lisp programming knowledge Thisresulted inseveraldifficulties regarding theusabilityof thesystemFirstly thestoredscripts inStabarenotavailableforreferenceorreviewexceptbyperusingtheactualLispcodeThislackoftransparencycanleadtoinaccurateandimpreciseunderstandingofthestoriesthataregeneratedbyStabasexplainingtheinputdataStab2rsquosStoryEditorprovidesvisualaccesstoboththelibraryofstoredscriptsaswellasthestoriesinstantiatedwhenStab2isrunforaninputdataset

SecondlysinceStabrsquosstoredscriptsarecodedinTMKLitisverydifficultforausertoauthoranewscriptorstoryevaluatethecorrectnessofastoredscriptandmakemodificationstoitStab2StoryEditorwithitsvisualinterface makes these tasks much easier though editing the knowledge elements in a story through theKnowledgeEditorstillisnon‐trivial

FinallytheinitialversionStabdidnotallowmodificationstothestructureofexistingscriptsatruntimeThisimpliesthat it isusefultorunStabjustonceonaninputdatasetHowever ingeneralahumananalystmaywanttorunStabseveraltimesndasheachwithdifferingvariationsofthescripts‐astheanalystrsquosinsightintothecriminal patterns evolves This is perhaps themost important side effect of the use of an interactive StoryEditor in Stab2 Stab2 enables an analyst to conduct ldquowhat ifrdquo simulations Using Stab2rsquos Story Editor andKnowledge Editor illustrated in Figure 7 an analystmay inspectmodify activate or deactivate the scriptsstored in Stab2rsquos Story Library the task and method elements in an individual script or the knowledgeelements likedomainconceptsand relationsTheanalystmay then runStab2 to simulate theeffectsof thepreciseconditionsshecreatedinStab2rsquosvirtualworld

Section34StoryAuthoringinStab2

Whenananalystwants tocreateanewstoryheneeds toopentheeditorandclickonaddstory iconThisopensawhiteareawithapaletteontherightWestartwithpickingtheldquoMainstoryldquoiconfromthepaletteanddropitintothewhiteboardThisrepresentsthenameofthestoryandisdepictedinFigure8(a)AfterthiswedragthebluerectanglethatreadsldquoIntermediaryGoalsrdquoandconnectthistothemainstoryusingaSub‐Goalconnector all ofwhich can be found in the palette The result is shown in Figure 8(b) Once themethod iscreatedwefillitinwithactionsperformedtoachievethatgoalThisisdoneusingtheldquoGoalsActionrdquoiconinthepaletteandthisisshowninFigure8(c)ThedraganddropcapabilitymakesitveryeasyforananalysttoconstructanewstoryplotThetreecangodownasmanylevelsdependingonthecomplexityofthestoryTherelationshipsbetweeneachofthegoalsarecapturedusingthelinksTwokindsoflinkagesarepossiblendashSub‐Goals that suggests that one goal is a child of another and Causal Connection that indicates relationshipsbetweendifferentactionsinsideeachintermediarygoal

Figure8(a)1

stSna

psho

tofStoryAutho

ringinStab2

ndashHerethehu

mananalystdragsth

eicon

mainstoryfrom

thepalette

andplacesitinth

eed

itorareaand

nam

esitasldquostoryplotrdquo

Figure8(b)A

2ndSna

psho

tofStoryAutho

ringin

Stab2

ndashHerethehu

mananalystdragsth

eInterm

ediaryGoalsicon

and

placesitbe

lowth

etopleveltaskAlsotheanalystdragsthesub‐goalslinkagefromthe

paletteand

con

nectsthetw

oblocksand

nam

esthe

bluerectangleas

ldquoMetho

d1rdquo

Figure8(c)ASna

psho

tofStoryAutho

ringinStab2

‐Heretheanalystd

ragsth

eGoalsA

ctionsicon

inside

thealreadycreatedmetho

dHe

then

con

nectstheseactio

nswith

thecausalcon

nectionslinkageinth

epaletteandnameseachactio

n

BehindtheScenes

Figure9(a)A

Sna

psho

tofaXMLfileforeachStoryPlot

Figure9(b)A

Sna

psho

tofaXMLfileforeachStoryPlotndash

Thisim

agehighlightsindividu

alta

gsThe

imageon

the

righth

ighlightsthechildTaskta

gandtheon

ethelefthighlightstheallM

etho

dsta

g

EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)

RelatedWork

ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors

StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)

Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo

Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)

CurrentWork

ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)

Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities

Acknowledgements

Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories

References

AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007

AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8

Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008

BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005

BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005

Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991

ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17

CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985

Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34

CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum

Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)

FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991

Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995

Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009

GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia

Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13

HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)

HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07

JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863

Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009

JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994

Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92

KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999

Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity

LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007

Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180

Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey

MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey

MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691

Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340

Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003

Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81

Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36

PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004

PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005

Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21

RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007

RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318

RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262

RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006

SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924

SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress

SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum

StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132

TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)

ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety

Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623

Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA

WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370

Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics

t ba SV Ts t t SA D t tsD u sDr a H snnr boao b o Tb l ampcr oo sl I n a eso or a u saboC oe ys I l nn ol a l nn ob oe sI cr o y I ob sI I ys I sn sI eal I l nl Ds n l a au SoC oe ol ah o e a og b l I sI c r o y I o o osamp I r b b sob a br nosI D gI l n D boo l oe y I o soe oe ob g I l b sI oe 13 a ca b I oo sl I b l oe b ascob bol a sI oe ol ah 13 s aa hu e ol ah o e a oD b oe amp o esI D ob gb I c b b boe amp o esI D b ascob ol l agsI D amp l ahu esb o DDsI D l amp o esI D ob gb sI o sI boI oso b oe ob gu e ol ah o e a oe I sI bc ob oe I So sI cr o y I o sI oe ys I sn I a c o b oe l y cal bbu oe I sI cr o y I o a br nob sI oe a oas yn l I b ascoC oe I oe b asco sb bsampsna nh bol a sI oe l agsI D amp l ahu oe I nh a oas y b asco sb n a h sI oe l agsI D amp l ahC oe I sosl I n ob g I l b oeo amp o e oe I sI cr o a n bl oD D r o l I nh l I b asco sI boI sb g cou

t ba s tTs tau o l r ocr ob l r a gsI b l sI l aamp o sl I p n n sI boI oso b ascob sI oe l agsI D amp amp l ahC n n I l bsI e br e b asco oeo a amp o e soe I sI cr o y I oC n n ys I l a e br e b ascoC I l I s I yn r l a e br e b ascou e h nnl l S b sI sDr a H snnr boao oe ob gb sI oe c l nsos n l I bcsa h b asco oeo a amp o e soe I sI cr o y I o sI l I ar I I sI D l o u e l I s I yn r l a b asco sb b l I oe I r amp a l ob gb sI b asco oeo a amp o e soe sI cr o y I obu l n o n u dPVV cal ys os nb l o Tba b l I sI DC gI l n D C a ca b I oo sl I b I l r ocr obu

c t n py nc r c v ct ro c n n c s

e sI o a osy o P bhbo amp sb I r Damp I oo sl I l oe l asDsI n o bhbo amp soe casamp a h l r b l I sampcal ysI Dr b a sI o a osl I u e bc ss Dl n b l o P I eI amp I ob sI nr p ds nnl l I nsI r oel asI D I sosI D l o Tb asamp bol as b sI ysbr n l aampC I dss I n r b a ol l I oal n n n b c ob l o Tb a b l I sI D oeal r De Dac es n sI o a u

i r Ty t I ei c c t n r c ci r t c c sua ct ro i c t r n t t

o s h

sDr a O snnr boao b oe l ampcr oo sl I n a eso or a l o PTb bol ah r oel asI D ol l nu e ol ah sol a sb r b sI a o sI D I amp I D sI D oe o PTb asamp bol as bu e b bol as b a sa o b I I o l oe es aa es n b ascob r b sI o u l y aC esn o PTb bol as b bosnn r b 13 b oe gI l n D a ca b I oo sl I nI Dr D C e I D oe sI o aI n o amp l n sI o P ol I n r ol amp o s oaI bno sl I l bol ah r oel a h r b a sI ol oe 13 gI l n D a ca b I oo sl I u

e I l n D sol a ol l sb r sno sI ol o PTb sI o a C I eI n b r b a bb ol oe gI l n D n amp I ob sI oe 13 a ca b I oo sl I l bol ahu o cal ys b oe r b a l I oal n l y a sI sI D oe I osos b sI bol ah b nn b oe a no sl I b amp l I D oe I osos bu e b l I a sI sDr a O a ab ol o b as sI oe ca ysl r b b osl I u

c t n pAsy ai onc S t ct ro i c t r n c s

sDr a ( snnr boao b oe ysbr n bhI oS r b h o PTb bol ah sol au e ob g sb a ca b I o b a oI Dn u b oe r b a amp l r b b l y a ob gC oe ol ah sol a sbcnh b oe 13 bc ss o sl I l oe ob gu e amp oel sb a ca b I o b b L r I l br ob gb oeo bc ss b oe l a asI D l oe br ob gb sI nr sI D so a osl I u ebr ob g sI or aI sb bc ss b a oI Dn soe sob l I amp oel b b oe I So nl a n y n l boa osl I u

i r My ai onc S t c ct ro ct r n c s

o o 13 h

sDr a B snnr boao b o PTb ol ah sol a sI o a u e amp s I c a o l oe sDr a l I os I b oe bol ah sI D so C sI oesb b oe l ol a b asco snnr boao sI sDr a Pu e c n oo l I oe ol c asDeo l sDr a B cal ys b oe s l I b l a sI D I ob g I amp oel n amp I ob ol oe bol ahu e sI l l I oe l ool amp n o c s ob oe I l n D sol au e sI l l I oe l ool amp asDeo sI s o b l oe a bol as b sI oe ol ah 13 s aa h I I n b oe r b a ol al b oe ns aa hu e Dac e sI oe l ool amp amps n I n b oe r b a ol I y sDo oe b asco l bol ahu

irkycs

uactroctrncr

TheStoryEditor inStab2gives theusermorecontrol thanwaspossible inStab In theoriginalStabsystemhierarchical scripts were hard‐coded in TMKL that itself is coded in the Lisp programming knowledge Thisresulted inseveraldifficulties regarding theusabilityof thesystemFirstly thestoredscripts inStabarenotavailableforreferenceorreviewexceptbyperusingtheactualLispcodeThislackoftransparencycanleadtoinaccurateandimpreciseunderstandingofthestoriesthataregeneratedbyStabasexplainingtheinputdataStab2rsquosStoryEditorprovidesvisualaccesstoboththelibraryofstoredscriptsaswellasthestoriesinstantiatedwhenStab2isrunforaninputdataset

SecondlysinceStabrsquosstoredscriptsarecodedinTMKLitisverydifficultforausertoauthoranewscriptorstoryevaluatethecorrectnessofastoredscriptandmakemodificationstoitStab2StoryEditorwithitsvisualinterface makes these tasks much easier though editing the knowledge elements in a story through theKnowledgeEditorstillisnon‐trivial

FinallytheinitialversionStabdidnotallowmodificationstothestructureofexistingscriptsatruntimeThisimpliesthat it isusefultorunStabjustonceonaninputdatasetHowever ingeneralahumananalystmaywanttorunStabseveraltimesndasheachwithdifferingvariationsofthescripts‐astheanalystrsquosinsightintothecriminal patterns evolves This is perhaps themost important side effect of the use of an interactive StoryEditor in Stab2 Stab2 enables an analyst to conduct ldquowhat ifrdquo simulations Using Stab2rsquos Story Editor andKnowledge Editor illustrated in Figure 7 an analystmay inspectmodify activate or deactivate the scriptsstored in Stab2rsquos Story Library the task and method elements in an individual script or the knowledgeelements likedomainconceptsand relationsTheanalystmay then runStab2 to simulate theeffectsof thepreciseconditionsshecreatedinStab2rsquosvirtualworld

Section34StoryAuthoringinStab2

Whenananalystwants tocreateanewstoryheneeds toopentheeditorandclickonaddstory iconThisopensawhiteareawithapaletteontherightWestartwithpickingtheldquoMainstoryldquoiconfromthepaletteanddropitintothewhiteboardThisrepresentsthenameofthestoryandisdepictedinFigure8(a)AfterthiswedragthebluerectanglethatreadsldquoIntermediaryGoalsrdquoandconnectthistothemainstoryusingaSub‐Goalconnector all ofwhich can be found in the palette The result is shown in Figure 8(b) Once themethod iscreatedwefillitinwithactionsperformedtoachievethatgoalThisisdoneusingtheldquoGoalsActionrdquoiconinthepaletteandthisisshowninFigure8(c)ThedraganddropcapabilitymakesitveryeasyforananalysttoconstructanewstoryplotThetreecangodownasmanylevelsdependingonthecomplexityofthestoryTherelationshipsbetweeneachofthegoalsarecapturedusingthelinksTwokindsoflinkagesarepossiblendashSub‐Goals that suggests that one goal is a child of another and Causal Connection that indicates relationshipsbetweendifferentactionsinsideeachintermediarygoal

Figure8(a)1

stSna

psho

tofStoryAutho

ringinStab2

ndashHerethehu

mananalystdragsth

eicon

mainstoryfrom

thepalette

andplacesitinth

eed

itorareaand

nam

esitasldquostoryplotrdquo

Figure8(b)A

2ndSna

psho

tofStoryAutho

ringin

Stab2

ndashHerethehu

mananalystdragsth

eInterm

ediaryGoalsicon

and

placesitbe

lowth

etopleveltaskAlsotheanalystdragsthesub‐goalslinkagefromthe

paletteand

con

nectsthetw

oblocksand

nam

esthe

bluerectangleas

ldquoMetho

d1rdquo

Figure8(c)ASna

psho

tofStoryAutho

ringinStab2

‐Heretheanalystd

ragsth

eGoalsA

ctionsicon

inside

thealreadycreatedmetho

dHe

then

con

nectstheseactio

nswith

thecausalcon

nectionslinkageinth

epaletteandnameseachactio

n

BehindtheScenes

Figure9(a)A

Sna

psho

tofaXMLfileforeachStoryPlot

Figure9(b)A

Sna

psho

tofaXMLfileforeachStoryPlotndash

Thisim

agehighlightsindividu

alta

gsThe

imageon

the

righth

ighlightsthechildTaskta

gandtheon

ethelefthighlightstheallM

etho

dsta

g

EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)

RelatedWork

ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors

StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)

Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo

Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)

CurrentWork

ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)

Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities

Acknowledgements

Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories

References

AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007

AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8

Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008

BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005

BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005

Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991

ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17

CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985

Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34

CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum

Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)

FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991

Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995

Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009

GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia

Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13

HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)

HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07

JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863

Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009

JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994

Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92

KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999

Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity

LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007

Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180

Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey

MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey

MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691

Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340

Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003

Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81

Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36

PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004

PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005

Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21

RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007

RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318

RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262

RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006

SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924

SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress

SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum

StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132

TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)

ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety

Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623

Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA

WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370

Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics

o s h

sDr a O snnr boao b oe l ampcr oo sl I n a eso or a l o PTb bol ah r oel asI D ol l nu e ol ah sol a sb r b sI a o sI D I amp I D sI D oe o PTb asamp bol as bu e b bol as b a sa o b I I o l oe es aa es n b ascob r b sI o u l y aC esn o PTb bol as b bosnn r b 13 b oe gI l n D a ca b I oo sl I nI Dr D C e I D oe sI o aI n o amp l n sI o P ol I n r ol amp o s oaI bno sl I l bol ah r oel a h r b a sI ol oe 13 gI l n D a ca b I oo sl I u

e I l n D sol a ol l sb r sno sI ol o PTb sI o a C I eI n b r b a bb ol oe gI l n D n amp I ob sI oe 13 a ca b I oo sl I l bol ahu o cal ys b oe r b a l I oal n l y a sI sI D oe I osos b sI bol ah b nn b oe a no sl I b amp l I D oe I osos bu e b l I a sI sDr a O a ab ol o b as sI oe ca ysl r b b osl I u

c t n pAsy ai onc S t ct ro i c t r n c s

sDr a ( snnr boao b oe ysbr n bhI oS r b h o PTb bol ah sol au e ob g sb a ca b I o b a oI Dn u b oe r b a amp l r b b l y a ob gC oe ol ah sol a sbcnh b oe 13 bc ss o sl I l oe ob gu e amp oel sb a ca b I o b b L r I l br ob gb oeo bc ss b oe l a asI D l oe br ob gb sI nr sI D so a osl I u ebr ob g sI or aI sb bc ss b a oI Dn soe sob l I amp oel b b oe I So nl a n y n l boa osl I u

i r My ai onc S t c ct ro ct r n c s

o o 13 h

sDr a B snnr boao b o PTb ol ah sol a sI o a u e amp s I c a o l oe sDr a l I os I b oe bol ah sI D so C sI oesb b oe l ol a b asco snnr boao sI sDr a Pu e c n oo l I oe ol c asDeo l sDr a B cal ys b oe s l I b l a sI D I ob g I amp oel n amp I ob ol oe bol ahu e sI l l I oe l ool amp n o c s ob oe I l n D sol au e sI l l I oe l ool amp asDeo sI s o b l oe a bol as b sI oe ol ah 13 s aa h I I n b oe r b a ol al b oe ns aa hu e Dac e sI oe l ool amp amps n I n b oe r b a ol I y sDo oe b asco l bol ahu

irkycs

uactroctrncr

TheStoryEditor inStab2gives theusermorecontrol thanwaspossible inStab In theoriginalStabsystemhierarchical scripts were hard‐coded in TMKL that itself is coded in the Lisp programming knowledge Thisresulted inseveraldifficulties regarding theusabilityof thesystemFirstly thestoredscripts inStabarenotavailableforreferenceorreviewexceptbyperusingtheactualLispcodeThislackoftransparencycanleadtoinaccurateandimpreciseunderstandingofthestoriesthataregeneratedbyStabasexplainingtheinputdataStab2rsquosStoryEditorprovidesvisualaccesstoboththelibraryofstoredscriptsaswellasthestoriesinstantiatedwhenStab2isrunforaninputdataset

SecondlysinceStabrsquosstoredscriptsarecodedinTMKLitisverydifficultforausertoauthoranewscriptorstoryevaluatethecorrectnessofastoredscriptandmakemodificationstoitStab2StoryEditorwithitsvisualinterface makes these tasks much easier though editing the knowledge elements in a story through theKnowledgeEditorstillisnon‐trivial

FinallytheinitialversionStabdidnotallowmodificationstothestructureofexistingscriptsatruntimeThisimpliesthat it isusefultorunStabjustonceonaninputdatasetHowever ingeneralahumananalystmaywanttorunStabseveraltimesndasheachwithdifferingvariationsofthescripts‐astheanalystrsquosinsightintothecriminal patterns evolves This is perhaps themost important side effect of the use of an interactive StoryEditor in Stab2 Stab2 enables an analyst to conduct ldquowhat ifrdquo simulations Using Stab2rsquos Story Editor andKnowledge Editor illustrated in Figure 7 an analystmay inspectmodify activate or deactivate the scriptsstored in Stab2rsquos Story Library the task and method elements in an individual script or the knowledgeelements likedomainconceptsand relationsTheanalystmay then runStab2 to simulate theeffectsof thepreciseconditionsshecreatedinStab2rsquosvirtualworld

Section34StoryAuthoringinStab2

Whenananalystwants tocreateanewstoryheneeds toopentheeditorandclickonaddstory iconThisopensawhiteareawithapaletteontherightWestartwithpickingtheldquoMainstoryldquoiconfromthepaletteanddropitintothewhiteboardThisrepresentsthenameofthestoryandisdepictedinFigure8(a)AfterthiswedragthebluerectanglethatreadsldquoIntermediaryGoalsrdquoandconnectthistothemainstoryusingaSub‐Goalconnector all ofwhich can be found in the palette The result is shown in Figure 8(b) Once themethod iscreatedwefillitinwithactionsperformedtoachievethatgoalThisisdoneusingtheldquoGoalsActionrdquoiconinthepaletteandthisisshowninFigure8(c)ThedraganddropcapabilitymakesitveryeasyforananalysttoconstructanewstoryplotThetreecangodownasmanylevelsdependingonthecomplexityofthestoryTherelationshipsbetweeneachofthegoalsarecapturedusingthelinksTwokindsoflinkagesarepossiblendashSub‐Goals that suggests that one goal is a child of another and Causal Connection that indicates relationshipsbetweendifferentactionsinsideeachintermediarygoal

Figure8(a)1

stSna

psho

tofStoryAutho

ringinStab2

ndashHerethehu

mananalystdragsth

eicon

mainstoryfrom

thepalette

andplacesitinth

eed

itorareaand

nam

esitasldquostoryplotrdquo

Figure8(b)A

2ndSna

psho

tofStoryAutho

ringin

Stab2

ndashHerethehu

mananalystdragsth

eInterm

ediaryGoalsicon

and

placesitbe

lowth

etopleveltaskAlsotheanalystdragsthesub‐goalslinkagefromthe

paletteand

con

nectsthetw

oblocksand

nam

esthe

bluerectangleas

ldquoMetho

d1rdquo

Figure8(c)ASna

psho

tofStoryAutho

ringinStab2

‐Heretheanalystd

ragsth

eGoalsA

ctionsicon

inside

thealreadycreatedmetho

dHe

then

con

nectstheseactio

nswith

thecausalcon

nectionslinkageinth

epaletteandnameseachactio

n

BehindtheScenes

Figure9(a)A

Sna

psho

tofaXMLfileforeachStoryPlot

Figure9(b)A

Sna

psho

tofaXMLfileforeachStoryPlotndash

Thisim

agehighlightsindividu

alta

gsThe

imageon

the

righth

ighlightsthechildTaskta

gandtheon

ethelefthighlightstheallM

etho

dsta

g

EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)

RelatedWork

ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors

StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)

Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo

Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)

CurrentWork

ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)

Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities

Acknowledgements

Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories

References

AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007

AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8

Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008

BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005

BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005

Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991

ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17

CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985

Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34

CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum

Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)

FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991

Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995

Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009

GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia

Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13

HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)

HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07

JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863

Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009

JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994

Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92

KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999

Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity

LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007

Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180

Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey

MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey

MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691

Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340

Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003

Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81

Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36

PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004

PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005

Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21

RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007

RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318

RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262

RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006

SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924

SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress

SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum

StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132

TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)

ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety

Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623

Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA

WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370

Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics

irkycs

uactroctrncr

TheStoryEditor inStab2gives theusermorecontrol thanwaspossible inStab In theoriginalStabsystemhierarchical scripts were hard‐coded in TMKL that itself is coded in the Lisp programming knowledge Thisresulted inseveraldifficulties regarding theusabilityof thesystemFirstly thestoredscripts inStabarenotavailableforreferenceorreviewexceptbyperusingtheactualLispcodeThislackoftransparencycanleadtoinaccurateandimpreciseunderstandingofthestoriesthataregeneratedbyStabasexplainingtheinputdataStab2rsquosStoryEditorprovidesvisualaccesstoboththelibraryofstoredscriptsaswellasthestoriesinstantiatedwhenStab2isrunforaninputdataset

SecondlysinceStabrsquosstoredscriptsarecodedinTMKLitisverydifficultforausertoauthoranewscriptorstoryevaluatethecorrectnessofastoredscriptandmakemodificationstoitStab2StoryEditorwithitsvisualinterface makes these tasks much easier though editing the knowledge elements in a story through theKnowledgeEditorstillisnon‐trivial

FinallytheinitialversionStabdidnotallowmodificationstothestructureofexistingscriptsatruntimeThisimpliesthat it isusefultorunStabjustonceonaninputdatasetHowever ingeneralahumananalystmaywanttorunStabseveraltimesndasheachwithdifferingvariationsofthescripts‐astheanalystrsquosinsightintothecriminal patterns evolves This is perhaps themost important side effect of the use of an interactive StoryEditor in Stab2 Stab2 enables an analyst to conduct ldquowhat ifrdquo simulations Using Stab2rsquos Story Editor andKnowledge Editor illustrated in Figure 7 an analystmay inspectmodify activate or deactivate the scriptsstored in Stab2rsquos Story Library the task and method elements in an individual script or the knowledgeelements likedomainconceptsand relationsTheanalystmay then runStab2 to simulate theeffectsof thepreciseconditionsshecreatedinStab2rsquosvirtualworld

Section34StoryAuthoringinStab2

Whenananalystwants tocreateanewstoryheneeds toopentheeditorandclickonaddstory iconThisopensawhiteareawithapaletteontherightWestartwithpickingtheldquoMainstoryldquoiconfromthepaletteanddropitintothewhiteboardThisrepresentsthenameofthestoryandisdepictedinFigure8(a)AfterthiswedragthebluerectanglethatreadsldquoIntermediaryGoalsrdquoandconnectthistothemainstoryusingaSub‐Goalconnector all ofwhich can be found in the palette The result is shown in Figure 8(b) Once themethod iscreatedwefillitinwithactionsperformedtoachievethatgoalThisisdoneusingtheldquoGoalsActionrdquoiconinthepaletteandthisisshowninFigure8(c)ThedraganddropcapabilitymakesitveryeasyforananalysttoconstructanewstoryplotThetreecangodownasmanylevelsdependingonthecomplexityofthestoryTherelationshipsbetweeneachofthegoalsarecapturedusingthelinksTwokindsoflinkagesarepossiblendashSub‐Goals that suggests that one goal is a child of another and Causal Connection that indicates relationshipsbetweendifferentactionsinsideeachintermediarygoal

Figure8(a)1

stSna

psho

tofStoryAutho

ringinStab2

ndashHerethehu

mananalystdragsth

eicon

mainstoryfrom

thepalette

andplacesitinth

eed

itorareaand

nam

esitasldquostoryplotrdquo

Figure8(b)A

2ndSna

psho

tofStoryAutho

ringin

Stab2

ndashHerethehu

mananalystdragsth

eInterm

ediaryGoalsicon

and

placesitbe

lowth

etopleveltaskAlsotheanalystdragsthesub‐goalslinkagefromthe

paletteand

con

nectsthetw

oblocksand

nam

esthe

bluerectangleas

ldquoMetho

d1rdquo

Figure8(c)ASna

psho

tofStoryAutho

ringinStab2

‐Heretheanalystd

ragsth

eGoalsA

ctionsicon

inside

thealreadycreatedmetho

dHe

then

con

nectstheseactio

nswith

thecausalcon

nectionslinkageinth

epaletteandnameseachactio

n

BehindtheScenes

Figure9(a)A

Sna

psho

tofaXMLfileforeachStoryPlot

Figure9(b)A

Sna

psho

tofaXMLfileforeachStoryPlotndash

Thisim

agehighlightsindividu

alta

gsThe

imageon

the

righth

ighlightsthechildTaskta

gandtheon

ethelefthighlightstheallM

etho

dsta

g

EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)

RelatedWork

ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors

StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)

Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo

Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)

CurrentWork

ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)

Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities

Acknowledgements

Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories

References

AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007

AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8

Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008

BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005

BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005

Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991

ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17

CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985

Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34

CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum

Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)

FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991

Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995

Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009

GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia

Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13

HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)

HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07

JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863

Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009

JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994

Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92

KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999

Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity

LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007

Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180

Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey

MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey

MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691

Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340

Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003

Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81

Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36

PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004

PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005

Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21

RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007

RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318

RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262

RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006

SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924

SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress

SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum

StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132

TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)

ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety

Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623

Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA

WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370

Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics

TheStoryEditor inStab2gives theusermorecontrol thanwaspossible inStab In theoriginalStabsystemhierarchical scripts were hard‐coded in TMKL that itself is coded in the Lisp programming knowledge Thisresulted inseveraldifficulties regarding theusabilityof thesystemFirstly thestoredscripts inStabarenotavailableforreferenceorreviewexceptbyperusingtheactualLispcodeThislackoftransparencycanleadtoinaccurateandimpreciseunderstandingofthestoriesthataregeneratedbyStabasexplainingtheinputdataStab2rsquosStoryEditorprovidesvisualaccesstoboththelibraryofstoredscriptsaswellasthestoriesinstantiatedwhenStab2isrunforaninputdataset

SecondlysinceStabrsquosstoredscriptsarecodedinTMKLitisverydifficultforausertoauthoranewscriptorstoryevaluatethecorrectnessofastoredscriptandmakemodificationstoitStab2StoryEditorwithitsvisualinterface makes these tasks much easier though editing the knowledge elements in a story through theKnowledgeEditorstillisnon‐trivial

FinallytheinitialversionStabdidnotallowmodificationstothestructureofexistingscriptsatruntimeThisimpliesthat it isusefultorunStabjustonceonaninputdatasetHowever ingeneralahumananalystmaywanttorunStabseveraltimesndasheachwithdifferingvariationsofthescripts‐astheanalystrsquosinsightintothecriminal patterns evolves This is perhaps themost important side effect of the use of an interactive StoryEditor in Stab2 Stab2 enables an analyst to conduct ldquowhat ifrdquo simulations Using Stab2rsquos Story Editor andKnowledge Editor illustrated in Figure 7 an analystmay inspectmodify activate or deactivate the scriptsstored in Stab2rsquos Story Library the task and method elements in an individual script or the knowledgeelements likedomainconceptsand relationsTheanalystmay then runStab2 to simulate theeffectsof thepreciseconditionsshecreatedinStab2rsquosvirtualworld

Section34StoryAuthoringinStab2

Whenananalystwants tocreateanewstoryheneeds toopentheeditorandclickonaddstory iconThisopensawhiteareawithapaletteontherightWestartwithpickingtheldquoMainstoryldquoiconfromthepaletteanddropitintothewhiteboardThisrepresentsthenameofthestoryandisdepictedinFigure8(a)AfterthiswedragthebluerectanglethatreadsldquoIntermediaryGoalsrdquoandconnectthistothemainstoryusingaSub‐Goalconnector all ofwhich can be found in the palette The result is shown in Figure 8(b) Once themethod iscreatedwefillitinwithactionsperformedtoachievethatgoalThisisdoneusingtheldquoGoalsActionrdquoiconinthepaletteandthisisshowninFigure8(c)ThedraganddropcapabilitymakesitveryeasyforananalysttoconstructanewstoryplotThetreecangodownasmanylevelsdependingonthecomplexityofthestoryTherelationshipsbetweeneachofthegoalsarecapturedusingthelinksTwokindsoflinkagesarepossiblendashSub‐Goals that suggests that one goal is a child of another and Causal Connection that indicates relationshipsbetweendifferentactionsinsideeachintermediarygoal

Figure8(a)1

stSna

psho

tofStoryAutho

ringinStab2

ndashHerethehu

mananalystdragsth

eicon

mainstoryfrom

thepalette

andplacesitinth

eed

itorareaand

nam

esitasldquostoryplotrdquo

Figure8(b)A

2ndSna

psho

tofStoryAutho

ringin

Stab2

ndashHerethehu

mananalystdragsth

eInterm

ediaryGoalsicon

and

placesitbe

lowth

etopleveltaskAlsotheanalystdragsthesub‐goalslinkagefromthe

paletteand

con

nectsthetw

oblocksand

nam

esthe

bluerectangleas

ldquoMetho

d1rdquo

Figure8(c)ASna

psho

tofStoryAutho

ringinStab2

‐Heretheanalystd

ragsth

eGoalsA

ctionsicon

inside

thealreadycreatedmetho

dHe

then

con

nectstheseactio

nswith

thecausalcon

nectionslinkageinth

epaletteandnameseachactio

n

BehindtheScenes

Figure9(a)A

Sna

psho

tofaXMLfileforeachStoryPlot

Figure9(b)A

Sna

psho

tofaXMLfileforeachStoryPlotndash

Thisim

agehighlightsindividu

alta

gsThe

imageon

the

righth

ighlightsthechildTaskta

gandtheon

ethelefthighlightstheallM

etho

dsta

g

EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)

RelatedWork

ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors

StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)

Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo

Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)

CurrentWork

ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)

Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities

Acknowledgements

Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories

References

AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007

AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8

Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008

BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005

BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005

Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991

ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17

CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985

Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34

CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum

Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)

FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991

Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995

Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009

GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia

Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13

HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)

HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07

JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863

Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009

JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994

Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92

KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999

Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity

LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007

Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180

Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey

MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey

MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691

Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340

Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003

Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81

Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36

PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004

PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005

Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21

RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007

RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318

RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262

RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006

SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924

SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress

SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum

StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132

TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)

ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety

Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623

Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA

WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370

Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics

Figure8(a)1

stSna

psho

tofStoryAutho

ringinStab2

ndashHerethehu

mananalystdragsth

eicon

mainstoryfrom

thepalette

andplacesitinth

eed

itorareaand

nam

esitasldquostoryplotrdquo

Figure8(b)A

2ndSna

psho

tofStoryAutho

ringin

Stab2

ndashHerethehu

mananalystdragsth

eInterm

ediaryGoalsicon

and

placesitbe

lowth

etopleveltaskAlsotheanalystdragsthesub‐goalslinkagefromthe

paletteand

con

nectsthetw

oblocksand

nam

esthe

bluerectangleas

ldquoMetho

d1rdquo

Figure8(c)ASna

psho

tofStoryAutho

ringinStab2

‐Heretheanalystd

ragsth

eGoalsA

ctionsicon

inside

thealreadycreatedmetho

dHe

then

con

nectstheseactio

nswith

thecausalcon

nectionslinkageinth

epaletteandnameseachactio

n

BehindtheScenes

Figure9(a)A

Sna

psho

tofaXMLfileforeachStoryPlot

Figure9(b)A

Sna

psho

tofaXMLfileforeachStoryPlotndash

Thisim

agehighlightsindividu

alta

gsThe

imageon

the

righth

ighlightsthechildTaskta

gandtheon

ethelefthighlightstheallM

etho

dsta

g

EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)

RelatedWork

ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors

StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)

Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo

Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)

CurrentWork

ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)

Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities

Acknowledgements

Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories

References

AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007

AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8

Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008

BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005

BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005

Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991

ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17

CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985

Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34

CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum

Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)

FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991

Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995

Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009

GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia

Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13

HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)

HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07

JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863

Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009

JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994

Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92

KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999

Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity

LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007

Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180

Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey

MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey

MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691

Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340

Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003

Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81

Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36

PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004

PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005

Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21

RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007

RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318

RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262

RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006

SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924

SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress

SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum

StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132

TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)

ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety

Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623

Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA

WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370

Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics

Figure8(b)A

2ndSna

psho

tofStoryAutho

ringin

Stab2

ndashHerethehu

mananalystdragsth

eInterm

ediaryGoalsicon

and

placesitbe

lowth

etopleveltaskAlsotheanalystdragsthesub‐goalslinkagefromthe

paletteand

con

nectsthetw

oblocksand

nam

esthe

bluerectangleas

ldquoMetho

d1rdquo

Figure8(c)ASna

psho

tofStoryAutho

ringinStab2

‐Heretheanalystd

ragsth

eGoalsA

ctionsicon

inside

thealreadycreatedmetho

dHe

then

con

nectstheseactio

nswith

thecausalcon

nectionslinkageinth

epaletteandnameseachactio

n

BehindtheScenes

Figure9(a)A

Sna

psho

tofaXMLfileforeachStoryPlot

Figure9(b)A

Sna

psho

tofaXMLfileforeachStoryPlotndash

Thisim

agehighlightsindividu

alta

gsThe

imageon

the

righth

ighlightsthechildTaskta

gandtheon

ethelefthighlightstheallM

etho

dsta

g

EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)

RelatedWork

ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors

StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)

Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo

Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)

CurrentWork

ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)

Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities

Acknowledgements

Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories

References

AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007

AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8

Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008

BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005

BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005

Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991

ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17

CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985

Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34

CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum

Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)

FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991

Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995

Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009

GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia

Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13

HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)

HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07

JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863

Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009

JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994

Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92

KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999

Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity

LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007

Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180

Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey

MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey

MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691

Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340

Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003

Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81

Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36

PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004

PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005

Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21

RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007

RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318

RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262

RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006

SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924

SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress

SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum

StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132

TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)

ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety

Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623

Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA

WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370

Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics

Figure8(c)ASna

psho

tofStoryAutho

ringinStab2

‐Heretheanalystd

ragsth

eGoalsA

ctionsicon

inside

thealreadycreatedmetho

dHe

then

con

nectstheseactio

nswith

thecausalcon

nectionslinkageinth

epaletteandnameseachactio

n

BehindtheScenes

Figure9(a)A

Sna

psho

tofaXMLfileforeachStoryPlot

Figure9(b)A

Sna

psho

tofaXMLfileforeachStoryPlotndash

Thisim

agehighlightsindividu

alta

gsThe

imageon

the

righth

ighlightsthechildTaskta

gandtheon

ethelefthighlightstheallM

etho

dsta

g

EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)

RelatedWork

ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors

StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)

Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo

Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)

CurrentWork

ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)

Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities

Acknowledgements

Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories

References

AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007

AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8

Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008

BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005

BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005

Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991

ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17

CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985

Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34

CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum

Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)

FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991

Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995

Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009

GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia

Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13

HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)

HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07

JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863

Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009

JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994

Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92

KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999

Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity

LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007

Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180

Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey

MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey

MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691

Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340

Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003

Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81

Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36

PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004

PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005

Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21

RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007

RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318

RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262

RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006

SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924

SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress

SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum

StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132

TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)

ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety

Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623

Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA

WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370

Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics

BehindtheScenes

Figure9(a)A

Sna

psho

tofaXMLfileforeachStoryPlot

Figure9(b)A

Sna

psho

tofaXMLfileforeachStoryPlotndash

Thisim

agehighlightsindividu

alta

gsThe

imageon

the

righth

ighlightsthechildTaskta

gandtheon

ethelefthighlightstheallM

etho

dsta

g

EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)

RelatedWork

ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors

StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)

Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo

Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)

CurrentWork

ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)

Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities

Acknowledgements

Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories

References

AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007

AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8

Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008

BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005

BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005

Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991

ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17

CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985

Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34

CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum

Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)

FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991

Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995

Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009

GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia

Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13

HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)

HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07

JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863

Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009

JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994

Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92

KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999

Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity

LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007

Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180

Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey

MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey

MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691

Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340

Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003

Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81

Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36

PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004

PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005

Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21

RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007

RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318

RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262

RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006

SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924

SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress

SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum

StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132

TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)

ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety

Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623

Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA

WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370

Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics

Figure9(b)A

Sna

psho

tofaXMLfileforeachStoryPlotndash

Thisim

agehighlightsindividu

alta

gsThe

imageon

the

righth

ighlightsthechildTaskta

gandtheon

ethelefthighlightstheallM

etho

dsta

g

EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)

RelatedWork

ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors

StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)

Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo

Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)

CurrentWork

ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)

Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities

Acknowledgements

Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories

References

AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007

AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8

Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008

BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005

BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005

Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991

ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17

CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985

Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34

CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum

Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)

FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991

Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995

Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009

GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia

Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13

HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)

HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07

JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863

Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009

JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994

Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92

KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999

Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity

LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007

Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180

Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey

MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey

MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691

Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340

Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003

Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81

Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36

PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004

PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005

Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21

RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007

RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318

RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262

RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006

SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924

SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress

SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum

StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132

TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)

ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety

Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623

Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA

WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370

Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics

EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)

RelatedWork

ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors

StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)

Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo

Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)

CurrentWork

ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)

Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities

Acknowledgements

Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories

References

AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007

AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8

Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008

BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005

BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005

Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991

ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17

CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985

Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34

CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum

Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)

FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991

Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995

Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009

GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia

Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13

HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)

HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07

JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863

Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009

JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994

Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92

KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999

Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity

LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007

Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180

Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey

MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey

MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691

Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340

Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003

Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81

Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36

PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004

PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005

Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21

RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007

RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318

RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262

RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006

SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924

SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress

SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum

StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132

TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)

ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety

Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623

Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA

WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370

Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics

CurrentWork

ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)

Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities

Acknowledgements

Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories

References

AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007

AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8

Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008

BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005

BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005

Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991

ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17

CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985

Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34

CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum

Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)

FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991

Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995

Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009

GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia

Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13

HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)

HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07

JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863

Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009

JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994

Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92

KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999

Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity

LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007

Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180

Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey

MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey

MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691

Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340

Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003

Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81

Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36

PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004

PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005

Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21

RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007

RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318

RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262

RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006

SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924

SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress

SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum

StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132

TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)

ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety

Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623

Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA

WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370

Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics

Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991

ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17

CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985

Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34

CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum

Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)

FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991

Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995

Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009

GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia

Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13

HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)

HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06

IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07

JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863

Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009

JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994

Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92

KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999

Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity

LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007

Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180

Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey

MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey

MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691

Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340

Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003

Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81

Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36

PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004

PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005

Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21

RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007

RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318

RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262

RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006

SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924

SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress

SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum

StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132

TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)

ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety

Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623

Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA

WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370

Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics

Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92

KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999

Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity

LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007

Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180

Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey

MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey

MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691

Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340

Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003

Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81

Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36

PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004

PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005

Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21

RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007

RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318

RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262

RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006

SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924

SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress

SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum

StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132

TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)

ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety

Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623

Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA

WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370

Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics

RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318

RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262

RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006

SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924

SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress

SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum

StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132

TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)

ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety

Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623

Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA

WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370

Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics