Post on 24-Jul-2020
InteractiveStoryAuthoringforKnowledge‐BasedSupportforInvestigativeAnalysisASecondSTABatMakingSenseofVASTData
AshokKGoelAvikSinharoySummerAdamsampAdityDokania
DesignampIntelligenceLaboratorySchoolofInteractiveComputingGeorgiaInstituteofTechnology
AtlantaGA30332USA
Abstract The sensemaking task in investigative analysis generates stories that connect entitiesandevents inan inputstreamofdata TheStabsystemrepresentscrimestoriesashierarchicalscriptswithgoalsandstatesItgeneratesmultiplestoriesasexplanatoryhypothesesforaninputdata stream containing interleaved sequences of events recognizes intent in a specific eventsequence and calculates confidence values for the generated hypotheses In this report wedescribeStab2anewinteractiveversionoftheknowledge‐basedStabsystemStab2containsastoryeditorthatenablesuserstoenterandeditcrimestoriesWeillustrateStab2withexamplesfromtheIEEEVASTcontestdatasets
Keywords Investigative Analysis Sense Making Story Understanding Plan RecognitionHierarchicalScriptsIntelligenceAnalysis
ThisisGeorgiaTechGVUTechnicalReportGIT‐GVU‐10‐02Foradditionalinformationpleasecontactthefirstauthoratgoelccgatechedu
Goel A K Sinharoy A Adams S amp Dokania A (2010) Interactive Story Authoring for Knowledge-Based Support for Investigative Analysis A Second STAB at Making Sense of VAST Data GVU Technical Report GIT-GVU-10-02
InteractiveStoryAuthoringforKnowledge‐BasedSupportforInvestigativeAnalysisASecondSTABatMakingSenseofVASTData
AshokKGoelAvikSinharoySummerAdamsampAdityDokania
Section1Introduction
Intelligence analysis investigative analysis and other related forms of information analysis share manycommoncharacteristicsandcomponentsOneunifyingelementinthevarioustypesofinformationanalysisisthetaskofsensemaking(Bodnar2005Heuer1999KleinMoonampHoffman2006Krizan1999PirolliampCard2005ThomasampCook2005)generationofamodelofasituationthatconnectsentitiesandeventsinaninputstreamofdataaboutthesituation(sometimescolloquiallycalledtheldquoconnectthedotsrdquoproblem)Theinputtothe sensemaking task in different types of information analysis is characterized by the same kinds ofcharacteristics the amount of data in the input stream is huge data comes frommultiple sources and inmultipleformsdatafromvarioussourcesmaybeunreliableandconflictingdataarrivesincrementallyandisconstantlyevolvingdatamaypertaintomultipleactorswheretheactionsofthevariousactorsneednotbecoordinatedtheactorsmaytrytohidedataabouttheiractionsandmayevenintroducespuriousdatatohidetheir actions datamay pertain to novel actors aswell as rare or novel actions and the amount of usefulevidence typically is a small fraction of the vast amount of data (the colloquial ldquoneedle in the haystackrdquoproblem)Thedesiredoutputof the sensemaking task indifferent typesof informationanalysis toohas thesamekindsofcharacteristicsmodelsthatexplaintheconnectionsamongtheentitiesandeventsthatspecifythe intentofthevariousactorsthatmakeverifiablepredictionsandthathaveconfidencevaluesassociatedwiththemPsychological studiesof sensemaking in intelligenceanalysis (Heuer1999) indicate thatcognitive limitationsandbiasesofhumananalystsresultinseveralkindsoferrorsThethreemainerrorsmadebyhumananalystsinhypothesisgenerationare(Heuer1999)(1)Duetolimitationsofhumanmemoryanalystsmayhavedifficultykeeping track of multiple explanatory hypotheses for a set of data over a long period of time (2) Due tocognitivefixationanalystsmayquicklydecideonasinglehypothesisforthedatasetandsticktoitevenasnewdata arrives (cognitive fixation) (3) Due to confirmation bias analystsmay look for data that supports thehypothesis onwhich they are fixated and not necessarily the data thatmay refute the hypothesis Thus ascientific and technological challenge for cognitive science artificial intelligence and human‐centeredcomputing is to develop interactive computational tools that can help human analysts overcome thesecognitivelimitations
Overthelastfewyearsseveralresearchershaveexploredtheuseofknowledge‐basedtechniquestosupporthumandecision‐makinginintelligenceandinvestigativeanalysis(Birnbaumetal2005Chenatal2003JarvisLuntampMyers2004MurdockAhaampBreslow2003PARC2004Sanfilippoetal2007Tecucietal2008Weltyetal2005Whitakeretal2004)Inourpreviousworkwehavedevelopedanautomatedknowledge‐basedsystemcalledStab(AdamsampGoel2007a2007b2008GoelAdamsCutshawampSugandh2009)forIEEEVAST2006and2007datasets(IEEEVAST2006IEEEVAST2007Plaisantetal2008)BrieflyStabcontainsalibraryofhierarchicallyorganizedabstractstoriesthatcapturepatternsofcriminalactivityintheIEEEVAST2006and2007datasetsThusinStabthemodelsthatconnecttheentitiesandeventsintheinputdataarecrimestoriesThe crime stories are represented in the TMKL knowledge representation language (MurdockampGoel 20032008) Stab takesas inputa setofeventsextracted from the IEEEVAST2006and2007datasets It givesasoutput a set of multiple competing instantiated stories as explanatory hypotheses that connect the inputeventsandascribegoalstoactorsalongwithconfidencevaluesforeachhypothesisaswellasasummaryofevidenceforeachhypothesisStabrsquosuserinterfaceallowstheusertoviewthelogicalstructureofagenerated
hypothesis inagraphicalrepresentation Italsoenablesselectionofportionsoftheevidence insupportofageneratedhypothesis for further analysisof theexplanatory relationshipbetween thegeneratedhypothesisandthesupportingevidenceAhumananalystmayuseStabasanexternalmemoryofdataandexplanatoryhypothesesInadditiontheanalystmayuseStabrsquosresultstoinformadditionalsearchforfurtherevidencefororagainstageneratedhypothesis
OfcourseStabalsohasseverallimitationsOneofthemajorlimitationsofStabisthatcrimestoriesinitsstorylibraryhavetobehandcoded inTMKLThismakes itdifficult forausertoenternewstoriesoreditexistingstoriesinStabrsquoslibraryToenableausertomoreflexiblyinteractwithStabwehavedevelopedStab2anewinteractiveversionofStabStab2containsastoryeditorthatenablesuserstoenterandeditcrimestoriesinagraphical notation Stab2 automatically converts a new (or modified) story into its internal knowledgerepresentation language(TMKL)Thisenablesusers to interactwithStabrsquosstoriesldquoonlinerdquo InthispaperwefirstbrieflydescribethebasicsofStabandthenpresentStab2rsquosstoryeditor
Section2Knowledge‐BasedSupportforSensemaking
InthissectionwebrieflydescribetheoriginalStabsystemforsensemakingoftheIEEEVASTcontestdatasetsAlthoughStabrsquosreasoningknowledgeandrepresentationsareintendedtobegeneralStabrsquoslibraryatpresentcontainsonlycrimestoriesrelevanttotheVAST‐2006andVAST‐2007datasets
VAST Datasets The VAST datasets are synthetic datasets generated by the Pacific Northwest NationalLaboratories forsupportingresearchanddevelopment intheemergingfieldofvisualanalytics (ThompsonampCook 2005) The datasets pertain to fictitious illegal and unethical activities as well as normal and typicalactivities in a fictitious town in the United States The VAST‐2006 dataset contains over a thousand newsstorieswritteninEnglishandascoreoftablesmapsandphotographsFigure1 illustratesanexamplenewsstoryfromtheVAST2006datasetTheVAST‐2007dataset isa little largerandslightlymorecomplicatedbutsimilarinnature
Inputs to StabWemanually screened the dataset for stories that indicated an illegal or unethical activitywhich leftaboutahundrednewsstoriesoutofthemorethanathousandoriginally inthedatasetWethenmanuallyextractedeventsandentitiespertainingtoillegalunethicalactivitiesTheseeventsentitiesformtheinput to STAB We also hand crafted representations for each event in terms of the knowledge states itproducesInadditionweexaminedthemapsphotosandtablesthatarepartoftheVASTdatasetandsimilarlyextracted and represented the relevant information about various entities Table 1 illustrates a sample ofinputstoSTABalongwiththeresultingknowledgestatecreatedbyaninputevent
Figure1ExamplenewsstoryfromtheVAST‐2006dataset
(Adaptedfrom(httpwwwcsumdeduhcilVASTcontest06)Table1AsampleofinputstoStab
SampleSTABInputs ResultingState
stolen(money$40Highway‐Tire‐Store) Has‐object
cured‐disease(Boynton‐LabsPhilip‐Boyntonprion‐disease) Is‐rich‐and‐famous
named‐after(labPhilip‐BoyntonDean‐USC) Expert‐involved
was‐founded(Boynton‐Labs) Is‐open
have‐developed(Boynton‐Labsprion‐disease) Exists‐new‐disease
announced‐investigation(USFDABoynton‐Labs) Is‐investigating
Injected‐cow(Boynton‐Labsprion‐disease) Cow‐is‐infected
treatment‐cow(Boynton‐Labsprion‐disease) Cow‐is‐cured
Stabrsquos Library of Crime Stories STAB contains a small library of hierarchically organized abstract stories orhierarchicalscriptsthatcapturethepatternsofcrimesthatoccurintheVAST‐2006andVAST‐2007datasetsFigure2illustratesasimplescriptinSTABrsquoslibraryRobaStoreThisscripthasthegoalofachievingthestateofHaveMoneygiventheinitialstateofNotHaveMoney(topoffigure)Thisgoal(ortask)accordingtotheRobaStorescriptisachievedbyaplan(ormethod)thathasseveralactions(orevents)initGotoStoreBreakintoStore TakeMoney (middle of the figure) Each of these events becomes a task at the next lower level ofabstraction in the hierarchical script and each task can be (potentially) achieved bymultiplemethods ForexampleaccordingtotheRobaStorescriptthetaskofBreakintoStorecanbeachievedbyEnteringthroughaWindow or Entering through a Door (bottom of figure) Each of thesemethods in turn specifies a processconsistingofmultipleeventsandsoon
STABrsquos hierarchical scripts explicitly represent both goal and state at multiple levels of abstraction Whilerepresentation of the state caused by an event is useful for inferring causality representation of goals ofsequences of events is useful for inferring intention Figure 3 illustrates a more complex script of politicalconspiracyinwhichapoliticalfiguremaygetanopponentoutofanelectoralraceeitherbyexposingdirtonhim(politicalblackmail)orhavinghimassassinatedNotethisscriptiscomposedofseveralsmallerscripts
We found that sevenhierarchical scripts appear to cover all the illegalunethical activities in theVAST‐2006datasetandthatanotherfourhierarchicalscriptsapparentlyaresufficienttocovertheVAST‐2007aswellWehandcraftedthislibraryofscriptsintoSTAB
Figure2ThecontentandstructureofastoryinSTAB1(AdaptedfromGoeletal2009)
(
Stabrsquos Knowledge Representation STABrsquos scripts are represented in the TMKL language A task in TMKL isdefinedbyinputknowledgeelementsoutputknowledgeelementsrequiredinputconditions(pre‐conditions)desiredoutputconditions (post‐conditions)andmethods for implementing the taskAmethod isdefined intermsofsubtasksofthetaskandorderingofthesubtasksandisrepresentedbyafinitestatemachineFinallyknowledge in TMKL is defined in terms of domain concepts and relationships among them the input andoutputknowledgeelements ina taskspecificationrefer tothesedomainconceptsandrelationsMurdockampGoel2008providesdetailsofTMKL
Figure3ThestoryforapoliticalconspiracyintendedtoremoveanopponentfromanelectoralraceActivatednodesaredenotedbyathickoutlinearoundyellowboxes(AdaptedfromGoelet
al2009)
Figure4High‐LevelArchitectureofSTAB(AdaptedfromGoeletal2009)
t ba SV Ts t t SA D t tsD u sDr a H snnr boao b o Tb l ampcr oo sl I n a eso or a u saboC oe ys I l nn ol a l nn ob oe sI cr o y I ob sI I ys I sn sI eal I l nl Ds n l a au SoC oe ol ah o e a og b l I sI c r o y I o o osamp I r b b sob a br nosI D gI l n D boo l oe y I o soe oe ob g I l b sI oe 13 a ca b I oo sl I b l oe b ascob bol a sI oe ol ah 13 s aa hu e ol ah o e a oD b oe amp o esI D ob gb I c b b boe amp o esI D b ascob ol l agsI D amp l ahu esb o DDsI D l amp o esI D ob gb sI o sI boI oso b oe ob gu e ol ah o e a oe I sI bc ob oe I So sI cr o y I o sI oe ys I sn I a c o b oe l y cal bbu oe I sI cr o y I o a br nob sI oe a oas yn l I b ascoC oe I oe b asco sb bsampsna nh bol a sI oe l agsI D amp l ahu oe I nh a oas y b asco sb n a h sI oe l agsI D amp l ahC oe I sosl I n ob g I l b oeo amp o e oe I sI cr o a n bl oD D r o l I nh l I b asco sI boI sb g cou
t ba s tTs tau o l r ocr ob l r a gsI b l sI l aamp o sl I p n n sI boI oso b ascob sI oe l agsI D amp amp l ahC n n I l bsI e br e b asco oeo a amp o e soe I sI cr o y I oC n n ys I l a e br e b ascoC I l I s I yn r l a e br e b ascou e h nnl l S b sI sDr a H snnr boao oe ob gb sI oe c l nsos n l I bcsa h b asco oeo a amp o e soe I sI cr o y I o sI l I ar I I sI D l o u e l I s I yn r l a b asco sb b l I oe I r amp a l ob gb sI b asco oeo a amp o e soe sI cr o y I obu l n o n u dPVV cal ys os nb l o Tba b l I sI DC gI l n D C a ca b I oo sl I b I l r ocr obu
c t n py nc r c v ct ro c n n c s
e sI o a osy o P bhbo amp sb I r Damp I oo sl I l oe l asDsI n o bhbo amp soe casamp a h l r b l I sampcal ysI Dr b a sI o a osl I u e bc ss Dl n b l o P I eI amp I ob sI nr p ds nnl l I nsI r oel asI D I sosI D l o Tb asamp bol as b sI ysbr n l aampC I dss I n r b a ol l I oal n n n b c ob l o Tb a b l I sI D oeal r De Dac es n sI o a u
i r Ty t I ei c c t n r c ci r t c c sua ct ro i c t r n t t
o s h
sDr a O snnr boao b oe l ampcr oo sl I n a eso or a l o PTb bol ah r oel asI D ol l nu e ol ah sol a sb r b sI a o sI D I amp I D sI D oe o PTb asamp bol as bu e b bol as b a sa o b I I o l oe es aa es n b ascob r b sI o u l y aC esn o PTb bol as b bosnn r b 13 b oe gI l n D a ca b I oo sl I nI Dr D C e I D oe sI o aI n o amp l n sI o P ol I n r ol amp o s oaI bno sl I l bol ah r oel a h r b a sI ol oe 13 gI l n D a ca b I oo sl I u
e I l n D sol a ol l sb r sno sI ol o PTb sI o a C I eI n b r b a bb ol oe gI l n D n amp I ob sI oe 13 a ca b I oo sl I l bol ahu o cal ys b oe r b a l I oal n l y a sI sI D oe I osos b sI bol ah b nn b oe a no sl I b amp l I D oe I osos bu e b l I a sI sDr a O a ab ol o b as sI oe ca ysl r b b osl I u
c t n pAsy ai onc S t ct ro i c t r n c s
sDr a ( snnr boao b oe ysbr n bhI oS r b h o PTb bol ah sol au e ob g sb a ca b I o b a oI Dn u b oe r b a amp l r b b l y a ob gC oe ol ah sol a sbcnh b oe 13 bc ss o sl I l oe ob gu e amp oel sb a ca b I o b b L r I l br ob gb oeo bc ss b oe l a asI D l oe br ob gb sI nr sI D so a osl I u ebr ob g sI or aI sb bc ss b a oI Dn soe sob l I amp oel b b oe I So nl a n y n l boa osl I u
i r My ai onc S t c ct ro ct r n c s
o o 13 h
sDr a B snnr boao b o PTb ol ah sol a sI o a u e amp s I c a o l oe sDr a l I os I b oe bol ah sI D so C sI oesb b oe l ol a b asco snnr boao sI sDr a Pu e c n oo l I oe ol c asDeo l sDr a B cal ys b oe s l I b l a sI D I ob g I amp oel n amp I ob ol oe bol ahu e sI l l I oe l ool amp n o c s ob oe I l n D sol au e sI l l I oe l ool amp asDeo sI s o b l oe a bol as b sI oe ol ah 13 s aa h I I n b oe r b a ol al b oe ns aa hu e Dac e sI oe l ool amp amps n I n b oe r b a ol I y sDo oe b asco l bol ahu
irkycs
uactroctrncr
TheStoryEditor inStab2gives theusermorecontrol thanwaspossible inStab In theoriginalStabsystemhierarchical scripts were hard‐coded in TMKL that itself is coded in the Lisp programming knowledge Thisresulted inseveraldifficulties regarding theusabilityof thesystemFirstly thestoredscripts inStabarenotavailableforreferenceorreviewexceptbyperusingtheactualLispcodeThislackoftransparencycanleadtoinaccurateandimpreciseunderstandingofthestoriesthataregeneratedbyStabasexplainingtheinputdataStab2rsquosStoryEditorprovidesvisualaccesstoboththelibraryofstoredscriptsaswellasthestoriesinstantiatedwhenStab2isrunforaninputdataset
SecondlysinceStabrsquosstoredscriptsarecodedinTMKLitisverydifficultforausertoauthoranewscriptorstoryevaluatethecorrectnessofastoredscriptandmakemodificationstoitStab2StoryEditorwithitsvisualinterface makes these tasks much easier though editing the knowledge elements in a story through theKnowledgeEditorstillisnon‐trivial
FinallytheinitialversionStabdidnotallowmodificationstothestructureofexistingscriptsatruntimeThisimpliesthat it isusefultorunStabjustonceonaninputdatasetHowever ingeneralahumananalystmaywanttorunStabseveraltimesndasheachwithdifferingvariationsofthescripts‐astheanalystrsquosinsightintothecriminal patterns evolves This is perhaps themost important side effect of the use of an interactive StoryEditor in Stab2 Stab2 enables an analyst to conduct ldquowhat ifrdquo simulations Using Stab2rsquos Story Editor andKnowledge Editor illustrated in Figure 7 an analystmay inspectmodify activate or deactivate the scriptsstored in Stab2rsquos Story Library the task and method elements in an individual script or the knowledgeelements likedomainconceptsand relationsTheanalystmay then runStab2 to simulate theeffectsof thepreciseconditionsshecreatedinStab2rsquosvirtualworld
Section34StoryAuthoringinStab2
Whenananalystwants tocreateanewstoryheneeds toopentheeditorandclickonaddstory iconThisopensawhiteareawithapaletteontherightWestartwithpickingtheldquoMainstoryldquoiconfromthepaletteanddropitintothewhiteboardThisrepresentsthenameofthestoryandisdepictedinFigure8(a)AfterthiswedragthebluerectanglethatreadsldquoIntermediaryGoalsrdquoandconnectthistothemainstoryusingaSub‐Goalconnector all ofwhich can be found in the palette The result is shown in Figure 8(b) Once themethod iscreatedwefillitinwithactionsperformedtoachievethatgoalThisisdoneusingtheldquoGoalsActionrdquoiconinthepaletteandthisisshowninFigure8(c)ThedraganddropcapabilitymakesitveryeasyforananalysttoconstructanewstoryplotThetreecangodownasmanylevelsdependingonthecomplexityofthestoryTherelationshipsbetweeneachofthegoalsarecapturedusingthelinksTwokindsoflinkagesarepossiblendashSub‐Goals that suggests that one goal is a child of another and Causal Connection that indicates relationshipsbetweendifferentactionsinsideeachintermediarygoal
Figure8(a)1
stSna
psho
tofStoryAutho
ringinStab2
ndashHerethehu
mananalystdragsth
eicon
mainstoryfrom
thepalette
andplacesitinth
eed
itorareaand
nam
esitasldquostoryplotrdquo
Figure8(b)A
2ndSna
psho
tofStoryAutho
ringin
Stab2
ndashHerethehu
mananalystdragsth
eInterm
ediaryGoalsicon
and
placesitbe
lowth
etopleveltaskAlsotheanalystdragsthesub‐goalslinkagefromthe
paletteand
con
nectsthetw
oblocksand
nam
esthe
bluerectangleas
ldquoMetho
d1rdquo
Figure8(c)ASna
psho
tofStoryAutho
ringinStab2
‐Heretheanalystd
ragsth
eGoalsA
ctionsicon
inside
thealreadycreatedmetho
dHe
then
con
nectstheseactio
nswith
thecausalcon
nectionslinkageinth
epaletteandnameseachactio
n
BehindtheScenes
Figure9(a)A
Sna
psho
tofaXMLfileforeachStoryPlot
Figure9(b)A
Sna
psho
tofaXMLfileforeachStoryPlotndash
Thisim
agehighlightsindividu
alta
gsThe
imageon
the
righth
ighlightsthechildTaskta
gandtheon
ethelefthighlightstheallM
etho
dsta
g
EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)
RelatedWork
ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors
StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)
Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo
Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)
CurrentWork
ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)
Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities
Acknowledgements
Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories
References
AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007
AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8
Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008
BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005
BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005
Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991
ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17
CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985
Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34
CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum
Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)
FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991
Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995
Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009
GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia
Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13
HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)
HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07
JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863
Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009
JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994
Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92
KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999
Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity
LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007
Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180
Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey
MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey
MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691
Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340
Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003
Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81
Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36
PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004
PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005
Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21
RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007
RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318
RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262
RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006
SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924
SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress
SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum
StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132
TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)
ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety
Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623
Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA
WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370
Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics
InteractiveStoryAuthoringforKnowledge‐BasedSupportforInvestigativeAnalysisASecondSTABatMakingSenseofVASTData
AshokKGoelAvikSinharoySummerAdamsampAdityDokania
Section1Introduction
Intelligence analysis investigative analysis and other related forms of information analysis share manycommoncharacteristicsandcomponentsOneunifyingelementinthevarioustypesofinformationanalysisisthetaskofsensemaking(Bodnar2005Heuer1999KleinMoonampHoffman2006Krizan1999PirolliampCard2005ThomasampCook2005)generationofamodelofasituationthatconnectsentitiesandeventsinaninputstreamofdataaboutthesituation(sometimescolloquiallycalledtheldquoconnectthedotsrdquoproblem)Theinputtothe sensemaking task in different types of information analysis is characterized by the same kinds ofcharacteristics the amount of data in the input stream is huge data comes frommultiple sources and inmultipleformsdatafromvarioussourcesmaybeunreliableandconflictingdataarrivesincrementallyandisconstantlyevolvingdatamaypertaintomultipleactorswheretheactionsofthevariousactorsneednotbecoordinatedtheactorsmaytrytohidedataabouttheiractionsandmayevenintroducespuriousdatatohidetheir actions datamay pertain to novel actors aswell as rare or novel actions and the amount of usefulevidence typically is a small fraction of the vast amount of data (the colloquial ldquoneedle in the haystackrdquoproblem)Thedesiredoutputof the sensemaking task indifferent typesof informationanalysis toohas thesamekindsofcharacteristicsmodelsthatexplaintheconnectionsamongtheentitiesandeventsthatspecifythe intentofthevariousactorsthatmakeverifiablepredictionsandthathaveconfidencevaluesassociatedwiththemPsychological studiesof sensemaking in intelligenceanalysis (Heuer1999) indicate thatcognitive limitationsandbiasesofhumananalystsresultinseveralkindsoferrorsThethreemainerrorsmadebyhumananalystsinhypothesisgenerationare(Heuer1999)(1)Duetolimitationsofhumanmemoryanalystsmayhavedifficultykeeping track of multiple explanatory hypotheses for a set of data over a long period of time (2) Due tocognitivefixationanalystsmayquicklydecideonasinglehypothesisforthedatasetandsticktoitevenasnewdata arrives (cognitive fixation) (3) Due to confirmation bias analystsmay look for data that supports thehypothesis onwhich they are fixated and not necessarily the data thatmay refute the hypothesis Thus ascientific and technological challenge for cognitive science artificial intelligence and human‐centeredcomputing is to develop interactive computational tools that can help human analysts overcome thesecognitivelimitations
Overthelastfewyearsseveralresearchershaveexploredtheuseofknowledge‐basedtechniquestosupporthumandecision‐makinginintelligenceandinvestigativeanalysis(Birnbaumetal2005Chenatal2003JarvisLuntampMyers2004MurdockAhaampBreslow2003PARC2004Sanfilippoetal2007Tecucietal2008Weltyetal2005Whitakeretal2004)Inourpreviousworkwehavedevelopedanautomatedknowledge‐basedsystemcalledStab(AdamsampGoel2007a2007b2008GoelAdamsCutshawampSugandh2009)forIEEEVAST2006and2007datasets(IEEEVAST2006IEEEVAST2007Plaisantetal2008)BrieflyStabcontainsalibraryofhierarchicallyorganizedabstractstoriesthatcapturepatternsofcriminalactivityintheIEEEVAST2006and2007datasetsThusinStabthemodelsthatconnecttheentitiesandeventsintheinputdataarecrimestoriesThe crime stories are represented in the TMKL knowledge representation language (MurdockampGoel 20032008) Stab takesas inputa setofeventsextracted from the IEEEVAST2006and2007datasets It givesasoutput a set of multiple competing instantiated stories as explanatory hypotheses that connect the inputeventsandascribegoalstoactorsalongwithconfidencevaluesforeachhypothesisaswellasasummaryofevidenceforeachhypothesisStabrsquosuserinterfaceallowstheusertoviewthelogicalstructureofagenerated
hypothesis inagraphicalrepresentation Italsoenablesselectionofportionsoftheevidence insupportofageneratedhypothesis for further analysisof theexplanatory relationshipbetween thegeneratedhypothesisandthesupportingevidenceAhumananalystmayuseStabasanexternalmemoryofdataandexplanatoryhypothesesInadditiontheanalystmayuseStabrsquosresultstoinformadditionalsearchforfurtherevidencefororagainstageneratedhypothesis
OfcourseStabalsohasseverallimitationsOneofthemajorlimitationsofStabisthatcrimestoriesinitsstorylibraryhavetobehandcoded inTMKLThismakes itdifficult forausertoenternewstoriesoreditexistingstoriesinStabrsquoslibraryToenableausertomoreflexiblyinteractwithStabwehavedevelopedStab2anewinteractiveversionofStabStab2containsastoryeditorthatenablesuserstoenterandeditcrimestoriesinagraphical notation Stab2 automatically converts a new (or modified) story into its internal knowledgerepresentation language(TMKL)Thisenablesusers to interactwithStabrsquosstoriesldquoonlinerdquo InthispaperwefirstbrieflydescribethebasicsofStabandthenpresentStab2rsquosstoryeditor
Section2Knowledge‐BasedSupportforSensemaking
InthissectionwebrieflydescribetheoriginalStabsystemforsensemakingoftheIEEEVASTcontestdatasetsAlthoughStabrsquosreasoningknowledgeandrepresentationsareintendedtobegeneralStabrsquoslibraryatpresentcontainsonlycrimestoriesrelevanttotheVAST‐2006andVAST‐2007datasets
VAST Datasets The VAST datasets are synthetic datasets generated by the Pacific Northwest NationalLaboratories forsupportingresearchanddevelopment intheemergingfieldofvisualanalytics (ThompsonampCook 2005) The datasets pertain to fictitious illegal and unethical activities as well as normal and typicalactivities in a fictitious town in the United States The VAST‐2006 dataset contains over a thousand newsstorieswritteninEnglishandascoreoftablesmapsandphotographsFigure1 illustratesanexamplenewsstoryfromtheVAST2006datasetTheVAST‐2007dataset isa little largerandslightlymorecomplicatedbutsimilarinnature
Inputs to StabWemanually screened the dataset for stories that indicated an illegal or unethical activitywhich leftaboutahundrednewsstoriesoutofthemorethanathousandoriginally inthedatasetWethenmanuallyextractedeventsandentitiespertainingtoillegalunethicalactivitiesTheseeventsentitiesformtheinput to STAB We also hand crafted representations for each event in terms of the knowledge states itproducesInadditionweexaminedthemapsphotosandtablesthatarepartoftheVASTdatasetandsimilarlyextracted and represented the relevant information about various entities Table 1 illustrates a sample ofinputstoSTABalongwiththeresultingknowledgestatecreatedbyaninputevent
Figure1ExamplenewsstoryfromtheVAST‐2006dataset
(Adaptedfrom(httpwwwcsumdeduhcilVASTcontest06)Table1AsampleofinputstoStab
SampleSTABInputs ResultingState
stolen(money$40Highway‐Tire‐Store) Has‐object
cured‐disease(Boynton‐LabsPhilip‐Boyntonprion‐disease) Is‐rich‐and‐famous
named‐after(labPhilip‐BoyntonDean‐USC) Expert‐involved
was‐founded(Boynton‐Labs) Is‐open
have‐developed(Boynton‐Labsprion‐disease) Exists‐new‐disease
announced‐investigation(USFDABoynton‐Labs) Is‐investigating
Injected‐cow(Boynton‐Labsprion‐disease) Cow‐is‐infected
treatment‐cow(Boynton‐Labsprion‐disease) Cow‐is‐cured
Stabrsquos Library of Crime Stories STAB contains a small library of hierarchically organized abstract stories orhierarchicalscriptsthatcapturethepatternsofcrimesthatoccurintheVAST‐2006andVAST‐2007datasetsFigure2illustratesasimplescriptinSTABrsquoslibraryRobaStoreThisscripthasthegoalofachievingthestateofHaveMoneygiventheinitialstateofNotHaveMoney(topoffigure)Thisgoal(ortask)accordingtotheRobaStorescriptisachievedbyaplan(ormethod)thathasseveralactions(orevents)initGotoStoreBreakintoStore TakeMoney (middle of the figure) Each of these events becomes a task at the next lower level ofabstraction in the hierarchical script and each task can be (potentially) achieved bymultiplemethods ForexampleaccordingtotheRobaStorescriptthetaskofBreakintoStorecanbeachievedbyEnteringthroughaWindow or Entering through a Door (bottom of figure) Each of thesemethods in turn specifies a processconsistingofmultipleeventsandsoon
STABrsquos hierarchical scripts explicitly represent both goal and state at multiple levels of abstraction Whilerepresentation of the state caused by an event is useful for inferring causality representation of goals ofsequences of events is useful for inferring intention Figure 3 illustrates a more complex script of politicalconspiracyinwhichapoliticalfiguremaygetanopponentoutofanelectoralraceeitherbyexposingdirtonhim(politicalblackmail)orhavinghimassassinatedNotethisscriptiscomposedofseveralsmallerscripts
We found that sevenhierarchical scripts appear to cover all the illegalunethical activities in theVAST‐2006datasetandthatanotherfourhierarchicalscriptsapparentlyaresufficienttocovertheVAST‐2007aswellWehandcraftedthislibraryofscriptsintoSTAB
Figure2ThecontentandstructureofastoryinSTAB1(AdaptedfromGoeletal2009)
(
Stabrsquos Knowledge Representation STABrsquos scripts are represented in the TMKL language A task in TMKL isdefinedbyinputknowledgeelementsoutputknowledgeelementsrequiredinputconditions(pre‐conditions)desiredoutputconditions (post‐conditions)andmethods for implementing the taskAmethod isdefined intermsofsubtasksofthetaskandorderingofthesubtasksandisrepresentedbyafinitestatemachineFinallyknowledge in TMKL is defined in terms of domain concepts and relationships among them the input andoutputknowledgeelements ina taskspecificationrefer tothesedomainconceptsandrelationsMurdockampGoel2008providesdetailsofTMKL
Figure3ThestoryforapoliticalconspiracyintendedtoremoveanopponentfromanelectoralraceActivatednodesaredenotedbyathickoutlinearoundyellowboxes(AdaptedfromGoelet
al2009)
Figure4High‐LevelArchitectureofSTAB(AdaptedfromGoeletal2009)
t ba SV Ts t t SA D t tsD u sDr a H snnr boao b o Tb l ampcr oo sl I n a eso or a u saboC oe ys I l nn ol a l nn ob oe sI cr o y I ob sI I ys I sn sI eal I l nl Ds n l a au SoC oe ol ah o e a og b l I sI c r o y I o o osamp I r b b sob a br nosI D gI l n D boo l oe y I o soe oe ob g I l b sI oe 13 a ca b I oo sl I b l oe b ascob bol a sI oe ol ah 13 s aa hu e ol ah o e a oD b oe amp o esI D ob gb I c b b boe amp o esI D b ascob ol l agsI D amp l ahu esb o DDsI D l amp o esI D ob gb sI o sI boI oso b oe ob gu e ol ah o e a oe I sI bc ob oe I So sI cr o y I o sI oe ys I sn I a c o b oe l y cal bbu oe I sI cr o y I o a br nob sI oe a oas yn l I b ascoC oe I oe b asco sb bsampsna nh bol a sI oe l agsI D amp l ahu oe I nh a oas y b asco sb n a h sI oe l agsI D amp l ahC oe I sosl I n ob g I l b oeo amp o e oe I sI cr o a n bl oD D r o l I nh l I b asco sI boI sb g cou
t ba s tTs tau o l r ocr ob l r a gsI b l sI l aamp o sl I p n n sI boI oso b ascob sI oe l agsI D amp amp l ahC n n I l bsI e br e b asco oeo a amp o e soe I sI cr o y I oC n n ys I l a e br e b ascoC I l I s I yn r l a e br e b ascou e h nnl l S b sI sDr a H snnr boao oe ob gb sI oe c l nsos n l I bcsa h b asco oeo a amp o e soe I sI cr o y I o sI l I ar I I sI D l o u e l I s I yn r l a b asco sb b l I oe I r amp a l ob gb sI b asco oeo a amp o e soe sI cr o y I obu l n o n u dPVV cal ys os nb l o Tba b l I sI DC gI l n D C a ca b I oo sl I b I l r ocr obu
c t n py nc r c v ct ro c n n c s
e sI o a osy o P bhbo amp sb I r Damp I oo sl I l oe l asDsI n o bhbo amp soe casamp a h l r b l I sampcal ysI Dr b a sI o a osl I u e bc ss Dl n b l o P I eI amp I ob sI nr p ds nnl l I nsI r oel asI D I sosI D l o Tb asamp bol as b sI ysbr n l aampC I dss I n r b a ol l I oal n n n b c ob l o Tb a b l I sI D oeal r De Dac es n sI o a u
i r Ty t I ei c c t n r c ci r t c c sua ct ro i c t r n t t
o s h
sDr a O snnr boao b oe l ampcr oo sl I n a eso or a l o PTb bol ah r oel asI D ol l nu e ol ah sol a sb r b sI a o sI D I amp I D sI D oe o PTb asamp bol as bu e b bol as b a sa o b I I o l oe es aa es n b ascob r b sI o u l y aC esn o PTb bol as b bosnn r b 13 b oe gI l n D a ca b I oo sl I nI Dr D C e I D oe sI o aI n o amp l n sI o P ol I n r ol amp o s oaI bno sl I l bol ah r oel a h r b a sI ol oe 13 gI l n D a ca b I oo sl I u
e I l n D sol a ol l sb r sno sI ol o PTb sI o a C I eI n b r b a bb ol oe gI l n D n amp I ob sI oe 13 a ca b I oo sl I l bol ahu o cal ys b oe r b a l I oal n l y a sI sI D oe I osos b sI bol ah b nn b oe a no sl I b amp l I D oe I osos bu e b l I a sI sDr a O a ab ol o b as sI oe ca ysl r b b osl I u
c t n pAsy ai onc S t ct ro i c t r n c s
sDr a ( snnr boao b oe ysbr n bhI oS r b h o PTb bol ah sol au e ob g sb a ca b I o b a oI Dn u b oe r b a amp l r b b l y a ob gC oe ol ah sol a sbcnh b oe 13 bc ss o sl I l oe ob gu e amp oel sb a ca b I o b b L r I l br ob gb oeo bc ss b oe l a asI D l oe br ob gb sI nr sI D so a osl I u ebr ob g sI or aI sb bc ss b a oI Dn soe sob l I amp oel b b oe I So nl a n y n l boa osl I u
i r My ai onc S t c ct ro ct r n c s
o o 13 h
sDr a B snnr boao b o PTb ol ah sol a sI o a u e amp s I c a o l oe sDr a l I os I b oe bol ah sI D so C sI oesb b oe l ol a b asco snnr boao sI sDr a Pu e c n oo l I oe ol c asDeo l sDr a B cal ys b oe s l I b l a sI D I ob g I amp oel n amp I ob ol oe bol ahu e sI l l I oe l ool amp n o c s ob oe I l n D sol au e sI l l I oe l ool amp asDeo sI s o b l oe a bol as b sI oe ol ah 13 s aa h I I n b oe r b a ol al b oe ns aa hu e Dac e sI oe l ool amp amps n I n b oe r b a ol I y sDo oe b asco l bol ahu
irkycs
uactroctrncr
TheStoryEditor inStab2gives theusermorecontrol thanwaspossible inStab In theoriginalStabsystemhierarchical scripts were hard‐coded in TMKL that itself is coded in the Lisp programming knowledge Thisresulted inseveraldifficulties regarding theusabilityof thesystemFirstly thestoredscripts inStabarenotavailableforreferenceorreviewexceptbyperusingtheactualLispcodeThislackoftransparencycanleadtoinaccurateandimpreciseunderstandingofthestoriesthataregeneratedbyStabasexplainingtheinputdataStab2rsquosStoryEditorprovidesvisualaccesstoboththelibraryofstoredscriptsaswellasthestoriesinstantiatedwhenStab2isrunforaninputdataset
SecondlysinceStabrsquosstoredscriptsarecodedinTMKLitisverydifficultforausertoauthoranewscriptorstoryevaluatethecorrectnessofastoredscriptandmakemodificationstoitStab2StoryEditorwithitsvisualinterface makes these tasks much easier though editing the knowledge elements in a story through theKnowledgeEditorstillisnon‐trivial
FinallytheinitialversionStabdidnotallowmodificationstothestructureofexistingscriptsatruntimeThisimpliesthat it isusefultorunStabjustonceonaninputdatasetHowever ingeneralahumananalystmaywanttorunStabseveraltimesndasheachwithdifferingvariationsofthescripts‐astheanalystrsquosinsightintothecriminal patterns evolves This is perhaps themost important side effect of the use of an interactive StoryEditor in Stab2 Stab2 enables an analyst to conduct ldquowhat ifrdquo simulations Using Stab2rsquos Story Editor andKnowledge Editor illustrated in Figure 7 an analystmay inspectmodify activate or deactivate the scriptsstored in Stab2rsquos Story Library the task and method elements in an individual script or the knowledgeelements likedomainconceptsand relationsTheanalystmay then runStab2 to simulate theeffectsof thepreciseconditionsshecreatedinStab2rsquosvirtualworld
Section34StoryAuthoringinStab2
Whenananalystwants tocreateanewstoryheneeds toopentheeditorandclickonaddstory iconThisopensawhiteareawithapaletteontherightWestartwithpickingtheldquoMainstoryldquoiconfromthepaletteanddropitintothewhiteboardThisrepresentsthenameofthestoryandisdepictedinFigure8(a)AfterthiswedragthebluerectanglethatreadsldquoIntermediaryGoalsrdquoandconnectthistothemainstoryusingaSub‐Goalconnector all ofwhich can be found in the palette The result is shown in Figure 8(b) Once themethod iscreatedwefillitinwithactionsperformedtoachievethatgoalThisisdoneusingtheldquoGoalsActionrdquoiconinthepaletteandthisisshowninFigure8(c)ThedraganddropcapabilitymakesitveryeasyforananalysttoconstructanewstoryplotThetreecangodownasmanylevelsdependingonthecomplexityofthestoryTherelationshipsbetweeneachofthegoalsarecapturedusingthelinksTwokindsoflinkagesarepossiblendashSub‐Goals that suggests that one goal is a child of another and Causal Connection that indicates relationshipsbetweendifferentactionsinsideeachintermediarygoal
Figure8(a)1
stSna
psho
tofStoryAutho
ringinStab2
ndashHerethehu
mananalystdragsth
eicon
mainstoryfrom
thepalette
andplacesitinth
eed
itorareaand
nam
esitasldquostoryplotrdquo
Figure8(b)A
2ndSna
psho
tofStoryAutho
ringin
Stab2
ndashHerethehu
mananalystdragsth
eInterm
ediaryGoalsicon
and
placesitbe
lowth
etopleveltaskAlsotheanalystdragsthesub‐goalslinkagefromthe
paletteand
con
nectsthetw
oblocksand
nam
esthe
bluerectangleas
ldquoMetho
d1rdquo
Figure8(c)ASna
psho
tofStoryAutho
ringinStab2
‐Heretheanalystd
ragsth
eGoalsA
ctionsicon
inside
thealreadycreatedmetho
dHe
then
con
nectstheseactio
nswith
thecausalcon
nectionslinkageinth
epaletteandnameseachactio
n
BehindtheScenes
Figure9(a)A
Sna
psho
tofaXMLfileforeachStoryPlot
Figure9(b)A
Sna
psho
tofaXMLfileforeachStoryPlotndash
Thisim
agehighlightsindividu
alta
gsThe
imageon
the
righth
ighlightsthechildTaskta
gandtheon
ethelefthighlightstheallM
etho
dsta
g
EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)
RelatedWork
ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors
StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)
Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo
Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)
CurrentWork
ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)
Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities
Acknowledgements
Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories
References
AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007
AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8
Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008
BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005
BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005
Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991
ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17
CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985
Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34
CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum
Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)
FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991
Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995
Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009
GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia
Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13
HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)
HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07
JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863
Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009
JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994
Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92
KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999
Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity
LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007
Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180
Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey
MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey
MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691
Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340
Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003
Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81
Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36
PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004
PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005
Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21
RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007
RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318
RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262
RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006
SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924
SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress
SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum
StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132
TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)
ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety
Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623
Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA
WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370
Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics
hypothesis inagraphicalrepresentation Italsoenablesselectionofportionsoftheevidence insupportofageneratedhypothesis for further analysisof theexplanatory relationshipbetween thegeneratedhypothesisandthesupportingevidenceAhumananalystmayuseStabasanexternalmemoryofdataandexplanatoryhypothesesInadditiontheanalystmayuseStabrsquosresultstoinformadditionalsearchforfurtherevidencefororagainstageneratedhypothesis
OfcourseStabalsohasseverallimitationsOneofthemajorlimitationsofStabisthatcrimestoriesinitsstorylibraryhavetobehandcoded inTMKLThismakes itdifficult forausertoenternewstoriesoreditexistingstoriesinStabrsquoslibraryToenableausertomoreflexiblyinteractwithStabwehavedevelopedStab2anewinteractiveversionofStabStab2containsastoryeditorthatenablesuserstoenterandeditcrimestoriesinagraphical notation Stab2 automatically converts a new (or modified) story into its internal knowledgerepresentation language(TMKL)Thisenablesusers to interactwithStabrsquosstoriesldquoonlinerdquo InthispaperwefirstbrieflydescribethebasicsofStabandthenpresentStab2rsquosstoryeditor
Section2Knowledge‐BasedSupportforSensemaking
InthissectionwebrieflydescribetheoriginalStabsystemforsensemakingoftheIEEEVASTcontestdatasetsAlthoughStabrsquosreasoningknowledgeandrepresentationsareintendedtobegeneralStabrsquoslibraryatpresentcontainsonlycrimestoriesrelevanttotheVAST‐2006andVAST‐2007datasets
VAST Datasets The VAST datasets are synthetic datasets generated by the Pacific Northwest NationalLaboratories forsupportingresearchanddevelopment intheemergingfieldofvisualanalytics (ThompsonampCook 2005) The datasets pertain to fictitious illegal and unethical activities as well as normal and typicalactivities in a fictitious town in the United States The VAST‐2006 dataset contains over a thousand newsstorieswritteninEnglishandascoreoftablesmapsandphotographsFigure1 illustratesanexamplenewsstoryfromtheVAST2006datasetTheVAST‐2007dataset isa little largerandslightlymorecomplicatedbutsimilarinnature
Inputs to StabWemanually screened the dataset for stories that indicated an illegal or unethical activitywhich leftaboutahundrednewsstoriesoutofthemorethanathousandoriginally inthedatasetWethenmanuallyextractedeventsandentitiespertainingtoillegalunethicalactivitiesTheseeventsentitiesformtheinput to STAB We also hand crafted representations for each event in terms of the knowledge states itproducesInadditionweexaminedthemapsphotosandtablesthatarepartoftheVASTdatasetandsimilarlyextracted and represented the relevant information about various entities Table 1 illustrates a sample ofinputstoSTABalongwiththeresultingknowledgestatecreatedbyaninputevent
Figure1ExamplenewsstoryfromtheVAST‐2006dataset
(Adaptedfrom(httpwwwcsumdeduhcilVASTcontest06)Table1AsampleofinputstoStab
SampleSTABInputs ResultingState
stolen(money$40Highway‐Tire‐Store) Has‐object
cured‐disease(Boynton‐LabsPhilip‐Boyntonprion‐disease) Is‐rich‐and‐famous
named‐after(labPhilip‐BoyntonDean‐USC) Expert‐involved
was‐founded(Boynton‐Labs) Is‐open
have‐developed(Boynton‐Labsprion‐disease) Exists‐new‐disease
announced‐investigation(USFDABoynton‐Labs) Is‐investigating
Injected‐cow(Boynton‐Labsprion‐disease) Cow‐is‐infected
treatment‐cow(Boynton‐Labsprion‐disease) Cow‐is‐cured
Stabrsquos Library of Crime Stories STAB contains a small library of hierarchically organized abstract stories orhierarchicalscriptsthatcapturethepatternsofcrimesthatoccurintheVAST‐2006andVAST‐2007datasetsFigure2illustratesasimplescriptinSTABrsquoslibraryRobaStoreThisscripthasthegoalofachievingthestateofHaveMoneygiventheinitialstateofNotHaveMoney(topoffigure)Thisgoal(ortask)accordingtotheRobaStorescriptisachievedbyaplan(ormethod)thathasseveralactions(orevents)initGotoStoreBreakintoStore TakeMoney (middle of the figure) Each of these events becomes a task at the next lower level ofabstraction in the hierarchical script and each task can be (potentially) achieved bymultiplemethods ForexampleaccordingtotheRobaStorescriptthetaskofBreakintoStorecanbeachievedbyEnteringthroughaWindow or Entering through a Door (bottom of figure) Each of thesemethods in turn specifies a processconsistingofmultipleeventsandsoon
STABrsquos hierarchical scripts explicitly represent both goal and state at multiple levels of abstraction Whilerepresentation of the state caused by an event is useful for inferring causality representation of goals ofsequences of events is useful for inferring intention Figure 3 illustrates a more complex script of politicalconspiracyinwhichapoliticalfiguremaygetanopponentoutofanelectoralraceeitherbyexposingdirtonhim(politicalblackmail)orhavinghimassassinatedNotethisscriptiscomposedofseveralsmallerscripts
We found that sevenhierarchical scripts appear to cover all the illegalunethical activities in theVAST‐2006datasetandthatanotherfourhierarchicalscriptsapparentlyaresufficienttocovertheVAST‐2007aswellWehandcraftedthislibraryofscriptsintoSTAB
Figure2ThecontentandstructureofastoryinSTAB1(AdaptedfromGoeletal2009)
(
Stabrsquos Knowledge Representation STABrsquos scripts are represented in the TMKL language A task in TMKL isdefinedbyinputknowledgeelementsoutputknowledgeelementsrequiredinputconditions(pre‐conditions)desiredoutputconditions (post‐conditions)andmethods for implementing the taskAmethod isdefined intermsofsubtasksofthetaskandorderingofthesubtasksandisrepresentedbyafinitestatemachineFinallyknowledge in TMKL is defined in terms of domain concepts and relationships among them the input andoutputknowledgeelements ina taskspecificationrefer tothesedomainconceptsandrelationsMurdockampGoel2008providesdetailsofTMKL
Figure3ThestoryforapoliticalconspiracyintendedtoremoveanopponentfromanelectoralraceActivatednodesaredenotedbyathickoutlinearoundyellowboxes(AdaptedfromGoelet
al2009)
Figure4High‐LevelArchitectureofSTAB(AdaptedfromGoeletal2009)
t ba SV Ts t t SA D t tsD u sDr a H snnr boao b o Tb l ampcr oo sl I n a eso or a u saboC oe ys I l nn ol a l nn ob oe sI cr o y I ob sI I ys I sn sI eal I l nl Ds n l a au SoC oe ol ah o e a og b l I sI c r o y I o o osamp I r b b sob a br nosI D gI l n D boo l oe y I o soe oe ob g I l b sI oe 13 a ca b I oo sl I b l oe b ascob bol a sI oe ol ah 13 s aa hu e ol ah o e a oD b oe amp o esI D ob gb I c b b boe amp o esI D b ascob ol l agsI D amp l ahu esb o DDsI D l amp o esI D ob gb sI o sI boI oso b oe ob gu e ol ah o e a oe I sI bc ob oe I So sI cr o y I o sI oe ys I sn I a c o b oe l y cal bbu oe I sI cr o y I o a br nob sI oe a oas yn l I b ascoC oe I oe b asco sb bsampsna nh bol a sI oe l agsI D amp l ahu oe I nh a oas y b asco sb n a h sI oe l agsI D amp l ahC oe I sosl I n ob g I l b oeo amp o e oe I sI cr o a n bl oD D r o l I nh l I b asco sI boI sb g cou
t ba s tTs tau o l r ocr ob l r a gsI b l sI l aamp o sl I p n n sI boI oso b ascob sI oe l agsI D amp amp l ahC n n I l bsI e br e b asco oeo a amp o e soe I sI cr o y I oC n n ys I l a e br e b ascoC I l I s I yn r l a e br e b ascou e h nnl l S b sI sDr a H snnr boao oe ob gb sI oe c l nsos n l I bcsa h b asco oeo a amp o e soe I sI cr o y I o sI l I ar I I sI D l o u e l I s I yn r l a b asco sb b l I oe I r amp a l ob gb sI b asco oeo a amp o e soe sI cr o y I obu l n o n u dPVV cal ys os nb l o Tba b l I sI DC gI l n D C a ca b I oo sl I b I l r ocr obu
c t n py nc r c v ct ro c n n c s
e sI o a osy o P bhbo amp sb I r Damp I oo sl I l oe l asDsI n o bhbo amp soe casamp a h l r b l I sampcal ysI Dr b a sI o a osl I u e bc ss Dl n b l o P I eI amp I ob sI nr p ds nnl l I nsI r oel asI D I sosI D l o Tb asamp bol as b sI ysbr n l aampC I dss I n r b a ol l I oal n n n b c ob l o Tb a b l I sI D oeal r De Dac es n sI o a u
i r Ty t I ei c c t n r c ci r t c c sua ct ro i c t r n t t
o s h
sDr a O snnr boao b oe l ampcr oo sl I n a eso or a l o PTb bol ah r oel asI D ol l nu e ol ah sol a sb r b sI a o sI D I amp I D sI D oe o PTb asamp bol as bu e b bol as b a sa o b I I o l oe es aa es n b ascob r b sI o u l y aC esn o PTb bol as b bosnn r b 13 b oe gI l n D a ca b I oo sl I nI Dr D C e I D oe sI o aI n o amp l n sI o P ol I n r ol amp o s oaI bno sl I l bol ah r oel a h r b a sI ol oe 13 gI l n D a ca b I oo sl I u
e I l n D sol a ol l sb r sno sI ol o PTb sI o a C I eI n b r b a bb ol oe gI l n D n amp I ob sI oe 13 a ca b I oo sl I l bol ahu o cal ys b oe r b a l I oal n l y a sI sI D oe I osos b sI bol ah b nn b oe a no sl I b amp l I D oe I osos bu e b l I a sI sDr a O a ab ol o b as sI oe ca ysl r b b osl I u
c t n pAsy ai onc S t ct ro i c t r n c s
sDr a ( snnr boao b oe ysbr n bhI oS r b h o PTb bol ah sol au e ob g sb a ca b I o b a oI Dn u b oe r b a amp l r b b l y a ob gC oe ol ah sol a sbcnh b oe 13 bc ss o sl I l oe ob gu e amp oel sb a ca b I o b b L r I l br ob gb oeo bc ss b oe l a asI D l oe br ob gb sI nr sI D so a osl I u ebr ob g sI or aI sb bc ss b a oI Dn soe sob l I amp oel b b oe I So nl a n y n l boa osl I u
i r My ai onc S t c ct ro ct r n c s
o o 13 h
sDr a B snnr boao b o PTb ol ah sol a sI o a u e amp s I c a o l oe sDr a l I os I b oe bol ah sI D so C sI oesb b oe l ol a b asco snnr boao sI sDr a Pu e c n oo l I oe ol c asDeo l sDr a B cal ys b oe s l I b l a sI D I ob g I amp oel n amp I ob ol oe bol ahu e sI l l I oe l ool amp n o c s ob oe I l n D sol au e sI l l I oe l ool amp asDeo sI s o b l oe a bol as b sI oe ol ah 13 s aa h I I n b oe r b a ol al b oe ns aa hu e Dac e sI oe l ool amp amps n I n b oe r b a ol I y sDo oe b asco l bol ahu
irkycs
uactroctrncr
TheStoryEditor inStab2gives theusermorecontrol thanwaspossible inStab In theoriginalStabsystemhierarchical scripts were hard‐coded in TMKL that itself is coded in the Lisp programming knowledge Thisresulted inseveraldifficulties regarding theusabilityof thesystemFirstly thestoredscripts inStabarenotavailableforreferenceorreviewexceptbyperusingtheactualLispcodeThislackoftransparencycanleadtoinaccurateandimpreciseunderstandingofthestoriesthataregeneratedbyStabasexplainingtheinputdataStab2rsquosStoryEditorprovidesvisualaccesstoboththelibraryofstoredscriptsaswellasthestoriesinstantiatedwhenStab2isrunforaninputdataset
SecondlysinceStabrsquosstoredscriptsarecodedinTMKLitisverydifficultforausertoauthoranewscriptorstoryevaluatethecorrectnessofastoredscriptandmakemodificationstoitStab2StoryEditorwithitsvisualinterface makes these tasks much easier though editing the knowledge elements in a story through theKnowledgeEditorstillisnon‐trivial
FinallytheinitialversionStabdidnotallowmodificationstothestructureofexistingscriptsatruntimeThisimpliesthat it isusefultorunStabjustonceonaninputdatasetHowever ingeneralahumananalystmaywanttorunStabseveraltimesndasheachwithdifferingvariationsofthescripts‐astheanalystrsquosinsightintothecriminal patterns evolves This is perhaps themost important side effect of the use of an interactive StoryEditor in Stab2 Stab2 enables an analyst to conduct ldquowhat ifrdquo simulations Using Stab2rsquos Story Editor andKnowledge Editor illustrated in Figure 7 an analystmay inspectmodify activate or deactivate the scriptsstored in Stab2rsquos Story Library the task and method elements in an individual script or the knowledgeelements likedomainconceptsand relationsTheanalystmay then runStab2 to simulate theeffectsof thepreciseconditionsshecreatedinStab2rsquosvirtualworld
Section34StoryAuthoringinStab2
Whenananalystwants tocreateanewstoryheneeds toopentheeditorandclickonaddstory iconThisopensawhiteareawithapaletteontherightWestartwithpickingtheldquoMainstoryldquoiconfromthepaletteanddropitintothewhiteboardThisrepresentsthenameofthestoryandisdepictedinFigure8(a)AfterthiswedragthebluerectanglethatreadsldquoIntermediaryGoalsrdquoandconnectthistothemainstoryusingaSub‐Goalconnector all ofwhich can be found in the palette The result is shown in Figure 8(b) Once themethod iscreatedwefillitinwithactionsperformedtoachievethatgoalThisisdoneusingtheldquoGoalsActionrdquoiconinthepaletteandthisisshowninFigure8(c)ThedraganddropcapabilitymakesitveryeasyforananalysttoconstructanewstoryplotThetreecangodownasmanylevelsdependingonthecomplexityofthestoryTherelationshipsbetweeneachofthegoalsarecapturedusingthelinksTwokindsoflinkagesarepossiblendashSub‐Goals that suggests that one goal is a child of another and Causal Connection that indicates relationshipsbetweendifferentactionsinsideeachintermediarygoal
Figure8(a)1
stSna
psho
tofStoryAutho
ringinStab2
ndashHerethehu
mananalystdragsth
eicon
mainstoryfrom
thepalette
andplacesitinth
eed
itorareaand
nam
esitasldquostoryplotrdquo
Figure8(b)A
2ndSna
psho
tofStoryAutho
ringin
Stab2
ndashHerethehu
mananalystdragsth
eInterm
ediaryGoalsicon
and
placesitbe
lowth
etopleveltaskAlsotheanalystdragsthesub‐goalslinkagefromthe
paletteand
con
nectsthetw
oblocksand
nam
esthe
bluerectangleas
ldquoMetho
d1rdquo
Figure8(c)ASna
psho
tofStoryAutho
ringinStab2
‐Heretheanalystd
ragsth
eGoalsA
ctionsicon
inside
thealreadycreatedmetho
dHe
then
con
nectstheseactio
nswith
thecausalcon
nectionslinkageinth
epaletteandnameseachactio
n
BehindtheScenes
Figure9(a)A
Sna
psho
tofaXMLfileforeachStoryPlot
Figure9(b)A
Sna
psho
tofaXMLfileforeachStoryPlotndash
Thisim
agehighlightsindividu
alta
gsThe
imageon
the
righth
ighlightsthechildTaskta
gandtheon
ethelefthighlightstheallM
etho
dsta
g
EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)
RelatedWork
ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors
StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)
Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo
Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)
CurrentWork
ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)
Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities
Acknowledgements
Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories
References
AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007
AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8
Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008
BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005
BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005
Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991
ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17
CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985
Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34
CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum
Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)
FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991
Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995
Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009
GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia
Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13
HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)
HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07
JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863
Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009
JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994
Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92
KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999
Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity
LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007
Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180
Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey
MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey
MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691
Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340
Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003
Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81
Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36
PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004
PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005
Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21
RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007
RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318
RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262
RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006
SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924
SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress
SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum
StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132
TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)
ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety
Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623
Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA
WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370
Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics
Figure1ExamplenewsstoryfromtheVAST‐2006dataset
(Adaptedfrom(httpwwwcsumdeduhcilVASTcontest06)Table1AsampleofinputstoStab
SampleSTABInputs ResultingState
stolen(money$40Highway‐Tire‐Store) Has‐object
cured‐disease(Boynton‐LabsPhilip‐Boyntonprion‐disease) Is‐rich‐and‐famous
named‐after(labPhilip‐BoyntonDean‐USC) Expert‐involved
was‐founded(Boynton‐Labs) Is‐open
have‐developed(Boynton‐Labsprion‐disease) Exists‐new‐disease
announced‐investigation(USFDABoynton‐Labs) Is‐investigating
Injected‐cow(Boynton‐Labsprion‐disease) Cow‐is‐infected
treatment‐cow(Boynton‐Labsprion‐disease) Cow‐is‐cured
Stabrsquos Library of Crime Stories STAB contains a small library of hierarchically organized abstract stories orhierarchicalscriptsthatcapturethepatternsofcrimesthatoccurintheVAST‐2006andVAST‐2007datasetsFigure2illustratesasimplescriptinSTABrsquoslibraryRobaStoreThisscripthasthegoalofachievingthestateofHaveMoneygiventheinitialstateofNotHaveMoney(topoffigure)Thisgoal(ortask)accordingtotheRobaStorescriptisachievedbyaplan(ormethod)thathasseveralactions(orevents)initGotoStoreBreakintoStore TakeMoney (middle of the figure) Each of these events becomes a task at the next lower level ofabstraction in the hierarchical script and each task can be (potentially) achieved bymultiplemethods ForexampleaccordingtotheRobaStorescriptthetaskofBreakintoStorecanbeachievedbyEnteringthroughaWindow or Entering through a Door (bottom of figure) Each of thesemethods in turn specifies a processconsistingofmultipleeventsandsoon
STABrsquos hierarchical scripts explicitly represent both goal and state at multiple levels of abstraction Whilerepresentation of the state caused by an event is useful for inferring causality representation of goals ofsequences of events is useful for inferring intention Figure 3 illustrates a more complex script of politicalconspiracyinwhichapoliticalfiguremaygetanopponentoutofanelectoralraceeitherbyexposingdirtonhim(politicalblackmail)orhavinghimassassinatedNotethisscriptiscomposedofseveralsmallerscripts
We found that sevenhierarchical scripts appear to cover all the illegalunethical activities in theVAST‐2006datasetandthatanotherfourhierarchicalscriptsapparentlyaresufficienttocovertheVAST‐2007aswellWehandcraftedthislibraryofscriptsintoSTAB
Figure2ThecontentandstructureofastoryinSTAB1(AdaptedfromGoeletal2009)
(
Stabrsquos Knowledge Representation STABrsquos scripts are represented in the TMKL language A task in TMKL isdefinedbyinputknowledgeelementsoutputknowledgeelementsrequiredinputconditions(pre‐conditions)desiredoutputconditions (post‐conditions)andmethods for implementing the taskAmethod isdefined intermsofsubtasksofthetaskandorderingofthesubtasksandisrepresentedbyafinitestatemachineFinallyknowledge in TMKL is defined in terms of domain concepts and relationships among them the input andoutputknowledgeelements ina taskspecificationrefer tothesedomainconceptsandrelationsMurdockampGoel2008providesdetailsofTMKL
Figure3ThestoryforapoliticalconspiracyintendedtoremoveanopponentfromanelectoralraceActivatednodesaredenotedbyathickoutlinearoundyellowboxes(AdaptedfromGoelet
al2009)
Figure4High‐LevelArchitectureofSTAB(AdaptedfromGoeletal2009)
t ba SV Ts t t SA D t tsD u sDr a H snnr boao b o Tb l ampcr oo sl I n a eso or a u saboC oe ys I l nn ol a l nn ob oe sI cr o y I ob sI I ys I sn sI eal I l nl Ds n l a au SoC oe ol ah o e a og b l I sI c r o y I o o osamp I r b b sob a br nosI D gI l n D boo l oe y I o soe oe ob g I l b sI oe 13 a ca b I oo sl I b l oe b ascob bol a sI oe ol ah 13 s aa hu e ol ah o e a oD b oe amp o esI D ob gb I c b b boe amp o esI D b ascob ol l agsI D amp l ahu esb o DDsI D l amp o esI D ob gb sI o sI boI oso b oe ob gu e ol ah o e a oe I sI bc ob oe I So sI cr o y I o sI oe ys I sn I a c o b oe l y cal bbu oe I sI cr o y I o a br nob sI oe a oas yn l I b ascoC oe I oe b asco sb bsampsna nh bol a sI oe l agsI D amp l ahu oe I nh a oas y b asco sb n a h sI oe l agsI D amp l ahC oe I sosl I n ob g I l b oeo amp o e oe I sI cr o a n bl oD D r o l I nh l I b asco sI boI sb g cou
t ba s tTs tau o l r ocr ob l r a gsI b l sI l aamp o sl I p n n sI boI oso b ascob sI oe l agsI D amp amp l ahC n n I l bsI e br e b asco oeo a amp o e soe I sI cr o y I oC n n ys I l a e br e b ascoC I l I s I yn r l a e br e b ascou e h nnl l S b sI sDr a H snnr boao oe ob gb sI oe c l nsos n l I bcsa h b asco oeo a amp o e soe I sI cr o y I o sI l I ar I I sI D l o u e l I s I yn r l a b asco sb b l I oe I r amp a l ob gb sI b asco oeo a amp o e soe sI cr o y I obu l n o n u dPVV cal ys os nb l o Tba b l I sI DC gI l n D C a ca b I oo sl I b I l r ocr obu
c t n py nc r c v ct ro c n n c s
e sI o a osy o P bhbo amp sb I r Damp I oo sl I l oe l asDsI n o bhbo amp soe casamp a h l r b l I sampcal ysI Dr b a sI o a osl I u e bc ss Dl n b l o P I eI amp I ob sI nr p ds nnl l I nsI r oel asI D I sosI D l o Tb asamp bol as b sI ysbr n l aampC I dss I n r b a ol l I oal n n n b c ob l o Tb a b l I sI D oeal r De Dac es n sI o a u
i r Ty t I ei c c t n r c ci r t c c sua ct ro i c t r n t t
o s h
sDr a O snnr boao b oe l ampcr oo sl I n a eso or a l o PTb bol ah r oel asI D ol l nu e ol ah sol a sb r b sI a o sI D I amp I D sI D oe o PTb asamp bol as bu e b bol as b a sa o b I I o l oe es aa es n b ascob r b sI o u l y aC esn o PTb bol as b bosnn r b 13 b oe gI l n D a ca b I oo sl I nI Dr D C e I D oe sI o aI n o amp l n sI o P ol I n r ol amp o s oaI bno sl I l bol ah r oel a h r b a sI ol oe 13 gI l n D a ca b I oo sl I u
e I l n D sol a ol l sb r sno sI ol o PTb sI o a C I eI n b r b a bb ol oe gI l n D n amp I ob sI oe 13 a ca b I oo sl I l bol ahu o cal ys b oe r b a l I oal n l y a sI sI D oe I osos b sI bol ah b nn b oe a no sl I b amp l I D oe I osos bu e b l I a sI sDr a O a ab ol o b as sI oe ca ysl r b b osl I u
c t n pAsy ai onc S t ct ro i c t r n c s
sDr a ( snnr boao b oe ysbr n bhI oS r b h o PTb bol ah sol au e ob g sb a ca b I o b a oI Dn u b oe r b a amp l r b b l y a ob gC oe ol ah sol a sbcnh b oe 13 bc ss o sl I l oe ob gu e amp oel sb a ca b I o b b L r I l br ob gb oeo bc ss b oe l a asI D l oe br ob gb sI nr sI D so a osl I u ebr ob g sI or aI sb bc ss b a oI Dn soe sob l I amp oel b b oe I So nl a n y n l boa osl I u
i r My ai onc S t c ct ro ct r n c s
o o 13 h
sDr a B snnr boao b o PTb ol ah sol a sI o a u e amp s I c a o l oe sDr a l I os I b oe bol ah sI D so C sI oesb b oe l ol a b asco snnr boao sI sDr a Pu e c n oo l I oe ol c asDeo l sDr a B cal ys b oe s l I b l a sI D I ob g I amp oel n amp I ob ol oe bol ahu e sI l l I oe l ool amp n o c s ob oe I l n D sol au e sI l l I oe l ool amp asDeo sI s o b l oe a bol as b sI oe ol ah 13 s aa h I I n b oe r b a ol al b oe ns aa hu e Dac e sI oe l ool amp amps n I n b oe r b a ol I y sDo oe b asco l bol ahu
irkycs
uactroctrncr
TheStoryEditor inStab2gives theusermorecontrol thanwaspossible inStab In theoriginalStabsystemhierarchical scripts were hard‐coded in TMKL that itself is coded in the Lisp programming knowledge Thisresulted inseveraldifficulties regarding theusabilityof thesystemFirstly thestoredscripts inStabarenotavailableforreferenceorreviewexceptbyperusingtheactualLispcodeThislackoftransparencycanleadtoinaccurateandimpreciseunderstandingofthestoriesthataregeneratedbyStabasexplainingtheinputdataStab2rsquosStoryEditorprovidesvisualaccesstoboththelibraryofstoredscriptsaswellasthestoriesinstantiatedwhenStab2isrunforaninputdataset
SecondlysinceStabrsquosstoredscriptsarecodedinTMKLitisverydifficultforausertoauthoranewscriptorstoryevaluatethecorrectnessofastoredscriptandmakemodificationstoitStab2StoryEditorwithitsvisualinterface makes these tasks much easier though editing the knowledge elements in a story through theKnowledgeEditorstillisnon‐trivial
FinallytheinitialversionStabdidnotallowmodificationstothestructureofexistingscriptsatruntimeThisimpliesthat it isusefultorunStabjustonceonaninputdatasetHowever ingeneralahumananalystmaywanttorunStabseveraltimesndasheachwithdifferingvariationsofthescripts‐astheanalystrsquosinsightintothecriminal patterns evolves This is perhaps themost important side effect of the use of an interactive StoryEditor in Stab2 Stab2 enables an analyst to conduct ldquowhat ifrdquo simulations Using Stab2rsquos Story Editor andKnowledge Editor illustrated in Figure 7 an analystmay inspectmodify activate or deactivate the scriptsstored in Stab2rsquos Story Library the task and method elements in an individual script or the knowledgeelements likedomainconceptsand relationsTheanalystmay then runStab2 to simulate theeffectsof thepreciseconditionsshecreatedinStab2rsquosvirtualworld
Section34StoryAuthoringinStab2
Whenananalystwants tocreateanewstoryheneeds toopentheeditorandclickonaddstory iconThisopensawhiteareawithapaletteontherightWestartwithpickingtheldquoMainstoryldquoiconfromthepaletteanddropitintothewhiteboardThisrepresentsthenameofthestoryandisdepictedinFigure8(a)AfterthiswedragthebluerectanglethatreadsldquoIntermediaryGoalsrdquoandconnectthistothemainstoryusingaSub‐Goalconnector all ofwhich can be found in the palette The result is shown in Figure 8(b) Once themethod iscreatedwefillitinwithactionsperformedtoachievethatgoalThisisdoneusingtheldquoGoalsActionrdquoiconinthepaletteandthisisshowninFigure8(c)ThedraganddropcapabilitymakesitveryeasyforananalysttoconstructanewstoryplotThetreecangodownasmanylevelsdependingonthecomplexityofthestoryTherelationshipsbetweeneachofthegoalsarecapturedusingthelinksTwokindsoflinkagesarepossiblendashSub‐Goals that suggests that one goal is a child of another and Causal Connection that indicates relationshipsbetweendifferentactionsinsideeachintermediarygoal
Figure8(a)1
stSna
psho
tofStoryAutho
ringinStab2
ndashHerethehu
mananalystdragsth
eicon
mainstoryfrom
thepalette
andplacesitinth
eed
itorareaand
nam
esitasldquostoryplotrdquo
Figure8(b)A
2ndSna
psho
tofStoryAutho
ringin
Stab2
ndashHerethehu
mananalystdragsth
eInterm
ediaryGoalsicon
and
placesitbe
lowth
etopleveltaskAlsotheanalystdragsthesub‐goalslinkagefromthe
paletteand
con
nectsthetw
oblocksand
nam
esthe
bluerectangleas
ldquoMetho
d1rdquo
Figure8(c)ASna
psho
tofStoryAutho
ringinStab2
‐Heretheanalystd
ragsth
eGoalsA
ctionsicon
inside
thealreadycreatedmetho
dHe
then
con
nectstheseactio
nswith
thecausalcon
nectionslinkageinth
epaletteandnameseachactio
n
BehindtheScenes
Figure9(a)A
Sna
psho
tofaXMLfileforeachStoryPlot
Figure9(b)A
Sna
psho
tofaXMLfileforeachStoryPlotndash
Thisim
agehighlightsindividu
alta
gsThe
imageon
the
righth
ighlightsthechildTaskta
gandtheon
ethelefthighlightstheallM
etho
dsta
g
EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)
RelatedWork
ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors
StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)
Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo
Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)
CurrentWork
ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)
Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities
Acknowledgements
Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories
References
AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007
AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8
Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008
BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005
BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005
Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991
ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17
CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985
Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34
CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum
Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)
FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991
Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995
Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009
GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia
Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13
HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)
HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07
JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863
Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009
JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994
Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92
KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999
Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity
LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007
Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180
Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey
MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey
MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691
Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340
Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003
Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81
Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36
PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004
PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005
Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21
RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007
RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318
RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262
RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006
SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924
SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress
SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum
StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132
TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)
ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety
Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623
Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA
WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370
Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics
Stabrsquos Library of Crime Stories STAB contains a small library of hierarchically organized abstract stories orhierarchicalscriptsthatcapturethepatternsofcrimesthatoccurintheVAST‐2006andVAST‐2007datasetsFigure2illustratesasimplescriptinSTABrsquoslibraryRobaStoreThisscripthasthegoalofachievingthestateofHaveMoneygiventheinitialstateofNotHaveMoney(topoffigure)Thisgoal(ortask)accordingtotheRobaStorescriptisachievedbyaplan(ormethod)thathasseveralactions(orevents)initGotoStoreBreakintoStore TakeMoney (middle of the figure) Each of these events becomes a task at the next lower level ofabstraction in the hierarchical script and each task can be (potentially) achieved bymultiplemethods ForexampleaccordingtotheRobaStorescriptthetaskofBreakintoStorecanbeachievedbyEnteringthroughaWindow or Entering through a Door (bottom of figure) Each of thesemethods in turn specifies a processconsistingofmultipleeventsandsoon
STABrsquos hierarchical scripts explicitly represent both goal and state at multiple levels of abstraction Whilerepresentation of the state caused by an event is useful for inferring causality representation of goals ofsequences of events is useful for inferring intention Figure 3 illustrates a more complex script of politicalconspiracyinwhichapoliticalfiguremaygetanopponentoutofanelectoralraceeitherbyexposingdirtonhim(politicalblackmail)orhavinghimassassinatedNotethisscriptiscomposedofseveralsmallerscripts
We found that sevenhierarchical scripts appear to cover all the illegalunethical activities in theVAST‐2006datasetandthatanotherfourhierarchicalscriptsapparentlyaresufficienttocovertheVAST‐2007aswellWehandcraftedthislibraryofscriptsintoSTAB
Figure2ThecontentandstructureofastoryinSTAB1(AdaptedfromGoeletal2009)
(
Stabrsquos Knowledge Representation STABrsquos scripts are represented in the TMKL language A task in TMKL isdefinedbyinputknowledgeelementsoutputknowledgeelementsrequiredinputconditions(pre‐conditions)desiredoutputconditions (post‐conditions)andmethods for implementing the taskAmethod isdefined intermsofsubtasksofthetaskandorderingofthesubtasksandisrepresentedbyafinitestatemachineFinallyknowledge in TMKL is defined in terms of domain concepts and relationships among them the input andoutputknowledgeelements ina taskspecificationrefer tothesedomainconceptsandrelationsMurdockampGoel2008providesdetailsofTMKL
Figure3ThestoryforapoliticalconspiracyintendedtoremoveanopponentfromanelectoralraceActivatednodesaredenotedbyathickoutlinearoundyellowboxes(AdaptedfromGoelet
al2009)
Figure4High‐LevelArchitectureofSTAB(AdaptedfromGoeletal2009)
t ba SV Ts t t SA D t tsD u sDr a H snnr boao b o Tb l ampcr oo sl I n a eso or a u saboC oe ys I l nn ol a l nn ob oe sI cr o y I ob sI I ys I sn sI eal I l nl Ds n l a au SoC oe ol ah o e a og b l I sI c r o y I o o osamp I r b b sob a br nosI D gI l n D boo l oe y I o soe oe ob g I l b sI oe 13 a ca b I oo sl I b l oe b ascob bol a sI oe ol ah 13 s aa hu e ol ah o e a oD b oe amp o esI D ob gb I c b b boe amp o esI D b ascob ol l agsI D amp l ahu esb o DDsI D l amp o esI D ob gb sI o sI boI oso b oe ob gu e ol ah o e a oe I sI bc ob oe I So sI cr o y I o sI oe ys I sn I a c o b oe l y cal bbu oe I sI cr o y I o a br nob sI oe a oas yn l I b ascoC oe I oe b asco sb bsampsna nh bol a sI oe l agsI D amp l ahu oe I nh a oas y b asco sb n a h sI oe l agsI D amp l ahC oe I sosl I n ob g I l b oeo amp o e oe I sI cr o a n bl oD D r o l I nh l I b asco sI boI sb g cou
t ba s tTs tau o l r ocr ob l r a gsI b l sI l aamp o sl I p n n sI boI oso b ascob sI oe l agsI D amp amp l ahC n n I l bsI e br e b asco oeo a amp o e soe I sI cr o y I oC n n ys I l a e br e b ascoC I l I s I yn r l a e br e b ascou e h nnl l S b sI sDr a H snnr boao oe ob gb sI oe c l nsos n l I bcsa h b asco oeo a amp o e soe I sI cr o y I o sI l I ar I I sI D l o u e l I s I yn r l a b asco sb b l I oe I r amp a l ob gb sI b asco oeo a amp o e soe sI cr o y I obu l n o n u dPVV cal ys os nb l o Tba b l I sI DC gI l n D C a ca b I oo sl I b I l r ocr obu
c t n py nc r c v ct ro c n n c s
e sI o a osy o P bhbo amp sb I r Damp I oo sl I l oe l asDsI n o bhbo amp soe casamp a h l r b l I sampcal ysI Dr b a sI o a osl I u e bc ss Dl n b l o P I eI amp I ob sI nr p ds nnl l I nsI r oel asI D I sosI D l o Tb asamp bol as b sI ysbr n l aampC I dss I n r b a ol l I oal n n n b c ob l o Tb a b l I sI D oeal r De Dac es n sI o a u
i r Ty t I ei c c t n r c ci r t c c sua ct ro i c t r n t t
o s h
sDr a O snnr boao b oe l ampcr oo sl I n a eso or a l o PTb bol ah r oel asI D ol l nu e ol ah sol a sb r b sI a o sI D I amp I D sI D oe o PTb asamp bol as bu e b bol as b a sa o b I I o l oe es aa es n b ascob r b sI o u l y aC esn o PTb bol as b bosnn r b 13 b oe gI l n D a ca b I oo sl I nI Dr D C e I D oe sI o aI n o amp l n sI o P ol I n r ol amp o s oaI bno sl I l bol ah r oel a h r b a sI ol oe 13 gI l n D a ca b I oo sl I u
e I l n D sol a ol l sb r sno sI ol o PTb sI o a C I eI n b r b a bb ol oe gI l n D n amp I ob sI oe 13 a ca b I oo sl I l bol ahu o cal ys b oe r b a l I oal n l y a sI sI D oe I osos b sI bol ah b nn b oe a no sl I b amp l I D oe I osos bu e b l I a sI sDr a O a ab ol o b as sI oe ca ysl r b b osl I u
c t n pAsy ai onc S t ct ro i c t r n c s
sDr a ( snnr boao b oe ysbr n bhI oS r b h o PTb bol ah sol au e ob g sb a ca b I o b a oI Dn u b oe r b a amp l r b b l y a ob gC oe ol ah sol a sbcnh b oe 13 bc ss o sl I l oe ob gu e amp oel sb a ca b I o b b L r I l br ob gb oeo bc ss b oe l a asI D l oe br ob gb sI nr sI D so a osl I u ebr ob g sI or aI sb bc ss b a oI Dn soe sob l I amp oel b b oe I So nl a n y n l boa osl I u
i r My ai onc S t c ct ro ct r n c s
o o 13 h
sDr a B snnr boao b o PTb ol ah sol a sI o a u e amp s I c a o l oe sDr a l I os I b oe bol ah sI D so C sI oesb b oe l ol a b asco snnr boao sI sDr a Pu e c n oo l I oe ol c asDeo l sDr a B cal ys b oe s l I b l a sI D I ob g I amp oel n amp I ob ol oe bol ahu e sI l l I oe l ool amp n o c s ob oe I l n D sol au e sI l l I oe l ool amp asDeo sI s o b l oe a bol as b sI oe ol ah 13 s aa h I I n b oe r b a ol al b oe ns aa hu e Dac e sI oe l ool amp amps n I n b oe r b a ol I y sDo oe b asco l bol ahu
irkycs
uactroctrncr
TheStoryEditor inStab2gives theusermorecontrol thanwaspossible inStab In theoriginalStabsystemhierarchical scripts were hard‐coded in TMKL that itself is coded in the Lisp programming knowledge Thisresulted inseveraldifficulties regarding theusabilityof thesystemFirstly thestoredscripts inStabarenotavailableforreferenceorreviewexceptbyperusingtheactualLispcodeThislackoftransparencycanleadtoinaccurateandimpreciseunderstandingofthestoriesthataregeneratedbyStabasexplainingtheinputdataStab2rsquosStoryEditorprovidesvisualaccesstoboththelibraryofstoredscriptsaswellasthestoriesinstantiatedwhenStab2isrunforaninputdataset
SecondlysinceStabrsquosstoredscriptsarecodedinTMKLitisverydifficultforausertoauthoranewscriptorstoryevaluatethecorrectnessofastoredscriptandmakemodificationstoitStab2StoryEditorwithitsvisualinterface makes these tasks much easier though editing the knowledge elements in a story through theKnowledgeEditorstillisnon‐trivial
FinallytheinitialversionStabdidnotallowmodificationstothestructureofexistingscriptsatruntimeThisimpliesthat it isusefultorunStabjustonceonaninputdatasetHowever ingeneralahumananalystmaywanttorunStabseveraltimesndasheachwithdifferingvariationsofthescripts‐astheanalystrsquosinsightintothecriminal patterns evolves This is perhaps themost important side effect of the use of an interactive StoryEditor in Stab2 Stab2 enables an analyst to conduct ldquowhat ifrdquo simulations Using Stab2rsquos Story Editor andKnowledge Editor illustrated in Figure 7 an analystmay inspectmodify activate or deactivate the scriptsstored in Stab2rsquos Story Library the task and method elements in an individual script or the knowledgeelements likedomainconceptsand relationsTheanalystmay then runStab2 to simulate theeffectsof thepreciseconditionsshecreatedinStab2rsquosvirtualworld
Section34StoryAuthoringinStab2
Whenananalystwants tocreateanewstoryheneeds toopentheeditorandclickonaddstory iconThisopensawhiteareawithapaletteontherightWestartwithpickingtheldquoMainstoryldquoiconfromthepaletteanddropitintothewhiteboardThisrepresentsthenameofthestoryandisdepictedinFigure8(a)AfterthiswedragthebluerectanglethatreadsldquoIntermediaryGoalsrdquoandconnectthistothemainstoryusingaSub‐Goalconnector all ofwhich can be found in the palette The result is shown in Figure 8(b) Once themethod iscreatedwefillitinwithactionsperformedtoachievethatgoalThisisdoneusingtheldquoGoalsActionrdquoiconinthepaletteandthisisshowninFigure8(c)ThedraganddropcapabilitymakesitveryeasyforananalysttoconstructanewstoryplotThetreecangodownasmanylevelsdependingonthecomplexityofthestoryTherelationshipsbetweeneachofthegoalsarecapturedusingthelinksTwokindsoflinkagesarepossiblendashSub‐Goals that suggests that one goal is a child of another and Causal Connection that indicates relationshipsbetweendifferentactionsinsideeachintermediarygoal
Figure8(a)1
stSna
psho
tofStoryAutho
ringinStab2
ndashHerethehu
mananalystdragsth
eicon
mainstoryfrom
thepalette
andplacesitinth
eed
itorareaand
nam
esitasldquostoryplotrdquo
Figure8(b)A
2ndSna
psho
tofStoryAutho
ringin
Stab2
ndashHerethehu
mananalystdragsth
eInterm
ediaryGoalsicon
and
placesitbe
lowth
etopleveltaskAlsotheanalystdragsthesub‐goalslinkagefromthe
paletteand
con
nectsthetw
oblocksand
nam
esthe
bluerectangleas
ldquoMetho
d1rdquo
Figure8(c)ASna
psho
tofStoryAutho
ringinStab2
‐Heretheanalystd
ragsth
eGoalsA
ctionsicon
inside
thealreadycreatedmetho
dHe
then
con
nectstheseactio
nswith
thecausalcon
nectionslinkageinth
epaletteandnameseachactio
n
BehindtheScenes
Figure9(a)A
Sna
psho
tofaXMLfileforeachStoryPlot
Figure9(b)A
Sna
psho
tofaXMLfileforeachStoryPlotndash
Thisim
agehighlightsindividu
alta
gsThe
imageon
the
righth
ighlightsthechildTaskta
gandtheon
ethelefthighlightstheallM
etho
dsta
g
EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)
RelatedWork
ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors
StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)
Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo
Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)
CurrentWork
ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)
Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities
Acknowledgements
Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories
References
AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007
AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8
Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008
BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005
BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005
Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991
ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17
CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985
Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34
CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum
Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)
FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991
Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995
Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009
GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia
Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13
HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)
HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07
JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863
Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009
JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994
Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92
KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999
Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity
LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007
Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180
Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey
MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey
MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691
Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340
Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003
Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81
Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36
PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004
PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005
Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21
RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007
RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318
RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262
RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006
SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924
SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress
SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum
StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132
TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)
ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety
Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623
Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA
WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370
Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics
Stabrsquos Knowledge Representation STABrsquos scripts are represented in the TMKL language A task in TMKL isdefinedbyinputknowledgeelementsoutputknowledgeelementsrequiredinputconditions(pre‐conditions)desiredoutputconditions (post‐conditions)andmethods for implementing the taskAmethod isdefined intermsofsubtasksofthetaskandorderingofthesubtasksandisrepresentedbyafinitestatemachineFinallyknowledge in TMKL is defined in terms of domain concepts and relationships among them the input andoutputknowledgeelements ina taskspecificationrefer tothesedomainconceptsandrelationsMurdockampGoel2008providesdetailsofTMKL
Figure3ThestoryforapoliticalconspiracyintendedtoremoveanopponentfromanelectoralraceActivatednodesaredenotedbyathickoutlinearoundyellowboxes(AdaptedfromGoelet
al2009)
Figure4High‐LevelArchitectureofSTAB(AdaptedfromGoeletal2009)
t ba SV Ts t t SA D t tsD u sDr a H snnr boao b o Tb l ampcr oo sl I n a eso or a u saboC oe ys I l nn ol a l nn ob oe sI cr o y I ob sI I ys I sn sI eal I l nl Ds n l a au SoC oe ol ah o e a og b l I sI c r o y I o o osamp I r b b sob a br nosI D gI l n D boo l oe y I o soe oe ob g I l b sI oe 13 a ca b I oo sl I b l oe b ascob bol a sI oe ol ah 13 s aa hu e ol ah o e a oD b oe amp o esI D ob gb I c b b boe amp o esI D b ascob ol l agsI D amp l ahu esb o DDsI D l amp o esI D ob gb sI o sI boI oso b oe ob gu e ol ah o e a oe I sI bc ob oe I So sI cr o y I o sI oe ys I sn I a c o b oe l y cal bbu oe I sI cr o y I o a br nob sI oe a oas yn l I b ascoC oe I oe b asco sb bsampsna nh bol a sI oe l agsI D amp l ahu oe I nh a oas y b asco sb n a h sI oe l agsI D amp l ahC oe I sosl I n ob g I l b oeo amp o e oe I sI cr o a n bl oD D r o l I nh l I b asco sI boI sb g cou
t ba s tTs tau o l r ocr ob l r a gsI b l sI l aamp o sl I p n n sI boI oso b ascob sI oe l agsI D amp amp l ahC n n I l bsI e br e b asco oeo a amp o e soe I sI cr o y I oC n n ys I l a e br e b ascoC I l I s I yn r l a e br e b ascou e h nnl l S b sI sDr a H snnr boao oe ob gb sI oe c l nsos n l I bcsa h b asco oeo a amp o e soe I sI cr o y I o sI l I ar I I sI D l o u e l I s I yn r l a b asco sb b l I oe I r amp a l ob gb sI b asco oeo a amp o e soe sI cr o y I obu l n o n u dPVV cal ys os nb l o Tba b l I sI DC gI l n D C a ca b I oo sl I b I l r ocr obu
c t n py nc r c v ct ro c n n c s
e sI o a osy o P bhbo amp sb I r Damp I oo sl I l oe l asDsI n o bhbo amp soe casamp a h l r b l I sampcal ysI Dr b a sI o a osl I u e bc ss Dl n b l o P I eI amp I ob sI nr p ds nnl l I nsI r oel asI D I sosI D l o Tb asamp bol as b sI ysbr n l aampC I dss I n r b a ol l I oal n n n b c ob l o Tb a b l I sI D oeal r De Dac es n sI o a u
i r Ty t I ei c c t n r c ci r t c c sua ct ro i c t r n t t
o s h
sDr a O snnr boao b oe l ampcr oo sl I n a eso or a l o PTb bol ah r oel asI D ol l nu e ol ah sol a sb r b sI a o sI D I amp I D sI D oe o PTb asamp bol as bu e b bol as b a sa o b I I o l oe es aa es n b ascob r b sI o u l y aC esn o PTb bol as b bosnn r b 13 b oe gI l n D a ca b I oo sl I nI Dr D C e I D oe sI o aI n o amp l n sI o P ol I n r ol amp o s oaI bno sl I l bol ah r oel a h r b a sI ol oe 13 gI l n D a ca b I oo sl I u
e I l n D sol a ol l sb r sno sI ol o PTb sI o a C I eI n b r b a bb ol oe gI l n D n amp I ob sI oe 13 a ca b I oo sl I l bol ahu o cal ys b oe r b a l I oal n l y a sI sI D oe I osos b sI bol ah b nn b oe a no sl I b amp l I D oe I osos bu e b l I a sI sDr a O a ab ol o b as sI oe ca ysl r b b osl I u
c t n pAsy ai onc S t ct ro i c t r n c s
sDr a ( snnr boao b oe ysbr n bhI oS r b h o PTb bol ah sol au e ob g sb a ca b I o b a oI Dn u b oe r b a amp l r b b l y a ob gC oe ol ah sol a sbcnh b oe 13 bc ss o sl I l oe ob gu e amp oel sb a ca b I o b b L r I l br ob gb oeo bc ss b oe l a asI D l oe br ob gb sI nr sI D so a osl I u ebr ob g sI or aI sb bc ss b a oI Dn soe sob l I amp oel b b oe I So nl a n y n l boa osl I u
i r My ai onc S t c ct ro ct r n c s
o o 13 h
sDr a B snnr boao b o PTb ol ah sol a sI o a u e amp s I c a o l oe sDr a l I os I b oe bol ah sI D so C sI oesb b oe l ol a b asco snnr boao sI sDr a Pu e c n oo l I oe ol c asDeo l sDr a B cal ys b oe s l I b l a sI D I ob g I amp oel n amp I ob ol oe bol ahu e sI l l I oe l ool amp n o c s ob oe I l n D sol au e sI l l I oe l ool amp asDeo sI s o b l oe a bol as b sI oe ol ah 13 s aa h I I n b oe r b a ol al b oe ns aa hu e Dac e sI oe l ool amp amps n I n b oe r b a ol I y sDo oe b asco l bol ahu
irkycs
uactroctrncr
TheStoryEditor inStab2gives theusermorecontrol thanwaspossible inStab In theoriginalStabsystemhierarchical scripts were hard‐coded in TMKL that itself is coded in the Lisp programming knowledge Thisresulted inseveraldifficulties regarding theusabilityof thesystemFirstly thestoredscripts inStabarenotavailableforreferenceorreviewexceptbyperusingtheactualLispcodeThislackoftransparencycanleadtoinaccurateandimpreciseunderstandingofthestoriesthataregeneratedbyStabasexplainingtheinputdataStab2rsquosStoryEditorprovidesvisualaccesstoboththelibraryofstoredscriptsaswellasthestoriesinstantiatedwhenStab2isrunforaninputdataset
SecondlysinceStabrsquosstoredscriptsarecodedinTMKLitisverydifficultforausertoauthoranewscriptorstoryevaluatethecorrectnessofastoredscriptandmakemodificationstoitStab2StoryEditorwithitsvisualinterface makes these tasks much easier though editing the knowledge elements in a story through theKnowledgeEditorstillisnon‐trivial
FinallytheinitialversionStabdidnotallowmodificationstothestructureofexistingscriptsatruntimeThisimpliesthat it isusefultorunStabjustonceonaninputdatasetHowever ingeneralahumananalystmaywanttorunStabseveraltimesndasheachwithdifferingvariationsofthescripts‐astheanalystrsquosinsightintothecriminal patterns evolves This is perhaps themost important side effect of the use of an interactive StoryEditor in Stab2 Stab2 enables an analyst to conduct ldquowhat ifrdquo simulations Using Stab2rsquos Story Editor andKnowledge Editor illustrated in Figure 7 an analystmay inspectmodify activate or deactivate the scriptsstored in Stab2rsquos Story Library the task and method elements in an individual script or the knowledgeelements likedomainconceptsand relationsTheanalystmay then runStab2 to simulate theeffectsof thepreciseconditionsshecreatedinStab2rsquosvirtualworld
Section34StoryAuthoringinStab2
Whenananalystwants tocreateanewstoryheneeds toopentheeditorandclickonaddstory iconThisopensawhiteareawithapaletteontherightWestartwithpickingtheldquoMainstoryldquoiconfromthepaletteanddropitintothewhiteboardThisrepresentsthenameofthestoryandisdepictedinFigure8(a)AfterthiswedragthebluerectanglethatreadsldquoIntermediaryGoalsrdquoandconnectthistothemainstoryusingaSub‐Goalconnector all ofwhich can be found in the palette The result is shown in Figure 8(b) Once themethod iscreatedwefillitinwithactionsperformedtoachievethatgoalThisisdoneusingtheldquoGoalsActionrdquoiconinthepaletteandthisisshowninFigure8(c)ThedraganddropcapabilitymakesitveryeasyforananalysttoconstructanewstoryplotThetreecangodownasmanylevelsdependingonthecomplexityofthestoryTherelationshipsbetweeneachofthegoalsarecapturedusingthelinksTwokindsoflinkagesarepossiblendashSub‐Goals that suggests that one goal is a child of another and Causal Connection that indicates relationshipsbetweendifferentactionsinsideeachintermediarygoal
Figure8(a)1
stSna
psho
tofStoryAutho
ringinStab2
ndashHerethehu
mananalystdragsth
eicon
mainstoryfrom
thepalette
andplacesitinth
eed
itorareaand
nam
esitasldquostoryplotrdquo
Figure8(b)A
2ndSna
psho
tofStoryAutho
ringin
Stab2
ndashHerethehu
mananalystdragsth
eInterm
ediaryGoalsicon
and
placesitbe
lowth
etopleveltaskAlsotheanalystdragsthesub‐goalslinkagefromthe
paletteand
con
nectsthetw
oblocksand
nam
esthe
bluerectangleas
ldquoMetho
d1rdquo
Figure8(c)ASna
psho
tofStoryAutho
ringinStab2
‐Heretheanalystd
ragsth
eGoalsA
ctionsicon
inside
thealreadycreatedmetho
dHe
then
con
nectstheseactio
nswith
thecausalcon
nectionslinkageinth
epaletteandnameseachactio
n
BehindtheScenes
Figure9(a)A
Sna
psho
tofaXMLfileforeachStoryPlot
Figure9(b)A
Sna
psho
tofaXMLfileforeachStoryPlotndash
Thisim
agehighlightsindividu
alta
gsThe
imageon
the
righth
ighlightsthechildTaskta
gandtheon
ethelefthighlightstheallM
etho
dsta
g
EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)
RelatedWork
ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors
StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)
Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo
Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)
CurrentWork
ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)
Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities
Acknowledgements
Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories
References
AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007
AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8
Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008
BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005
BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005
Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991
ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17
CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985
Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34
CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum
Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)
FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991
Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995
Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009
GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia
Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13
HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)
HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07
JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863
Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009
JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994
Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92
KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999
Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity
LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007
Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180
Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey
MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey
MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691
Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340
Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003
Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81
Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36
PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004
PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005
Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21
RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007
RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318
RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262
RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006
SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924
SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress
SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum
StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132
TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)
ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety
Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623
Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA
WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370
Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics
t ba SV Ts t t SA D t tsD u sDr a H snnr boao b o Tb l ampcr oo sl I n a eso or a u saboC oe ys I l nn ol a l nn ob oe sI cr o y I ob sI I ys I sn sI eal I l nl Ds n l a au SoC oe ol ah o e a og b l I sI c r o y I o o osamp I r b b sob a br nosI D gI l n D boo l oe y I o soe oe ob g I l b sI oe 13 a ca b I oo sl I b l oe b ascob bol a sI oe ol ah 13 s aa hu e ol ah o e a oD b oe amp o esI D ob gb I c b b boe amp o esI D b ascob ol l agsI D amp l ahu esb o DDsI D l amp o esI D ob gb sI o sI boI oso b oe ob gu e ol ah o e a oe I sI bc ob oe I So sI cr o y I o sI oe ys I sn I a c o b oe l y cal bbu oe I sI cr o y I o a br nob sI oe a oas yn l I b ascoC oe I oe b asco sb bsampsna nh bol a sI oe l agsI D amp l ahu oe I nh a oas y b asco sb n a h sI oe l agsI D amp l ahC oe I sosl I n ob g I l b oeo amp o e oe I sI cr o a n bl oD D r o l I nh l I b asco sI boI sb g cou
t ba s tTs tau o l r ocr ob l r a gsI b l sI l aamp o sl I p n n sI boI oso b ascob sI oe l agsI D amp amp l ahC n n I l bsI e br e b asco oeo a amp o e soe I sI cr o y I oC n n ys I l a e br e b ascoC I l I s I yn r l a e br e b ascou e h nnl l S b sI sDr a H snnr boao oe ob gb sI oe c l nsos n l I bcsa h b asco oeo a amp o e soe I sI cr o y I o sI l I ar I I sI D l o u e l I s I yn r l a b asco sb b l I oe I r amp a l ob gb sI b asco oeo a amp o e soe sI cr o y I obu l n o n u dPVV cal ys os nb l o Tba b l I sI DC gI l n D C a ca b I oo sl I b I l r ocr obu
c t n py nc r c v ct ro c n n c s
e sI o a osy o P bhbo amp sb I r Damp I oo sl I l oe l asDsI n o bhbo amp soe casamp a h l r b l I sampcal ysI Dr b a sI o a osl I u e bc ss Dl n b l o P I eI amp I ob sI nr p ds nnl l I nsI r oel asI D I sosI D l o Tb asamp bol as b sI ysbr n l aampC I dss I n r b a ol l I oal n n n b c ob l o Tb a b l I sI D oeal r De Dac es n sI o a u
i r Ty t I ei c c t n r c ci r t c c sua ct ro i c t r n t t
o s h
sDr a O snnr boao b oe l ampcr oo sl I n a eso or a l o PTb bol ah r oel asI D ol l nu e ol ah sol a sb r b sI a o sI D I amp I D sI D oe o PTb asamp bol as bu e b bol as b a sa o b I I o l oe es aa es n b ascob r b sI o u l y aC esn o PTb bol as b bosnn r b 13 b oe gI l n D a ca b I oo sl I nI Dr D C e I D oe sI o aI n o amp l n sI o P ol I n r ol amp o s oaI bno sl I l bol ah r oel a h r b a sI ol oe 13 gI l n D a ca b I oo sl I u
e I l n D sol a ol l sb r sno sI ol o PTb sI o a C I eI n b r b a bb ol oe gI l n D n amp I ob sI oe 13 a ca b I oo sl I l bol ahu o cal ys b oe r b a l I oal n l y a sI sI D oe I osos b sI bol ah b nn b oe a no sl I b amp l I D oe I osos bu e b l I a sI sDr a O a ab ol o b as sI oe ca ysl r b b osl I u
c t n pAsy ai onc S t ct ro i c t r n c s
sDr a ( snnr boao b oe ysbr n bhI oS r b h o PTb bol ah sol au e ob g sb a ca b I o b a oI Dn u b oe r b a amp l r b b l y a ob gC oe ol ah sol a sbcnh b oe 13 bc ss o sl I l oe ob gu e amp oel sb a ca b I o b b L r I l br ob gb oeo bc ss b oe l a asI D l oe br ob gb sI nr sI D so a osl I u ebr ob g sI or aI sb bc ss b a oI Dn soe sob l I amp oel b b oe I So nl a n y n l boa osl I u
i r My ai onc S t c ct ro ct r n c s
o o 13 h
sDr a B snnr boao b o PTb ol ah sol a sI o a u e amp s I c a o l oe sDr a l I os I b oe bol ah sI D so C sI oesb b oe l ol a b asco snnr boao sI sDr a Pu e c n oo l I oe ol c asDeo l sDr a B cal ys b oe s l I b l a sI D I ob g I amp oel n amp I ob ol oe bol ahu e sI l l I oe l ool amp n o c s ob oe I l n D sol au e sI l l I oe l ool amp asDeo sI s o b l oe a bol as b sI oe ol ah 13 s aa h I I n b oe r b a ol al b oe ns aa hu e Dac e sI oe l ool amp amps n I n b oe r b a ol I y sDo oe b asco l bol ahu
irkycs
uactroctrncr
TheStoryEditor inStab2gives theusermorecontrol thanwaspossible inStab In theoriginalStabsystemhierarchical scripts were hard‐coded in TMKL that itself is coded in the Lisp programming knowledge Thisresulted inseveraldifficulties regarding theusabilityof thesystemFirstly thestoredscripts inStabarenotavailableforreferenceorreviewexceptbyperusingtheactualLispcodeThislackoftransparencycanleadtoinaccurateandimpreciseunderstandingofthestoriesthataregeneratedbyStabasexplainingtheinputdataStab2rsquosStoryEditorprovidesvisualaccesstoboththelibraryofstoredscriptsaswellasthestoriesinstantiatedwhenStab2isrunforaninputdataset
SecondlysinceStabrsquosstoredscriptsarecodedinTMKLitisverydifficultforausertoauthoranewscriptorstoryevaluatethecorrectnessofastoredscriptandmakemodificationstoitStab2StoryEditorwithitsvisualinterface makes these tasks much easier though editing the knowledge elements in a story through theKnowledgeEditorstillisnon‐trivial
FinallytheinitialversionStabdidnotallowmodificationstothestructureofexistingscriptsatruntimeThisimpliesthat it isusefultorunStabjustonceonaninputdatasetHowever ingeneralahumananalystmaywanttorunStabseveraltimesndasheachwithdifferingvariationsofthescripts‐astheanalystrsquosinsightintothecriminal patterns evolves This is perhaps themost important side effect of the use of an interactive StoryEditor in Stab2 Stab2 enables an analyst to conduct ldquowhat ifrdquo simulations Using Stab2rsquos Story Editor andKnowledge Editor illustrated in Figure 7 an analystmay inspectmodify activate or deactivate the scriptsstored in Stab2rsquos Story Library the task and method elements in an individual script or the knowledgeelements likedomainconceptsand relationsTheanalystmay then runStab2 to simulate theeffectsof thepreciseconditionsshecreatedinStab2rsquosvirtualworld
Section34StoryAuthoringinStab2
Whenananalystwants tocreateanewstoryheneeds toopentheeditorandclickonaddstory iconThisopensawhiteareawithapaletteontherightWestartwithpickingtheldquoMainstoryldquoiconfromthepaletteanddropitintothewhiteboardThisrepresentsthenameofthestoryandisdepictedinFigure8(a)AfterthiswedragthebluerectanglethatreadsldquoIntermediaryGoalsrdquoandconnectthistothemainstoryusingaSub‐Goalconnector all ofwhich can be found in the palette The result is shown in Figure 8(b) Once themethod iscreatedwefillitinwithactionsperformedtoachievethatgoalThisisdoneusingtheldquoGoalsActionrdquoiconinthepaletteandthisisshowninFigure8(c)ThedraganddropcapabilitymakesitveryeasyforananalysttoconstructanewstoryplotThetreecangodownasmanylevelsdependingonthecomplexityofthestoryTherelationshipsbetweeneachofthegoalsarecapturedusingthelinksTwokindsoflinkagesarepossiblendashSub‐Goals that suggests that one goal is a child of another and Causal Connection that indicates relationshipsbetweendifferentactionsinsideeachintermediarygoal
Figure8(a)1
stSna
psho
tofStoryAutho
ringinStab2
ndashHerethehu
mananalystdragsth
eicon
mainstoryfrom
thepalette
andplacesitinth
eed
itorareaand
nam
esitasldquostoryplotrdquo
Figure8(b)A
2ndSna
psho
tofStoryAutho
ringin
Stab2
ndashHerethehu
mananalystdragsth
eInterm
ediaryGoalsicon
and
placesitbe
lowth
etopleveltaskAlsotheanalystdragsthesub‐goalslinkagefromthe
paletteand
con
nectsthetw
oblocksand
nam
esthe
bluerectangleas
ldquoMetho
d1rdquo
Figure8(c)ASna
psho
tofStoryAutho
ringinStab2
‐Heretheanalystd
ragsth
eGoalsA
ctionsicon
inside
thealreadycreatedmetho
dHe
then
con
nectstheseactio
nswith
thecausalcon
nectionslinkageinth
epaletteandnameseachactio
n
BehindtheScenes
Figure9(a)A
Sna
psho
tofaXMLfileforeachStoryPlot
Figure9(b)A
Sna
psho
tofaXMLfileforeachStoryPlotndash
Thisim
agehighlightsindividu
alta
gsThe
imageon
the
righth
ighlightsthechildTaskta
gandtheon
ethelefthighlightstheallM
etho
dsta
g
EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)
RelatedWork
ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors
StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)
Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo
Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)
CurrentWork
ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)
Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities
Acknowledgements
Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories
References
AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007
AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8
Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008
BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005
BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005
Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991
ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17
CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985
Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34
CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum
Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)
FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991
Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995
Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009
GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia
Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13
HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)
HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07
JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863
Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009
JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994
Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92
KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999
Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity
LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007
Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180
Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey
MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey
MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691
Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340
Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003
Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81
Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36
PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004
PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005
Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21
RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007
RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318
RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262
RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006
SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924
SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress
SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum
StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132
TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)
ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety
Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623
Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA
WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370
Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics
o s h
sDr a O snnr boao b oe l ampcr oo sl I n a eso or a l o PTb bol ah r oel asI D ol l nu e ol ah sol a sb r b sI a o sI D I amp I D sI D oe o PTb asamp bol as bu e b bol as b a sa o b I I o l oe es aa es n b ascob r b sI o u l y aC esn o PTb bol as b bosnn r b 13 b oe gI l n D a ca b I oo sl I nI Dr D C e I D oe sI o aI n o amp l n sI o P ol I n r ol amp o s oaI bno sl I l bol ah r oel a h r b a sI ol oe 13 gI l n D a ca b I oo sl I u
e I l n D sol a ol l sb r sno sI ol o PTb sI o a C I eI n b r b a bb ol oe gI l n D n amp I ob sI oe 13 a ca b I oo sl I l bol ahu o cal ys b oe r b a l I oal n l y a sI sI D oe I osos b sI bol ah b nn b oe a no sl I b amp l I D oe I osos bu e b l I a sI sDr a O a ab ol o b as sI oe ca ysl r b b osl I u
c t n pAsy ai onc S t ct ro i c t r n c s
sDr a ( snnr boao b oe ysbr n bhI oS r b h o PTb bol ah sol au e ob g sb a ca b I o b a oI Dn u b oe r b a amp l r b b l y a ob gC oe ol ah sol a sbcnh b oe 13 bc ss o sl I l oe ob gu e amp oel sb a ca b I o b b L r I l br ob gb oeo bc ss b oe l a asI D l oe br ob gb sI nr sI D so a osl I u ebr ob g sI or aI sb bc ss b a oI Dn soe sob l I amp oel b b oe I So nl a n y n l boa osl I u
i r My ai onc S t c ct ro ct r n c s
o o 13 h
sDr a B snnr boao b o PTb ol ah sol a sI o a u e amp s I c a o l oe sDr a l I os I b oe bol ah sI D so C sI oesb b oe l ol a b asco snnr boao sI sDr a Pu e c n oo l I oe ol c asDeo l sDr a B cal ys b oe s l I b l a sI D I ob g I amp oel n amp I ob ol oe bol ahu e sI l l I oe l ool amp n o c s ob oe I l n D sol au e sI l l I oe l ool amp asDeo sI s o b l oe a bol as b sI oe ol ah 13 s aa h I I n b oe r b a ol al b oe ns aa hu e Dac e sI oe l ool amp amps n I n b oe r b a ol I y sDo oe b asco l bol ahu
irkycs
uactroctrncr
TheStoryEditor inStab2gives theusermorecontrol thanwaspossible inStab In theoriginalStabsystemhierarchical scripts were hard‐coded in TMKL that itself is coded in the Lisp programming knowledge Thisresulted inseveraldifficulties regarding theusabilityof thesystemFirstly thestoredscripts inStabarenotavailableforreferenceorreviewexceptbyperusingtheactualLispcodeThislackoftransparencycanleadtoinaccurateandimpreciseunderstandingofthestoriesthataregeneratedbyStabasexplainingtheinputdataStab2rsquosStoryEditorprovidesvisualaccesstoboththelibraryofstoredscriptsaswellasthestoriesinstantiatedwhenStab2isrunforaninputdataset
SecondlysinceStabrsquosstoredscriptsarecodedinTMKLitisverydifficultforausertoauthoranewscriptorstoryevaluatethecorrectnessofastoredscriptandmakemodificationstoitStab2StoryEditorwithitsvisualinterface makes these tasks much easier though editing the knowledge elements in a story through theKnowledgeEditorstillisnon‐trivial
FinallytheinitialversionStabdidnotallowmodificationstothestructureofexistingscriptsatruntimeThisimpliesthat it isusefultorunStabjustonceonaninputdatasetHowever ingeneralahumananalystmaywanttorunStabseveraltimesndasheachwithdifferingvariationsofthescripts‐astheanalystrsquosinsightintothecriminal patterns evolves This is perhaps themost important side effect of the use of an interactive StoryEditor in Stab2 Stab2 enables an analyst to conduct ldquowhat ifrdquo simulations Using Stab2rsquos Story Editor andKnowledge Editor illustrated in Figure 7 an analystmay inspectmodify activate or deactivate the scriptsstored in Stab2rsquos Story Library the task and method elements in an individual script or the knowledgeelements likedomainconceptsand relationsTheanalystmay then runStab2 to simulate theeffectsof thepreciseconditionsshecreatedinStab2rsquosvirtualworld
Section34StoryAuthoringinStab2
Whenananalystwants tocreateanewstoryheneeds toopentheeditorandclickonaddstory iconThisopensawhiteareawithapaletteontherightWestartwithpickingtheldquoMainstoryldquoiconfromthepaletteanddropitintothewhiteboardThisrepresentsthenameofthestoryandisdepictedinFigure8(a)AfterthiswedragthebluerectanglethatreadsldquoIntermediaryGoalsrdquoandconnectthistothemainstoryusingaSub‐Goalconnector all ofwhich can be found in the palette The result is shown in Figure 8(b) Once themethod iscreatedwefillitinwithactionsperformedtoachievethatgoalThisisdoneusingtheldquoGoalsActionrdquoiconinthepaletteandthisisshowninFigure8(c)ThedraganddropcapabilitymakesitveryeasyforananalysttoconstructanewstoryplotThetreecangodownasmanylevelsdependingonthecomplexityofthestoryTherelationshipsbetweeneachofthegoalsarecapturedusingthelinksTwokindsoflinkagesarepossiblendashSub‐Goals that suggests that one goal is a child of another and Causal Connection that indicates relationshipsbetweendifferentactionsinsideeachintermediarygoal
Figure8(a)1
stSna
psho
tofStoryAutho
ringinStab2
ndashHerethehu
mananalystdragsth
eicon
mainstoryfrom
thepalette
andplacesitinth
eed
itorareaand
nam
esitasldquostoryplotrdquo
Figure8(b)A
2ndSna
psho
tofStoryAutho
ringin
Stab2
ndashHerethehu
mananalystdragsth
eInterm
ediaryGoalsicon
and
placesitbe
lowth
etopleveltaskAlsotheanalystdragsthesub‐goalslinkagefromthe
paletteand
con
nectsthetw
oblocksand
nam
esthe
bluerectangleas
ldquoMetho
d1rdquo
Figure8(c)ASna
psho
tofStoryAutho
ringinStab2
‐Heretheanalystd
ragsth
eGoalsA
ctionsicon
inside
thealreadycreatedmetho
dHe
then
con
nectstheseactio
nswith
thecausalcon
nectionslinkageinth
epaletteandnameseachactio
n
BehindtheScenes
Figure9(a)A
Sna
psho
tofaXMLfileforeachStoryPlot
Figure9(b)A
Sna
psho
tofaXMLfileforeachStoryPlotndash
Thisim
agehighlightsindividu
alta
gsThe
imageon
the
righth
ighlightsthechildTaskta
gandtheon
ethelefthighlightstheallM
etho
dsta
g
EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)
RelatedWork
ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors
StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)
Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo
Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)
CurrentWork
ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)
Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities
Acknowledgements
Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories
References
AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007
AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8
Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008
BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005
BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005
Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991
ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17
CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985
Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34
CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum
Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)
FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991
Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995
Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009
GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia
Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13
HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)
HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07
JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863
Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009
JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994
Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92
KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999
Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity
LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007
Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180
Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey
MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey
MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691
Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340
Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003
Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81
Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36
PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004
PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005
Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21
RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007
RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318
RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262
RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006
SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924
SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress
SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum
StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132
TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)
ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety
Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623
Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA
WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370
Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics
irkycs
uactroctrncr
TheStoryEditor inStab2gives theusermorecontrol thanwaspossible inStab In theoriginalStabsystemhierarchical scripts were hard‐coded in TMKL that itself is coded in the Lisp programming knowledge Thisresulted inseveraldifficulties regarding theusabilityof thesystemFirstly thestoredscripts inStabarenotavailableforreferenceorreviewexceptbyperusingtheactualLispcodeThislackoftransparencycanleadtoinaccurateandimpreciseunderstandingofthestoriesthataregeneratedbyStabasexplainingtheinputdataStab2rsquosStoryEditorprovidesvisualaccesstoboththelibraryofstoredscriptsaswellasthestoriesinstantiatedwhenStab2isrunforaninputdataset
SecondlysinceStabrsquosstoredscriptsarecodedinTMKLitisverydifficultforausertoauthoranewscriptorstoryevaluatethecorrectnessofastoredscriptandmakemodificationstoitStab2StoryEditorwithitsvisualinterface makes these tasks much easier though editing the knowledge elements in a story through theKnowledgeEditorstillisnon‐trivial
FinallytheinitialversionStabdidnotallowmodificationstothestructureofexistingscriptsatruntimeThisimpliesthat it isusefultorunStabjustonceonaninputdatasetHowever ingeneralahumananalystmaywanttorunStabseveraltimesndasheachwithdifferingvariationsofthescripts‐astheanalystrsquosinsightintothecriminal patterns evolves This is perhaps themost important side effect of the use of an interactive StoryEditor in Stab2 Stab2 enables an analyst to conduct ldquowhat ifrdquo simulations Using Stab2rsquos Story Editor andKnowledge Editor illustrated in Figure 7 an analystmay inspectmodify activate or deactivate the scriptsstored in Stab2rsquos Story Library the task and method elements in an individual script or the knowledgeelements likedomainconceptsand relationsTheanalystmay then runStab2 to simulate theeffectsof thepreciseconditionsshecreatedinStab2rsquosvirtualworld
Section34StoryAuthoringinStab2
Whenananalystwants tocreateanewstoryheneeds toopentheeditorandclickonaddstory iconThisopensawhiteareawithapaletteontherightWestartwithpickingtheldquoMainstoryldquoiconfromthepaletteanddropitintothewhiteboardThisrepresentsthenameofthestoryandisdepictedinFigure8(a)AfterthiswedragthebluerectanglethatreadsldquoIntermediaryGoalsrdquoandconnectthistothemainstoryusingaSub‐Goalconnector all ofwhich can be found in the palette The result is shown in Figure 8(b) Once themethod iscreatedwefillitinwithactionsperformedtoachievethatgoalThisisdoneusingtheldquoGoalsActionrdquoiconinthepaletteandthisisshowninFigure8(c)ThedraganddropcapabilitymakesitveryeasyforananalysttoconstructanewstoryplotThetreecangodownasmanylevelsdependingonthecomplexityofthestoryTherelationshipsbetweeneachofthegoalsarecapturedusingthelinksTwokindsoflinkagesarepossiblendashSub‐Goals that suggests that one goal is a child of another and Causal Connection that indicates relationshipsbetweendifferentactionsinsideeachintermediarygoal
Figure8(a)1
stSna
psho
tofStoryAutho
ringinStab2
ndashHerethehu
mananalystdragsth
eicon
mainstoryfrom
thepalette
andplacesitinth
eed
itorareaand
nam
esitasldquostoryplotrdquo
Figure8(b)A
2ndSna
psho
tofStoryAutho
ringin
Stab2
ndashHerethehu
mananalystdragsth
eInterm
ediaryGoalsicon
and
placesitbe
lowth
etopleveltaskAlsotheanalystdragsthesub‐goalslinkagefromthe
paletteand
con
nectsthetw
oblocksand
nam
esthe
bluerectangleas
ldquoMetho
d1rdquo
Figure8(c)ASna
psho
tofStoryAutho
ringinStab2
‐Heretheanalystd
ragsth
eGoalsA
ctionsicon
inside
thealreadycreatedmetho
dHe
then
con
nectstheseactio
nswith
thecausalcon
nectionslinkageinth
epaletteandnameseachactio
n
BehindtheScenes
Figure9(a)A
Sna
psho
tofaXMLfileforeachStoryPlot
Figure9(b)A
Sna
psho
tofaXMLfileforeachStoryPlotndash
Thisim
agehighlightsindividu
alta
gsThe
imageon
the
righth
ighlightsthechildTaskta
gandtheon
ethelefthighlightstheallM
etho
dsta
g
EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)
RelatedWork
ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors
StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)
Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo
Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)
CurrentWork
ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)
Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities
Acknowledgements
Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories
References
AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007
AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8
Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008
BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005
BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005
Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991
ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17
CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985
Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34
CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum
Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)
FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991
Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995
Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009
GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia
Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13
HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)
HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07
JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863
Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009
JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994
Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92
KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999
Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity
LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007
Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180
Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey
MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey
MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691
Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340
Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003
Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81
Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36
PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004
PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005
Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21
RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007
RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318
RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262
RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006
SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924
SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress
SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum
StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132
TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)
ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety
Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623
Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA
WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370
Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics
TheStoryEditor inStab2gives theusermorecontrol thanwaspossible inStab In theoriginalStabsystemhierarchical scripts were hard‐coded in TMKL that itself is coded in the Lisp programming knowledge Thisresulted inseveraldifficulties regarding theusabilityof thesystemFirstly thestoredscripts inStabarenotavailableforreferenceorreviewexceptbyperusingtheactualLispcodeThislackoftransparencycanleadtoinaccurateandimpreciseunderstandingofthestoriesthataregeneratedbyStabasexplainingtheinputdataStab2rsquosStoryEditorprovidesvisualaccesstoboththelibraryofstoredscriptsaswellasthestoriesinstantiatedwhenStab2isrunforaninputdataset
SecondlysinceStabrsquosstoredscriptsarecodedinTMKLitisverydifficultforausertoauthoranewscriptorstoryevaluatethecorrectnessofastoredscriptandmakemodificationstoitStab2StoryEditorwithitsvisualinterface makes these tasks much easier though editing the knowledge elements in a story through theKnowledgeEditorstillisnon‐trivial
FinallytheinitialversionStabdidnotallowmodificationstothestructureofexistingscriptsatruntimeThisimpliesthat it isusefultorunStabjustonceonaninputdatasetHowever ingeneralahumananalystmaywanttorunStabseveraltimesndasheachwithdifferingvariationsofthescripts‐astheanalystrsquosinsightintothecriminal patterns evolves This is perhaps themost important side effect of the use of an interactive StoryEditor in Stab2 Stab2 enables an analyst to conduct ldquowhat ifrdquo simulations Using Stab2rsquos Story Editor andKnowledge Editor illustrated in Figure 7 an analystmay inspectmodify activate or deactivate the scriptsstored in Stab2rsquos Story Library the task and method elements in an individual script or the knowledgeelements likedomainconceptsand relationsTheanalystmay then runStab2 to simulate theeffectsof thepreciseconditionsshecreatedinStab2rsquosvirtualworld
Section34StoryAuthoringinStab2
Whenananalystwants tocreateanewstoryheneeds toopentheeditorandclickonaddstory iconThisopensawhiteareawithapaletteontherightWestartwithpickingtheldquoMainstoryldquoiconfromthepaletteanddropitintothewhiteboardThisrepresentsthenameofthestoryandisdepictedinFigure8(a)AfterthiswedragthebluerectanglethatreadsldquoIntermediaryGoalsrdquoandconnectthistothemainstoryusingaSub‐Goalconnector all ofwhich can be found in the palette The result is shown in Figure 8(b) Once themethod iscreatedwefillitinwithactionsperformedtoachievethatgoalThisisdoneusingtheldquoGoalsActionrdquoiconinthepaletteandthisisshowninFigure8(c)ThedraganddropcapabilitymakesitveryeasyforananalysttoconstructanewstoryplotThetreecangodownasmanylevelsdependingonthecomplexityofthestoryTherelationshipsbetweeneachofthegoalsarecapturedusingthelinksTwokindsoflinkagesarepossiblendashSub‐Goals that suggests that one goal is a child of another and Causal Connection that indicates relationshipsbetweendifferentactionsinsideeachintermediarygoal
Figure8(a)1
stSna
psho
tofStoryAutho
ringinStab2
ndashHerethehu
mananalystdragsth
eicon
mainstoryfrom
thepalette
andplacesitinth
eed
itorareaand
nam
esitasldquostoryplotrdquo
Figure8(b)A
2ndSna
psho
tofStoryAutho
ringin
Stab2
ndashHerethehu
mananalystdragsth
eInterm
ediaryGoalsicon
and
placesitbe
lowth
etopleveltaskAlsotheanalystdragsthesub‐goalslinkagefromthe
paletteand
con
nectsthetw
oblocksand
nam
esthe
bluerectangleas
ldquoMetho
d1rdquo
Figure8(c)ASna
psho
tofStoryAutho
ringinStab2
‐Heretheanalystd
ragsth
eGoalsA
ctionsicon
inside
thealreadycreatedmetho
dHe
then
con
nectstheseactio
nswith
thecausalcon
nectionslinkageinth
epaletteandnameseachactio
n
BehindtheScenes
Figure9(a)A
Sna
psho
tofaXMLfileforeachStoryPlot
Figure9(b)A
Sna
psho
tofaXMLfileforeachStoryPlotndash
Thisim
agehighlightsindividu
alta
gsThe
imageon
the
righth
ighlightsthechildTaskta
gandtheon
ethelefthighlightstheallM
etho
dsta
g
EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)
RelatedWork
ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors
StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)
Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo
Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)
CurrentWork
ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)
Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities
Acknowledgements
Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories
References
AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007
AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8
Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008
BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005
BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005
Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991
ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17
CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985
Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34
CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum
Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)
FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991
Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995
Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009
GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia
Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13
HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)
HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07
JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863
Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009
JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994
Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92
KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999
Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity
LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007
Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180
Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey
MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey
MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691
Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340
Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003
Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81
Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36
PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004
PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005
Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21
RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007
RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318
RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262
RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006
SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924
SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress
SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum
StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132
TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)
ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety
Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623
Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA
WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370
Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics
Figure8(a)1
stSna
psho
tofStoryAutho
ringinStab2
ndashHerethehu
mananalystdragsth
eicon
mainstoryfrom
thepalette
andplacesitinth
eed
itorareaand
nam
esitasldquostoryplotrdquo
Figure8(b)A
2ndSna
psho
tofStoryAutho
ringin
Stab2
ndashHerethehu
mananalystdragsth
eInterm
ediaryGoalsicon
and
placesitbe
lowth
etopleveltaskAlsotheanalystdragsthesub‐goalslinkagefromthe
paletteand
con
nectsthetw
oblocksand
nam
esthe
bluerectangleas
ldquoMetho
d1rdquo
Figure8(c)ASna
psho
tofStoryAutho
ringinStab2
‐Heretheanalystd
ragsth
eGoalsA
ctionsicon
inside
thealreadycreatedmetho
dHe
then
con
nectstheseactio
nswith
thecausalcon
nectionslinkageinth
epaletteandnameseachactio
n
BehindtheScenes
Figure9(a)A
Sna
psho
tofaXMLfileforeachStoryPlot
Figure9(b)A
Sna
psho
tofaXMLfileforeachStoryPlotndash
Thisim
agehighlightsindividu
alta
gsThe
imageon
the
righth
ighlightsthechildTaskta
gandtheon
ethelefthighlightstheallM
etho
dsta
g
EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)
RelatedWork
ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors
StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)
Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo
Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)
CurrentWork
ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)
Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities
Acknowledgements
Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories
References
AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007
AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8
Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008
BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005
BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005
Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991
ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17
CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985
Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34
CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum
Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)
FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991
Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995
Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009
GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia
Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13
HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)
HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07
JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863
Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009
JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994
Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92
KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999
Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity
LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007
Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180
Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey
MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey
MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691
Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340
Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003
Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81
Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36
PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004
PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005
Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21
RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007
RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318
RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262
RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006
SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924
SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress
SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum
StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132
TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)
ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety
Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623
Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA
WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370
Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics
Figure8(b)A
2ndSna
psho
tofStoryAutho
ringin
Stab2
ndashHerethehu
mananalystdragsth
eInterm
ediaryGoalsicon
and
placesitbe
lowth
etopleveltaskAlsotheanalystdragsthesub‐goalslinkagefromthe
paletteand
con
nectsthetw
oblocksand
nam
esthe
bluerectangleas
ldquoMetho
d1rdquo
Figure8(c)ASna
psho
tofStoryAutho
ringinStab2
‐Heretheanalystd
ragsth
eGoalsA
ctionsicon
inside
thealreadycreatedmetho
dHe
then
con
nectstheseactio
nswith
thecausalcon
nectionslinkageinth
epaletteandnameseachactio
n
BehindtheScenes
Figure9(a)A
Sna
psho
tofaXMLfileforeachStoryPlot
Figure9(b)A
Sna
psho
tofaXMLfileforeachStoryPlotndash
Thisim
agehighlightsindividu
alta
gsThe
imageon
the
righth
ighlightsthechildTaskta
gandtheon
ethelefthighlightstheallM
etho
dsta
g
EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)
RelatedWork
ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors
StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)
Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo
Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)
CurrentWork
ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)
Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities
Acknowledgements
Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories
References
AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007
AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8
Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008
BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005
BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005
Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991
ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17
CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985
Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34
CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum
Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)
FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991
Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995
Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009
GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia
Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13
HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)
HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07
JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863
Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009
JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994
Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92
KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999
Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity
LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007
Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180
Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey
MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey
MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691
Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340
Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003
Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81
Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36
PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004
PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005
Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21
RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007
RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318
RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262
RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006
SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924
SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress
SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum
StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132
TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)
ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety
Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623
Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA
WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370
Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics
Figure8(c)ASna
psho
tofStoryAutho
ringinStab2
‐Heretheanalystd
ragsth
eGoalsA
ctionsicon
inside
thealreadycreatedmetho
dHe
then
con
nectstheseactio
nswith
thecausalcon
nectionslinkageinth
epaletteandnameseachactio
n
BehindtheScenes
Figure9(a)A
Sna
psho
tofaXMLfileforeachStoryPlot
Figure9(b)A
Sna
psho
tofaXMLfileforeachStoryPlotndash
Thisim
agehighlightsindividu
alta
gsThe
imageon
the
righth
ighlightsthechildTaskta
gandtheon
ethelefthighlightstheallM
etho
dsta
g
EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)
RelatedWork
ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors
StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)
Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo
Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)
CurrentWork
ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)
Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities
Acknowledgements
Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories
References
AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007
AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8
Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008
BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005
BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005
Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991
ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17
CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985
Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34
CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum
Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)
FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991
Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995
Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009
GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia
Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13
HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)
HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07
JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863
Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009
JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994
Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92
KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999
Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity
LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007
Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180
Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey
MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey
MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691
Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340
Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003
Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81
Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36
PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004
PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005
Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21
RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007
RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318
RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262
RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006
SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924
SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress
SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum
StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132
TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)
ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety
Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623
Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA
WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370
Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics
BehindtheScenes
Figure9(a)A
Sna
psho
tofaXMLfileforeachStoryPlot
Figure9(b)A
Sna
psho
tofaXMLfileforeachStoryPlotndash
Thisim
agehighlightsindividu
alta
gsThe
imageon
the
righth
ighlightsthechildTaskta
gandtheon
ethelefthighlightstheallM
etho
dsta
g
EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)
RelatedWork
ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors
StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)
Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo
Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)
CurrentWork
ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)
Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities
Acknowledgements
Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories
References
AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007
AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8
Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008
BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005
BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005
Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991
ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17
CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985
Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34
CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum
Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)
FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991
Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995
Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009
GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia
Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13
HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)
HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07
JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863
Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009
JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994
Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92
KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999
Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity
LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007
Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180
Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey
MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey
MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691
Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340
Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003
Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81
Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36
PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004
PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005
Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21
RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007
RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318
RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262
RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006
SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924
SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress
SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum
StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132
TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)
ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety
Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623
Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA
WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370
Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics
Figure9(b)A
Sna
psho
tofaXMLfileforeachStoryPlotndash
Thisim
agehighlightsindividu
alta
gsThe
imageon
the
righth
ighlightsthechildTaskta
gandtheon
ethelefthighlightstheallM
etho
dsta
g
EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)
RelatedWork
ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors
StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)
Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo
Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)
CurrentWork
ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)
Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities
Acknowledgements
Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories
References
AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007
AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8
Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008
BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005
BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005
Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991
ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17
CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985
Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34
CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum
Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)
FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991
Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995
Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009
GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia
Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13
HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)
HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07
JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863
Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009
JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994
Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92
KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999
Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity
LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007
Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180
Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey
MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey
MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691
Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340
Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003
Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81
Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36
PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004
PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005
Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21
RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007
RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318
RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262
RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006
SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924
SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress
SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum
StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132
TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)
ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety
Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623
Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA
WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370
Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics
EachofthesestoriescreatedareconvertedintomachineunderstandablelanguageandstoredinTMKLformatasXMLfileswhichisreadandprocessedduringruntimeEachstoryplotlookslikeFigure9(a)aboveAswecanseefromthefigureeachelementcreatedinthestoryhasacorrespondingtagintheXMLdocumentForexamplethemainstoryisstoredasthetopTasktagThexmitypeofthistopTaskismetamodeltopLevelTaskThenameof this tag isgivenas ldquostoryplotrdquo this is thename theanalyst specifiedwhilecreating thestorySimilarly if you see there is allMethods tag created which is of typemetamodelMethod and the name isldquoMethod1rdquothesameasthatmentionedbytheanalystSimilarlythereexistsatagforchildTasksEachGoalAction has its own childTasks tag and attributes like type name and makes One would also notice thetransition tag that has attributes such as prevTask nextTask and helps keep track of causal links betweendifferent actions Each such tag is provided with a machine‐generated id to keep track of the relationsinternally For eg the topTask tag has an xmi id ndashldquo_FXI1YICLEDhelliprdquoThe allMethods tag ends once all itschildTasksandtransitionsinthatmethodhavebeenassignedatagthiscanbeclearlyseenthroughFigure9(b)
RelatedWork
ThenameldquoStabrdquoofour interactiveknowledge‐basedsystemforsensemaking in investigativeanalysiscomesfromSToryABductionAbduction is inferencetothebestexplanationforasetofdata(Bylanderetal1991CharniakampMcDermott 1985 Fischer et al 1991Goel et al 1995 Josephsonamp Josephson 1994)We viewsensemaking in investigativeanalysisasconstructingastorythatexplainsan inputsetofdatabyconnectingeventsthroughstatesandascribinggoalstoactors
StabconstructsastoryforagiveninputdatasetbyretrievingandinstantiatinghierarchicalscriptsstoredinalibraryScriptsintroducedbySchankampAbelson(1977)inAIhavebeenusedinawidevarietyofknowledge‐basedsystemsespeciallyforstoryunderstanding(egCullingford1981Ram1991Mueller2004)
Stabrsquos TMKL knowledge representation language is similar to but more expressive than Hierarchical TaskNetworks (HTNs) (ErolHendlerampNau1994) for knowledge representation (Lee‐Urban2005)TMKL ismoreexpressivethanHTNinpartbecauseTMKLenablesexplicitrepresentationofsubgoalsandmultipleplansforachievingagoalWhenHoangLee‐UrbanandMunoz‐Avila(2005)designedagame‐playingagentinbothTMKLandHTNtheyfoundthatldquoTMKLprovidesconstructsforloopingconditionalexecutionassignmentfunctionswithreturnvaluesandotherfeaturesnotfound inHTNrdquoTheyalsofoundthatsinceHTNimplicitlyprovidessupportforthesamefeaturesldquotranslationfromTMKLtoHTNisalwayspossiblerdquo
Stab2rsquosStoryEditorandKnowledgeEditorenable theauthoringandeditingof stories represented inTMKLInteractive story authoring tools have become common in interactive games Stab2rsquos Story Editor is moresimilar to interactive story authoring tools used in interactive drama (eg Magerko 2005Mateas amp Stern2005RiedlampYoung2006)AlthoughweoriginallydevelopedTMKLtocaptureanagentrsquosself‐modelofitsownknowledgereasoningandarchitecture (MurdockampGoel 2003 2008) it has sincebeenused in several other applications TheAHEADsystem(MurdockAhaampBreslow2003)usesTMKLforrepresentinghypothesesaboutasymmetricthreatsInparticularAHEADusespastcasesofsuchthreats togeneratearguments forandagainstagivenhypothesisTheexplicitspecificationofthegoalsofsubsequencesofactionsallowsAHEADtoretrievepastcasesrelevanttospecificsubsequencesTIELT(MolineauxampAha2005)anenvironmentforevaluatinglearningincomputergamesusesTMKLaspartofitsagentdescriptionlanguageInadifferentprojectwearepresentlyusingTMKLforspecifyingthedesignofgame‐playingagents(Jonesetal2009)
CurrentWork
ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)
Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities
Acknowledgements
Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories
References
AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007
AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8
Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008
BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005
BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005
Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991
ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17
CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985
Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34
CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum
Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)
FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991
Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995
Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009
GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia
Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13
HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)
HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07
JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863
Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009
JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994
Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92
KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999
Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity
LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007
Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180
Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey
MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey
MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691
Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340
Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003
Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81
Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36
PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004
PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005
Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21
RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007
RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318
RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262
RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006
SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924
SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress
SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum
StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132
TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)
ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety
Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623
Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA
WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370
Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics
CurrentWork
ThestoryauthoringandeditingcapabilityofStab2enablesahumananalysttoconductldquowhat‐Ifrdquosimulationswith criminal patterns in order to make sense of the VAST datasets In its present state of developmenthowever Stab2 hasmany limitations Firstly the input events to Stab2 at present are extracted from newstories and represented by hand This is becausewe have been unable to find an automated tool that canextract events from natural language texts with precision and accuracy Secondly the scripts in Stab2 atpresent arenotbasedonany closed setofprimitive actionsor tasksWeareexploring theuseof Schankrsquosprimitiveactions(Schank1983)asthebuildingblocksforourhierarchicalscriptsrepresentedinTMKLThirdlyStab2atpresentdoesnotpropagatethevaluesofvariablesbetweenthenodesinastoryWeareaugmentingTMKLtoenableautomaticvariablepropagationFourthlyStab2atpresentdoesnotexplain itrsquosreasoningorjustifiesitsconclusionsWeareexploringtheuseofTMKLforsupportingself‐explanationinStab2(Goeletal2009RajaampGoel2007)
Stab is one of several ongoing research projects at the Southeast Regional Visual Analytics Center(httpsrvacunccedu) Theseprojects includedesignof interactive techniques and tools for informationvisualization(egStaskoGorgampLiu2008Wangetal2009)constructionofknowledge‐basedtechniquesandtoolsforinformationforaging(egLiuRajaVaidyanath2007Yueetal2009)anddevelopmentofcognitivelyinspiredcomputationalframeworksforvisualanalytics(egChangetal2009GreenRibarskyampFischer2008LiuNersessianampStasko2008Ribarsky FischerampPottenger2009)Ourcurrentworkon theStabproject isfocusing on integrating the interactive knowledge‐based system with Jigsaw (Stasko Gorg amp Liu 2008) aninteractivetoolforvisualizingcomplexrelationshipsamongmultipleentities
Acknowledgements
Wearegrateful to JohnStaskoforhissupportandencouragementof thisworkThis researchhasbenefitedfrommanydiscussionswithhimWethankJeanScholtz JimThomasandKristinCookof theNationalVisualAnalytics Center (NVAC) for their support and encouragementWe also thankWilliam Ribarsky Anita RajaCarstern Gorg and James Foley of the Southeastern Regional Visualization and Analytics Center (SRVAChttpsrvacunccedu)formanydiscussionsonthedesignanddevelopmentofStab2ThedescriptionofStab1inthisreporthasbeenadaptedfromGoeletal2009OurworkontheSTABprojectwassponsoredbyNVACunder the auspices of SRVAC NVAC is a US Department of Homeland Security Program led by the PacificNorthwestNationalLaboratories
References
AdamsSampGoelA(2007a)STABMakingSenseofVASTDataInProc IEEEConferenceonIntelligenceandSecurityInformaticsMay2007
AdamsSampGoelA (2007b)ASTABatMakingSenseofVASTData InProcAAAI‐2007WorkshoponPlanActivityandIntentRecognitionVancouverCanadaJuly2007pp1‐8
Adams S Goel Aamp SugandhN (2008)Using AI for Sensemaking in Intelligence Analysis In Proc SIAMDataminingWorkshoponLinkAnalysisCounterTerrorismandSecurityAtlantaApril2008
BirnbaumLForbusKWagnerEBaker JampWitbrockM (2005)Analogy Intelligent IRandKnowledgeIntegrationforIntelligenceAnalysisInProcAAAISpringSymposiumonAITechnologiesforHomelandSecurityStanfordUniversityMarch2005
BodnarE (2005)MakingSenseofMassiveDatabyHypothesisTesting InProceedingsof2005 InternationalConferenceonIntelligenceAnalysisMay2005
Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991
ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17
CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985
Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34
CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum
Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)
FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991
Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995
Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009
GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia
Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13
HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)
HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07
JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863
Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009
JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994
Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92
KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999
Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity
LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007
Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180
Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey
MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey
MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691
Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340
Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003
Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81
Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36
PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004
PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005
Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21
RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007
RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318
RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262
RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006
SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924
SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress
SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum
StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132
TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)
ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety
Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623
Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA
WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370
Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics
Bylander T Allemang D TannerMamp Josephson J (1991) The Computational Complexity of AbductionArtificialIntelligence49(1‐3)25‐601991
ChangRZiemkiewiczCGreenTampRibarskyW(2009)DefiningInsightforVisualAnalyticsIEEEComputerGraphicsandApplications29(2)14‐17
CharniakEampMcDermottD(1985)IntroductiontoArtificialIntelligenceReadingMAAddison‐Wesley1985
Chen H Schroeder J Hauck R Ridgeway L Atabaksh H Gupta H Boarman C Rasmussen K ampClementsA (2003)CoplinkConnect Informationandknowledgemanagement for lawenforcementCACM46(1)28ndash34
CullingfordR(1981)SAMandMicroSAMInRSchankampCRiesbeck(Eds)InsideComputerUnderstandingFiveProgramsPlusMiniaturesHillsdaleNJErlbaum
Erol K Hendler J amp Nau D (1994) HTN Planning Complexity and Expressivity In Proc Twelfth NationalConferenceonArtificialIntelligence(AAAI‐94)
FischerOGoelASvirbely JampSmith J (1991)TheRoleofEssentialExplanations inAbductionArtificialIntelligenceinMedicine3(1991)181‐1911991
Goel A Josephson J Fischer O amp Sadayappan P (1995) Practical Abduction CharacterizationDecomposition and Distribution Journal of Experimental and Theoretical Artificial Intelligence 7429‐4501995
Goel A Adams A Cutshaw N amp Sugandh N(2009) Playing Detective Using AI for Sensemaking inInvestigativeAnalysisGeorgiaTechGVUTechnicalReport(GIT‐GVU‐09‐09‐03)January2009
GoelAMorseERajaAScholtzJampStaskoJ(2009)IntrospectiveSelf‐ExplanationsforReportGenerationin IntelligenceAnalysis In Proc IJCAI‐09Workshop on Explanation‐Aware Computing July 11‐12 PasadenaCalifornia
Green T Ribarsky W amp Fisher B (2009) Building and Applying a Human Cognition Model for VisualAnalyticsInformationVisualization8(1)1‐13
HoangH Lee‐Urban SampMuntildeoz‐AvilaH (2005)Hierarchical Plan Representations for Encoding StrategicGameAIInProcArtificialIntelligenceandInteractiveDigitalEntertainmentConference(AIIDE‐05)
HeuerR(1999)PsychologyofIntelligenceAnalysisCIACenterfortheStudyofIntelligence
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest06
IEEEVAST2007ContestDatasethttpwwwcsumdeduhcilVASTcontest07
JarvisPLuntPampMyersK(2004)IdentifyingTerroristActivitywithAIPlanRecognitionTechnologyInProcInnovativeApplicationsofAI(IAAI)pp858‐863
Jones J Parnin C Sinharoy A Rugaber S amp Goel A (2009) Adapting Game‐Playing Agents to GameRequirements In Proc Fifth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment(AIIDE‐09)pp148‐153StanfordUniversityCaliforniaUSAOctober2009
JosephsonJampJosephsonS(editors)(1994)AbductiveInferenceComputationPhilosophyandTechnologyCambridgeUniversityPress1994
Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92
KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999
Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity
LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007
Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180
Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey
MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey
MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691
Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340
Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003
Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81
Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36
PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004
PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005
Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21
RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007
RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318
RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262
RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006
SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924
SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress
SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum
StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132
TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)
ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety
Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623
Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA
WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370
Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics
Klein GMoon B amp Hoffman R (2006)Making sense of sensemaking II Amacrocognitivemodel IEEEIntelligentSystems21(5)88‐92
KrizanL(1999)IntelligenceEssentialsforEveryoneJointMilitaryIntelligenceCollege1999
Lee‐UrbanS(2005)TMKModelstoHTNsTranslatingProcessModelsintoHierarchicalTaskNetworksMSThesisDepartmentofComputerScienceLehighUniversity
LiuD Raja AampVaidyanath J (2007) TIBORAResource‐bounded Information ForagingAgent forVisualAnalyticsInProceedings2007IEEEWICACMInternationalConferenceonIntelligentAgentTechnology(IAT2007)SiliconValleyCANov2‐52007
Liu LNersessianNamp Stasko J (2008)Distributed Cognition as a Theoretical Framework for InformationVisualizationIEEETransactionsonVisualizationandComputerGraphicsVol14No6NovemberDecember2008pp1173‐1180
Magerko B (2005) Story representation and interactive drama In Proc 1st Artificial Intelligence andInteractiveDigitalEntertainmentConference(AIIDE05)MarinadelRey
MateasMampSternA(2005)StructuringcontentintheFaccediladeinteractivedramaarchitectureInProcs1stArtificialIntelligenceandInteractiveDigitalEntertainment(AIIDE05)MarinadelRey
MolineauxMampAhaD(2005)TIELTATestbedforGamingEnvironmentsInProc2005NationalConferenceonAI(AAAI2005)pp1690‐1691
Mueller E (2004) Understanding Script‐Based Stories using Commonsense Reasoning Cognitive SystemsResearch5(4)307‐340
Murdock J Aha D amp Breslow L (2003) Assessing Elaborated Hypotheses An Interpretive Case‐BasedReasoning Approach In Proc Fifth International Conference on Case‐Based Reasoning Trondheim NorwayJune2003
Murdock J amp Goel A (2003) Localizing Planning using Functional Process Models In Proc InternationalConferenceonAutomatedPlanningandScheduling(ICAPS‐03)June2003pp73‐81
Murdock J amp Goel A (2008) Meta‐Case‐Based Reasoning Self‐Improvement through Self‐UnderstandingJournalofExperimentalandTheoreticalArtificialIntelligence20(1)1‐36
PARCAITeamampHeuerR (2004)ACHATool forAnalyzingCompetingHypothesesPARCTechnicalReportOctober2004
PirolliPampCardS(2005)TheSensemakingProcessandLeveragePointsforAnalystTechnologyasIdentifiedThroughCognitiveTaskAnalysisInProceedingsof2005InternationalConferenceonIntelligenceAnalysisMay2005
Plaisant CGrinsteinG Scholtz JWhitingMOConnell T Laskowski S Chien L Tat AWrightWGorgCLiuZParekhNSinghalKStaskoJ(January2008)EvaluatingVisualAnalyticsThe2007VisualAnalyticsScienceandTechnologySymposiumContestIEEEComputerGraphicsandApplicationsVol28No2MarchApril2008pp12‐21
RajaAampGoelA(2007)IntrospectiveSelf‐ExplanationinAnalyticalAgentsInProceedingsofAAMAS2007WorkshoponMetareasoninginAgent‐basedSystemspp76‐91HawaiiMay2007
RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318
RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262
RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006
SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924
SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress
SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum
StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132
TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)
ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety
Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623
Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA
WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370
Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics
RamA(1991)Atheoryofquestionsandquestionasking1113160TheJournaloftheLearningSciences1(3‐4)273‐318
RibarskyWFisherBampPottengerW(2009)TheScienceofAnalyticalReasoningInformationVisualization8(4)252‐262
RiedlMampYoungR(FromlinearstorygenerationtobranchingstorygraphsIEEEComputerGraphicsandApplicationsSpecialIssueonInteractiveNarrative26(3)2006
SanfilippoACowellATratzSBoekACowellAPosseCampPouchardL (2007)ContentAnalysis forProactiveIntelligenceMarshalingFrameEvidenceInProcAAAI2007pp919‐924
SchankR(1983)DynamicMemoryATheoryofRemindingandLearninginComputersandPeopleCambridgeUniversityPress
SchankRampAbelsonR (1977)Scripts PlansGoalsandUnderstandingAn Inquiry intoHumanKnowledgeStructuresHillsdaleNJErlbaum
StaskoJGorgCampLiuZ(2008)JigsawSupportingInvestigativeAnalysisthroughInteractiveVisualizationInformationVisualizationVol7No2Summer2008pp118‐132
TecuciGBoicuMMarcuDBoicuCampBarbulescuM (July2008)Disciple‐LTALearningTutoringandAnalyticAssistanceJournalofIntelligenceCommunityResearchandDevelopment(JICRD)
ThomasJampCookK(2005)IlluminatingthePathIEEEComputerSociety
Wang X Jeong D Dou W Lee S Ribarsky W amp Chang R (2009) Defining and applying knowledgeconversionprocessestoavisualanalyticssystemComputersampGraphics33(5)616‐623
Welty C Murdock J Pinheiro da Silva P McGuinness D Ferrucci D amp Fikes R (2005) TrackingInformationExtraction from IntelligenceDocuments InProceedingsof the2005 InternationalConferenceonIntelligenceAnalysisMcLeanVA
WhitakerESimpsonRBurkhartLMacTavishRampLobbC(2004)ReusingIntelligenceAnalystsSearchPlansInProcAnnualMeetingoftheHumanFactorsandErgonomicspp367‐370
Yue J Raja A Liu DWang X amp RibarskyW (2009) A Blackboard‐based Approach towards PredictiveAnalyticsInProcAAAISpringSymposiumonTechnosocialPredictiveAnalytics