1

Objectives Develop new methods to fuse hard sensor data

Develop understanding of hybrid human

cognition and automated processing for

improved situational awareness

Establish test and evaluation approach and

associated data sets

Create an integration environment for team

collaboration, T&E, and transition

Scientific/Technical Approach Establish a framework for human-centric fusion

Develop a T&E approach progressing from

synthetic data to human in the loop experiments

Create an architecture and infrastructure for

algorithm integration and transition

Design and implement algorithms for fusion of

physical sensor data including new sensor types

Accomplishments Evolved SYNCOIN: new meta-data/sensor data

Generated TML for 4th year demonstrations

Performed new hard sensor data processing

Developed data visualization toolkit

Conducted concept/cognitive analysis

Developed automated inference tools including

Complex Event Processing & Intelligent Agents

Demonstrated utilization of SYNCOIN data in

AXISPro

ta

PSU Overview & SYNCOIN David Hall, Jake Graham, Jeff Rimland, Rick Tutwiler

and Guoray Cai

PSU/UB Sensemaking Architecture

Hard sensor fusion

Visual Analytics Workbench

Cognitive Task Analysis

Complex Event Integration

3

Text Entities

Activities

Locations

Attributes

Human

Observers

Physical

Sensors

Computer

Aided Human

Encoding (HE)

Defined fuzzy

relational operations

Fused Hard

Data (Entities)

Co

mp

ute

r a

ssis

ted

Hu

ma

n m

eta

-da

ta g

en

era

tio

n

RDF-like

triples

Auto Correlation

(ID/loc)

Me

ta-D

ata

Ba

se

Visual Analytics Workbench Hard Sensor Fusion

Analyst input

PIRs

RFIs

I&W

Working

Hypotheses

etc

1. 1.

2.

3.

4.

PSU Processing Flow and

Functions

Overview of PSU 4rd Year

Accomplishments

1. Synthetic hard/soft data set Continued evolution of the SYNCOIN data set

New physical sensor data

New soft and hard sensor data links and meta-data

Conducted human in the loop cognitive task analyses

2. Hard sensor data fusion New algorithms for fusion of hard sensor data

Range imaging tracking, (Interacting Multiple Mode (IMM) Kalman Filters) Tracking

VNIR Color Particle Filter Tracking

VNIR Image fusion and Multi-Model Object characterization

Range/Depth Automated Segmentation Algorithm

3. New automated inference tools Intelligent agents for improved focus of analyst attention

Complex event processing

4. Visualization toolkit Implemented web-based interactive visual analysis (IVA) toolkit

Created relational database to link SYNCOIN geo, temporal and human network data

5. Integration & network based processing Robust cyber-infrastructure for distributed H/S processing

StreamBase CEP, Advanced message queuing protocol (AMQP), RabbitMQ, Open Geospatial Consortium standards for

TML and Event Pattern Markup Language, AchemyAPL, RDF/OWL

Demonstrated SYNCOIN data in AXISPro

Students supported/Degrees Awarded: - 6 graduates/undergraduate students: J. Rimland, D. Chen, R. Grace, A. Godbole, M. Lesniewski, A. Sridara

- 3 Undergraduate students: J. Shields, B. Ripka and G. Traylor

- 4 faculty: D. Hall, J. Graham, M. McNeese, R. Tutwiler, and G. Cai

- Degrees awarded: (MS, PhD): A. Godbole (M.S.) and J. Rimland (PhD)

Publications: - Refereed conference papers 10

- Book and book chapters - 3

- Technical reports 1

- Presentations - 10

Technology Transitions:

- Interactions with industry Distributed SYNCOIN to 11 organizations and individuals

Raytheon Corporation

QuinetiQ (UK)

Aptima

VIStology INC

Saffron Technology

Arctan

Overwatch

- Interactions with other government agencies Network Science Collaborative Technology Alliance (U. of Illinois)

Institute for Defense Analysis

Evaluation of Techniques for Uncertainty Representation Working Group (ISIF)

Project Statistics and Summary

5

SYNCOIN

Conceptual Framework Synthetic soft-sensor database consistent with dominant security environment of Iraq 2010

COIN Inspired story line

Focus on people, events, locations and movements in and around Bagdad

Soft data augmented with physical sensor data collected at Pleasant Gap Fire Safety Facility

Multiple threads and storylines

Ground-truth specification of people, events, activities, locations, timeline

Approach Interpretation of ethno-religious groups, culture and political landscape & interactions with allied forces during OIF

Military aspects of COIN domain (IED events, support networks and motivations)

Realistic without revealing tactical or operational tradecraft

Data augmented with meta-data obtained using students acting in analyst teams

Example Sensor Exercise

Scenario/Theme: IED Attack / Coordinated Sniper Fire

Participants: 21 (PSU+TSU)/ 3 vehicles

Event Days: 3-5

Sensors: 9+ Cameras, 1 Flash LIDAR, 2+ KINECT

Mobile Devices: GeoSuite Mobile App on Android

Event/Activity Synchronization: auditory/visual cues

Each Data Collect requires several weeks of planning/coordination;

3-5 days of data collection; followed by multiple weeks of

processing equating to over .5 TB of data

Cognitive Task Analyses and Meta-

Data Generation

SYNCOIN scene setter

Background assumptions

Command-level tasking

SYNCOIN message set

Derived PIRs and anticipated I&W for Collection

Sequential Processing and Analysis of SYNCOIN messages

Interim Assessment

Tentative Working Hypotheses

Predicted Threat Activity Reports

Evaluation of Observed (Actual) Indicators

Revision and update of Predicted Threat Activity Reports

Meta-data for Enhancement of SYNCOIN Data Set

Student Teams Acting in Analysis Role

9

Hard/Soft Fusion Demonstration

Peter Willet, University of Connecticut

Gavin Powell, ADS Innovation Works, UK, government technical area lead for TA 6 - Distributed Coalition Information Processing

for Decision-Making

David Nicholson, BAE Systems, London, UK

David Dearing, Stottler Henke Associates

David Braines, Hursley Emerging Technology Services

Erick Blasch, Air Force Research Laboratory Sensors Directorate (AFRL/SNAA)

Marco Pravia, BAE Systems

Kamal Premaratne, University of Miami

James Law, SPAWARSYSCEN U. S. Navy Space and Naval Warfare Systems Center

Chase Cotton, Network Science Collaborative Technology Alliance Program (CTA), U. S. Army Research Laboratory

ETURWG Evaluation of Techniques for Uncertainty Representation Working Group, International Society of Information Fusion

(ISIF)

International Technology Alliance

Brian Simpson, Raytheon Corporation

Simon Maskell, QinetiQ, UK

Charlotte Shabarkh, Aptima, Woburn, MA

Brian Ulicny, VIStology, INC, Framingham, MA

Dr. Joan Carter, Institute for Defense Analysis, Alexandria, VA

Network Science Collaborative Technology Alliance, University of Illinois, Champaign, IL

Jim Fleming, Saffron Technology, Cary, NC

Charles Morefield, Arctan, Arlington, VA

Rick Beckett, Overwatch, Textron, Philadelphia, PA

Distribution of SYNCOIN Data

11

Proposed Plans for Year 5

Hard sensor data fusion Complete automated classification of human forms from 2-D and 3-D depth map fused data

Design & implement a multi- -scene characterization)

Final data collect using KINECT merged color tracker and depth map tracker

Automated sense-making algorithms Apply and extend Complex Event Processing/Multi-Agent Systems to SYNCOIN data

Develop coherence network algorithm for SYNCOIN

Link Complex Event Processing/Multi-Agent Systems and Coherence Network processing

Visual analytics Extend current toolkit (e.g., text analysis, recommender support, etc.)

Enable collaborative analysis with multiple distributed analysts

Integrate visual interaction (human-supported cognition) with computer automated processing

Extend cognitive task analysis to improve understanding of analytical reasoning processes

Field a beta version of the toolkit and conduct human in the loop experiments