Post on 25-Dec-2015
Video Analysis
Content Extraction
1
Dennis Moellman, VACE Program Manager
VACE Executive Brief
for MLMI
Video Analysis
Content Extraction
2
Briefing Outline
• Introduction • Phase II• Evaluation• Technology Transfer• Phase III• Conclusion
Video Analysis
Content Extraction
3
Introduction
• ARDA – Advanced Research and Development Activity– A high-risk/high-payoff R&D effort sponsored by US
DoD/IC
• ARDA taking a new identity– In FY2007 under the DNI
• Report to: ADNI(S&T)
– Renamed: Disruptive Technology Office
• VACE – Video Analysis and Content Extraction– Three Phase initiative begun in 2000 and ending 2009
• Winding down Phase II• Entering into Phase III
What is ARDA/DTO/VACE?
Video Analysis
Content Extraction
4
Context
• Problem Creation:– Video is an ever expanding source of imagery and open source
intelligence such that it commands a place in the all-source analysis.
• Research Problem:– Lack of robust software automation tools to assist human
analysts:• Human operators are required to manually monitor video signals• Human intervention is required to annotate video for indexing
purposes• Content based routing based on automated processing is lacking• Flexible ad hoc search and browsing tools do not exist
• Video Extent:– Broadcast News; Surveillance; UAV; Meetings; and
Ground Reconnaissance
Video Exploitation Barriers
Video Analysis
Content Extraction
5
Research Approach
• Research Objectives:– Basic technology breakthroughs– Video analysis system components– Video analysis systems– Formal evaluations: procedures, metrics and data sets
• Evaluate Success:– Quantitative Testing
Metric Current NeedAccuracy <Human >>HumanSpeed >Real time <<Real time
– Technology Transition• Over 70 technologies identified as deliverables• 50% have been delivered to the government• Over 20 undergoing government evaluation
Video Exploitation
Video Analysis
Content Extraction
6
Management Approach
Management Philosophy – NABC
• N – Need
• A – Approach
• B – Benefit• C – Competition
Geared for Success
Video Analysis
Content Extraction
7
Language/User Technology
Visualization
EnhancementFilters
Source Video
RecognitionEngine
UnderstandingEngine
ExtractionEngine
Reference
01101010101011111010010101101101010101
0110100110
01101011Intelligent
Content Services
Concept Applications
Interests
System View
Video Analysis
Content Extraction
8
Motion Analysis
Evaluation
Inte
llige
nt
Cont
ent
Serv
ices
Summarization
Video Browsing
Content-based RoutingAdvanced query/retrieval using Q&A technologies
Video Monitoring
Video Mining
Indexing
Change Detection
Con
tent
Extr
action Object Recognition
Mensuration
Scene Modeling
Event Recognition
Object Modeling
Simple Event Detection
Event Understanding
Complex Event Detection
Object/Scene ClassificationObject Detection & Tracking
Enab
ling
Tech
nol
ogie
s
Image Enhancement/Stabilization
Multi-modal fusion
Event Ontology
Event Expression Language
Camera Parameter Estimation
Automated Annotation Language
Integrity Analysis
Phase 2 Phase 3Phase 1 Future
Motion Analysis
Filtering
VACE Interests
Technology Roadmap
Video Analysis
Content Extraction
9
Funding
FY06 Allocations FY07 Allocations4%
12%
20%
64%36%
39%
4%
10%
11%
Commitment to Success
Video Analysis
Content Extraction
10
Phase II
• Researcher Involvement: – Fourteen contracts– Researchers represent a cross section of industry
and academia throughout the U.S. partnering to reach a common goal
• Government Involvement: – Tap technical experts, analysts and COTRs from
DoD/IC agencies– Each agency is represented on the VACE
Advisory Committee, an advisory group to the ARDA/DTO Program Manager
Programmatics
Video Analysis
Content Extraction
11
Phase II
Demographics
Virage
MIT
ColumbiaUniv.
Univ. of Maryland (2)
AFIT
Univ. of Illinois-Urbana-Champaign (2)
PurdueUniv.
Univ. ofChicago
Univ. ofWashington
SRI
Univ. of Southern
California / Info.Science Inst.
Univ. of Central Florida
GeorgiaInst. Of Tech.
TelcordiaTechnologies
Sub Contractors (14)
Univ. of SouthernCalifornia
IBM T. J.Watson Center
BoeingPhantom
Works
CarnegieMellon
Univ. (2) (Robotics Inst.)
(Informedia) .
Wright StateUniv.
TASC
Sarnoff Corp (2)
BBN
Salient Stills
Alphatech
Univ. of Illinois-Urbana-Champaign
Prime Contractors (14)
Univ. of Maryland
Video Analysis
Content Extraction
12
Phase II
ProjectsTitle Organization Principal Investigator
Foreign Broadcast News Exploitation
ENVIE: Extensible News Video Information
Exploitation
Carnegie Mellon University
Howard Wactlar
Reconstructing and Mining of Semantic Threads Across Multiple Video Broadcast
News Sources using Multi-Level Concept Modeling
IBM T.J. Watson Research Center / Columbia Univ.
John Smith /
Prof. Shih-Fu Chang
Formal and Informal Meetings
From Video to Information: Cross-Modal Analysis of
Planning Meetings
Wright State University
VaTech/AFIT / Univ. of Chicago / Purdue
Univ. / Univ. of Illinois-Urbana-Champaign
Francis Quek / Ronald Tuttle /
David McNeill & Bennett Bertenthal / Thomas Huang / Mary Harper
Event Recognition from Video of Formal and Informal
Meetings Using Behavioral Models and Multi-modal
Events
BAE Systems/ MIT / Univ. of Maryland /
Virage
Victor Tom/ William Freeman & John Fisher /
Yaser Yacoob & Larry Davis / Andy Merlino
Video Analysis
Content Extraction
13
Phase II
Projects
Title Organization Principal Investigator
Abstraction and Inference about Surveillance Activities
Video Event Awareness Sarnoff Corporation /
Telcordia Technologies Rafael Alonso /
Dimitrios Georgakopoulos
Integrated Research on Visual Surveillance
University of Maryland Larry Davis,
Yiannis Aloimonos & Rama Chellappa
UAV Motion Imagery
Adaptive Video Processing for Enhanced Object and Event Recognition in UAV Imagery
Boeing Phantom Works (Descoped to UAV data
collection)Robert Higgins
Task and Event Driven Compression (TEDC) for UAV
Video Sarnoff Corporation Hui Cheng
Ground Reconnaissance Video
Content and Event Extraction from Ground Reconnaissance
Video
TASC, Inc. / Univ. of Central Florida / Univ. of
California-Irvine
Sadiye Guler / Mubarak Shah / Ramesh Jain
Video Analysis
Content Extraction
14
Phase II
Projects
Title Organization Principal Investigator
Cross Cutting / Enabling Technologies
Probabilistic Graphical Model Based Tools For
Video Analysis
University of Illinois, Urbana - Champaign
Thomas Huang
Automatic Video Resolution Enhancement
Salient Stills, Inc. John Jeffrey Hunter
Robust Coarse-to-Fine Object Recognition in
Video
CMU/Pittsburgh Pattern Recognition
Henry Schneiderman & Tsuhan Chen
Multi-Lingual Video OCR BBN Technologies /
SRI International John Makhoul / Greg
Myers
Model-based Object and Video Event Recognition
USC - Institute for Robotics and
Intelligent Systems / USC - Information Sciences Institute
Ram Nevatia, Gerard Medioni & Isaac Cohen /
Jerry Hobbs
Video Analysis
Content Extraction
15
Evaluation
• Programmatic:– Inform ARDA/DTO management of
progress/challenges
• Developmental:– Speed progress via iterative self testing– Enable research and evaluation via essential data and
tools – build lasting resources
• Key is selecting the right tasks and metrics– Gear evaluation tasks to research suite– Collect data to support all research
Goals
Video Analysis
Content Extraction
17
Evaluation
NIST Process
Formal Evaluation
Determine Sponsor Requirements
Assess required/existing
resources
Develop detailed plans with
researcher input
Dry-Run shakedown
Technical Workshopsand reports
Recommendations
PlanningPlanning ProductsProducts ResultsResultsEvaluation Plan
Task Definitions
Protocols/Metrics
Rollout Schedule
Data Identification
Evaluation Resources
Training Data
Development Data
Evaluation Data
Ground Truth and other metadata
Scoring and Truthing Tools
Video Analysis
Content Extraction
18
Evaluation
NIST Mechanics
Video Data
Algorithms
Evaluation
Results
Annotation
System Output
Ground Truth
Measures
Video Analysis
Content Extraction
19
Evaluation
2005-2006 Evaluations
Evaluation TypeEvaluation TypeDetectionDetection TrackingTracking RecognitionRecognition
DomainDomain PP FF VV TT PP FF VV TT PP FF VV TT
Meeting RoomMeeting Room xx xx xx xx
Broadcast NewsBroadcast News xx xx xx xx xx xx xx xx xx
UAVUAV xx xx xx xx
SurveillanceSurveillance xx xx xx xx xx xx
Ground ReconGround Recon
P = Person; F = Face; V = Vehicle; T = Text
Video Analysis
Content Extraction
20
Evaluation
• Evaluation Metrics:– Detection: SFDA (Sequence Frame Detection
Accuracy) • Metric for determining the accuracy of a detection
algorithm with respect to space, time, and the number of objects
– Tracking: STDA (Sequence Tracking Detection Accuracy)
• Metric for determining detection accuracy along with the ability of a system to assign and track the ID of an object across frames
– Text Recognition: WER (Word Error Rate) and CER (Character Error Rate)
• In-scene and overlay text in video
• Focused Diagnostic Metrics (11)
Quantitative Metrics
Video Analysis
Content Extraction
21
Evaluation
Phase II Best Results
Spatial Detection and Tracking Error
0%
10%
20%
30%
40%
50%
60%
70%
80%
Text(Broacast
News)
Face(Broacast
News)
Face(MeetingRoom)
Hand(MeetingRoom)
Person(MeetingRoom)
MovingVehicle(UAV)
Evaluation Tasks
Per
cen
t E
rro
r
Detection
Tracking
Video Analysis
Content Extraction
22
Evaluation
PPATT UIUC UCF0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
(20
%T
) S
FD
A
Boxplot of (20%T) SFDA scores: Face in BNews
Sites
Face Detection: BNews (Score Distribution)
Video Analysis
Content Extraction
23
Evaluation
COLIB SRI_1 SRI_2 CMU_L0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
(20
%T
) S
FD
A
Sites
Boxplot of (20%T) SFDA Scores: EngText in BNews
Text Detection: BNews (SFDA Score distribution)
Video Analysis
Content Extraction
24
Evaluation
• Benefit of open evaluations– Knowledge about others’ capabilities and community feedback– increased competition -> progress
• Benefit of evaluation workshops– Encourage peer review and information exchange, minimize
“wheel reinvention”, focus research on common problems, venue for publication
• Current VACE-related open evaluations– VACE: Core Evaluations – CLEAR: Classification of Events, Activities, and Relationships – RT: Rich Transcription– TRECVID: Text Retrieval Conference Video Track– ETISEO: Evaluation du Traitment et de l’Interpretation de
Sequences Video
Open Evaluations and Workshops -- International
Video Analysis
Content Extraction
25
Evaluation
Expanded Source Data
Task Sub-conditionConference
MeetingsSeminar Meetings Surveillance
Broadcast News
Studio Poses UAV
3D Sing Per Track Video CHIL
Audio CHIL
Audio+Video CHIL
2D Multi Per Detect VACE VACE
2D Multi Per Track VACE CHIL VACE VACE
Person ID Video CHIL
Audio CHIL
Audio + Video CHIL
2D Face Det VACE VACE
2D Face Track VACE CHIL VACE
Head Pose Est CHIL CHIL
Hand Detect VACE
Hand Track VACE VACE
Text Detect VACE
Text Track VACE
Text Recognition VACE
Vehicle Detect VACE VACE
Vehicle Track VACE VACE
Feature Extract TRECVID
Shot Boundary Detect TRECVID
Search TRECVID
Acoustic Event Detection CHIL
Environment Class CHIL
Video Analysis
Content Extraction
26
Evaluation
2005 2006 2007 2008A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D
Evaluation Schedule VACE Phase 2 VACE Phase 3TRECVID CLEAR RT
NIST/DTO VACE
VACE Phase 2 Core
Evaluation Series 1 (Text, Face in Meeting, BNews)
Evaluation Series 2 (Hand in Meeting, Vehicle in UAV)
Evaluation Series 3 (Person in Meeting)
Evaluation Series 4 (Person/Vehicle in Surveillance)
VACE Phase 3 Core
Evaluation Series 5, 6, 7, 8
TRECVID
(Search and Extraction)
CLEAR
(Person/Face/Pose in Meeing)
(Extraction/Localiazation)
Rich Transcription
(English/Arabic Text in BNews)
(Multi-Modal Recognition)
ETISEO
4/30 6/30
5/96/6 11/8
Report Results
8/249/23 11/8
Report Results
9/810/3 11/8
Report Results
6/28
Report Results
2/23/2 4/5
Report Results
8/2 9/610/16
Report Results
2/43/114/22
Report Results
7/148/2310/2
Report Results
4/12 9/3011/14
Report Results
4/7 9/20 11/13
Report Results
4/6 9/20 11/13
Report Results
4/7 9/19 11/11
Report Results
2/1 3/94/8
Report Results
3/14/25/11
Report Results
3/34/4 5/9
Report Results
5/4
Report Results
4/306/1 7/5
Report Results
4/306/67/9
Report Results
1/2
Cycle 1
5/1 5/8
Cycle 2
11/17 12/8
Seminar
Schedule
Video Analysis
Content Extraction
27
TECH TRANSFER
Purpose: Move technology from lab to operation• Technology Readiness Activity
– An independent repository for test and assessment– Migrate technology out of lab environment– Assess technology maturity – Provide recommendations to DTO and researchers
DTO Test and Assessment Activities
Video Analysis
Content Extraction
28
Level Definitions Entry Condition Contractor Activity
1 Basic principles observed and reported Some peer review of ideas Reporting on basic idea
2 Technology concept and/or application formulated
Target applications are proposed Speculative work; invention
3 Analytical and experimental critical function and/or characteristic proof of concept
Algorithms run in contractor labs and basic testing is possible (internal, some external may be possible)
Doing Analytical studies with weakly integrated components
4 Component/breadboard validation in lab Proof of concept exists; test plans exist; external testing is possible
Low fidelity integration of components
5 Component/breadboard validation in relevant environment
Integrated system functions outside contractor lab; some TRA tests completed
Working with realistic situations
6 System/subsystem model or prototype in demonstration in relevant environment
IC/DoD users identified; target environment defined; simulated testing possible
Demonstrating engineering (software qualities) feasibility
7 System prototype demo in operational environment
Test lab trials in simulated environment completed; installed in operational environment
Completing the product
8 Actual system completed and qualified through test and demonstration
Product completed; Test lab trial completed successfully
Releasing the product; Repairing minor bugs
9 Actual system proven through successful mission operations
Proven value-added in an operational environment
Repairing minor bugs; noting proven operational results
DoD Technology Readiness Levels (TRL)
TECH TRANSFER
Video Analysis
Content Extraction
29
DTOControl
DTOInfluence
DOD Technology Risk Scale
HIGH LOWRISK
Technology Transfer
Use in assessing project’s• Technology maturity• Risk level• Commercialization potential
6 7
8 9
4 5
1 2 3
IC/DODTest
Facility(s)
Info-XTest
FacilityContractorTest
Facility
UNCLASSIFIED
UNCLASSIFIEDCLASSIFIED
UNCLASSIFIEDCLASSIFIED
Production
Applying TRL
Video Analysis
Content Extraction
30
Technology Transfer
TRL Assesment
0
1
2
3
45
6
7
8
9
Ba
ckg
rou
nd
Su
btr
act
ion
Sta
bili
zatio
n
Ob
ject
Tra
ckin
g_
Sta
tCa
me
ra
Imp
ort
an
ceM
ap
Ge
ne
rato
r
Mu
ltila
yerM
osa
icG
en
era
tor
Eve
ntE
stim
atio
n
Ma
rve
l
Info
rme
dia
ST
AR
T
Gu
time
PIQ
UA
NT
-II
MP
SG
Op
inio
nF
ind
er
Kin
evi
s
Ge
oT
ime
TR
L
Assesment prior to delivery
Current assesment
Projected at end of Contract
TRA Maturity Assessments
Video Analysis
Content Extraction
31
Phase III BAA
• Contracting Agency: DOI, Ft. Huachuca, AZ– DOI provides COR– ARDA/DTO retain DoD/IC agency COTR’s and add more
• Currently in Proposal Review Process– Span 3 FY’s and 4 CY’s– Remains open thru 6/30/08
• Funding objective: $30M over program life– Anticipate to grow in FY07 and beyond
• Address the same data source domains as Phase II• Will conduct formal evaluations• Will conduct maturity evaluations and tech transfer
Programmatics
Video Analysis
Content Extraction
32
Phase III BAA
• Emphasis on technology and system approach– Move up technology path where applicable– Stress ubiquity
• Divided into two tiers:– Tier 1: One year base with option year
• Technology focus• Open to all – US and international• More awards for lesser funding
– Tier 2: Two year base with option year(s)• Comprehensive component/system level initiative• Must be US prime• Fewer awards for greater funding
Programmatics
Video Analysis
Content Extraction
33
Phase III BAA
Schedule2005 2006
Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct
Planning Schedule Milestone Major Task
Activity
Scope Planning
Planning Group Meetings
BAA RFP
First Draft
Second Draft
Third Draft
Final Draft
Contracting Final Review
BAA
Announcement
Questions/Proposals Due
Proposal Evaluation
Source Selection Meeting
Funding Recommendations
Contract Awards
Phase 3 Kickoff Workshop
6/14 6/30 8/31
8/2 9/27 10/18 11/7
DOI
11/22
7/18 2/2
7/18 8/2
Six weeks after kickoff
9/1 9/30
Six weeks after first draft
10/10 11/4
Completed prior to Fall Workshop
11/21 12/2
Sent to Contracting Office
12/2
Draft Final BAA
12/16 1/8 1/27
Final BAA
12/15 6/30
12/15
Draft BAA Posting
1/20
Brief
2/1
Final BAA Posting
1/6
Questions/Comments
3/3
Proposals
3/6 4/144/25
4/26-27
4/28
5/1 6/30
8/9-10
Video Analysis
Content Extraction
34
Summary
• VACE is interested in:– Solving real problems with risky, radical
approaches– Processing multiple data domains and multimodal
data domains– Developing technology point solutions as well as
component/system solutions– Evaluating technology process– Transferring technology into user’s space
Take-Aways
Video Analysis
Content Extraction
35
Conclusion
• Invitations:– Welcome to participate in VACE Phase III– Welcome to participate in VACE Phase III Evaluations
Potential DTO Collaboration
Video Analysis
Content Extraction
36
Contacts
Dennis Moellman, Program Manager
Phones: 202-231-4453 (Dennis Moellman) 443-479-4365 (Paul Matthews) 301-688-7092
(DTO Office)800-276-3747 (DTO
Office)
FAX: 202-231-4242 (Dennis Moellman)301-688-7410 (DTO
Office)
E-Mail: dennis.moellman@dia.mil (Internet Mail) pmmatth@nsa.gov
Location: Room 12A69 NBP #1Suite 6644
9800 Savage RoadFort Meade, MD 20755-6644