Dennis Moellman, VACE Program Manager
description
Transcript of Dennis Moellman, VACE Program Manager
Video AnalysisContent Extraction
1
Dennis Moellman, VACE Program Manager
VACE Executive Brief
for MLMI
Video AnalysisContent Extraction
2
Briefing Outline
• Introduction • Phase II• Evaluation• Technology Transfer• Phase III• Conclusion
Video AnalysisContent Extraction
3
Introduction
• ARDA – Advanced Research and Development Activity– A high-risk/high-payoff R&D effort sponsored by US
DoD/IC• ARDA taking a new identity
– In FY2007 under the DNI• Report to: ADNI(S&T)
– Renamed: Disruptive Technology Office• VACE – Video Analysis and Content Extraction
– Three Phase initiative begun in 2000 and ending 2009• Winding down Phase II• Entering into Phase III
What is ARDA/DTO/VACE?
Video AnalysisContent Extraction
4
Context
• Problem Creation:– Video is an ever expanding source of imagery and open source
intelligence such that it commands a place in the all-source analysis.
• Research Problem:– Lack of robust software automation tools to assist human
analysts:• Human operators are required to manually monitor video signals• Human intervention is required to annotate video for indexing
purposes• Content based routing based on automated processing is lacking• Flexible ad hoc search and browsing tools do not exist
• Video Extent:– Broadcast News; Surveillance; UAV; Meetings; and
Ground Reconnaissance
Video Exploitation Barriers
Video AnalysisContent Extraction
5
Research Approach
• Research Objectives:– Basic technology breakthroughs– Video analysis system components– Video analysis systems– Formal evaluations: procedures, metrics and data sets
• Evaluate Success:– Quantitative Testing
Metric Current NeedAccuracy <Human >>HumanSpeed >Real time <<Real time
– Technology Transition• Over 70 technologies identified as deliverables• 50% have been delivered to the government• Over 20 undergoing government evaluation
Video Exploitation
Video AnalysisContent Extraction
6
Management Approach
Management Philosophy – NABC
• N – Need • A – Approach • B – Benefit• C – Competition
Geared for Success
Video AnalysisContent Extraction
7
Language/User Technology
Visualization
EnhancementFilters
Source Video
RecognitionEngine
UnderstandingEngine
ExtractionEngine
Reference
01101010101011111010010101101101010101
0110100110
01101011Intelligent
Content Services
Concept Applications
InterestsSystem View
Video AnalysisContent Extraction
8
Motion Analysis
Evaluation
Inte
llige
nt
Cont
ent
Serv
ices
SummarizationVideo Browsing
Content-based RoutingAdvanced query/retrieval using Q&A technologies
Video Monitoring
Video Mining
Indexing
Change Detection
Cont
ent
Extra
ctio
n Object Recognition
MensurationScene Modeling
Event Recognition
Object ModelingSimple Event Detection
Event UnderstandingComplex Event Detection
Object/Scene ClassificationObject Detection & Tracking
Enab
ling
Tech
nolo
gie
s
Image Enhancement/Stabilization
Multi-modal fusion
Event OntologyEvent Expression Language
Camera Parameter Estimation
Automated Annotation Language
Integrity Analysis
Phase 2 Phase 3Phase 1 Future
Motion Analysis
Filtering
VACE InterestsTechnology Roadmap
Video AnalysisContent Extraction
9
Funding
FY06 Allocations FY07 Allocations4%
12%
20%
64% 36%
39%
4%10%
11%
Commitment to Success
Video AnalysisContent Extraction
10
Phase II
• Researcher Involvement: – Fourteen contracts– Researchers represent a cross section of industry
and academia throughout the U.S. partnering to reach a common goal
• Government Involvement: – Tap technical experts, analysts and COTRs from
DoD/IC agencies– Each agency is represented on the VACE
Advisory Committee, an advisory group to the ARDA/DTO Program Manager
Programmatics
Video AnalysisContent Extraction
11
Phase II Demographics
Virage
MIT
ColumbiaUniv.
Univ. of Maryland (2)
AFIT
Univ. of Illinois-Urbana-Champaign (2)
PurdueUniv.
Univ. ofChicago
Univ. ofWashington
SRI
Univ. of Southern
California / Info.Science Inst.
Univ. of Central Florida
GeorgiaInst. Of Tech.Telcordia
Technologies
Sub Contractors (14)
Univ. of SouthernCalifornia
IBM T. J.Watson Center
BoeingPhantom
Works
CarnegieMellon
Univ. (2) (Robotics Inst.)
(Informedia) .
Wright StateUniv.
TASC
Sarnoff Corp (2)
BBN
Salient Stills
Alphatech
Univ. of Illinois-Urbana-Champaign
Prime Contractors (14)
Univ. of Maryland
Video AnalysisContent Extraction
12
Phase II Projects
Title Organization Principal Investigator
Foreign Broadcast News Exploitation ENVIE: Extensible News
Video Information Exploitation
Carnegie Mellon University Howard Wactlar
Reconstructing and Mining of Semantic Threads Across Multiple Video Broadcast
News Sources using Multi-Level Concept Modeling
IBM T.J. Watson Research Center / Columbia Univ.
John Smith / Prof. Shih-Fu Chang
Formal and Informal Meetings
From Video to Information: Cross-Modal Analysis of
Planning Meetings
Wright State University
VaTech/AFIT / Univ. of Chicago / Purdue
Univ. / Univ. of Illinois-Urbana-Champaign
Francis Quek / Ronald Tuttle /
David McNeill & Bennett Bertenthal / Thomas Huang / Mary Harper
Event Recognition from Video of Formal and Informal
Meetings Using Behavioral Models and Multi-modal
Events
BAE Systems/ MIT / Univ. of Maryland /
Virage
Victor Tom/ William Freeman & John Fisher /
Yaser Yacoob & Larry Davis / Andy Merlino
Video AnalysisContent Extraction
13
Phase II Projects
Title Organization Principal Investigator
Abstraction and Inference about Surveillance Activities
Video Event Awareness Sarnoff Corporation / Telcordia Technologies
Rafael Alonso / Dimitrios Georgakopoulos
Integrated Research on Visual Surveillance University of Maryland
Larry Davis, Yiannis Aloimonos &
Rama Chellappa
UAV Motion Imagery Adaptive Video Processing for
Enhanced Object and Event Recognition in UAV Imagery
Boeing Phantom Works (Descoped to UAV data
collection)Robert Higgins
Task and Event Driven Compression (TEDC) for UAV
Video Sarnoff Corporation Hui Cheng
Ground Reconnaissance Video Content and Event Extraction from Ground Reconnaissance
Video
TASC, Inc. / Univ. of Central Florida / Univ. of
California-Irvine
Sadiye Guler / Mubarak Shah / Ramesh Jain
Video AnalysisContent Extraction
14
Phase II
ProjectsTitle Organization Principal Investigator
Cross Cutting / Enabling Technologies Probabilistic Graphical Model Based Tools For
Video Analysis
University of Illinois, Urbana - Champaign Thomas Huang
Automatic Video Resolution Enhancement Salient Stills, Inc. John Jeffrey Hunter
Robust Coarse-to-Fine Object Recognition in
Video
CMU/Pittsburgh Pattern Recognition
Henry Schneiderman & Tsuhan Chen
Multi-Lingual Video OCR BBN Technologies / SRI International
John Makhoul / Greg Myers
Model-based Object and Video Event Recognition
USC - Institute for Robotics and
Intelligent Systems / USC - Information Sciences Institute
Ram Nevatia, Gerard Medioni & Isaac Cohen /
Jerry Hobbs
Video AnalysisContent Extraction
15
Evaluation
• Programmatic:– Inform ARDA/DTO management of
progress/challenges• Developmental:
– Speed progress via iterative self testing– Enable research and evaluation via essential data and
tools – build lasting resources• Key is selecting the right tasks and metrics
– Gear evaluation tasks to research suite– Collect data to support all research
Goals
Video AnalysisContent Extraction
16
EvaluationThe Team
NIST
USF
Video Mining
Video AnalysisContent Extraction
17
EvaluationNIST Process
Formal Evaluation
Determine Sponsor Requirements
Assess required/existing
resources
Develop detailed plans with
researcher input
Dry-Run shakedown
Technical Workshopsand reports
Recommendations
PlanningPlanning ProductsProducts ResultsResultsEvaluation PlanTask Definitions
Protocols/Metrics
Rollout Schedule
Data Identification
Evaluation ResourcesTraining Data
Development Data
Evaluation Data
Ground Truth and other metadata
Scoring and Truthing Tools
Video AnalysisContent Extraction
18
Evaluation
NIST Mechanics
Video Data
Algorithms
Evaluation Results
Annotation
System Output
Ground Truth
Measures
Video AnalysisContent Extraction
19
Evaluation
2005-2006 Evaluations
Evaluation TypeEvaluation TypeDetectionDetection TrackingTracking RecognitionRecognition
DomainDomain PP FF VV TT PP FF VV TT PP FF VV TT
Meeting RoomMeeting Room xx xx xx xx
Broadcast NewsBroadcast News xx xx xx xx xx xx xx xx xx
UAVUAV xx xx xx xx
SurveillanceSurveillance xx xx xx xx xx xx
Ground ReconGround Recon
P = Person; F = Face; V = Vehicle; T = Text
Video AnalysisContent Extraction
20
Evaluation
• Evaluation Metrics:– Detection: SFDA (Sequence Frame Detection
Accuracy) • Metric for determining the accuracy of a detection
algorithm with respect to space, time, and the number of objects
– Tracking: STDA (Sequence Tracking Detection Accuracy)
• Metric for determining detection accuracy along with the ability of a system to assign and track the ID of an object across frames
– Text Recognition: WER (Word Error Rate) and CER (Character Error Rate)
• In-scene and overlay text in video• Focused Diagnostic Metrics (11)
Quantitative Metrics
Video AnalysisContent Extraction
21
EvaluationPhase II Best Results
Spatial Detection and Tracking Error
0%10%20%30%40%50%60%70%80%
Text(Broacast
News)
Face(Broacast
News)
Face(MeetingRoom)
Hand(MeetingRoom)
Person(MeetingRoom)
MovingVehicle(UAV)
Evaluation Tasks
Perc
ent E
rror
Detection
Tracking
Video AnalysisContent Extraction
22
Evaluation
PPATT UIUC UCF0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
(20%
T) S
FDA
Boxplot of (20%T) SFDA scores: Face in BNews
Sites
Face Detection: BNews (Score Distribution)
Video AnalysisContent Extraction
23
Evaluation
COLIB SRI_1 SRI_2 CMU_L0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
(20%
T) S
FDA
Sites
Boxplot of (20%T) SFDA Scores: EngText in BNews
Text Detection: BNews (SFDA Score distribution)
Video AnalysisContent Extraction
24
Evaluation
• Benefit of open evaluations– Knowledge about others’ capabilities and community feedback– increased competition -> progress
• Benefit of evaluation workshops– Encourage peer review and information exchange, minimize
“wheel reinvention”, focus research on common problems, venue for publication
• Current VACE-related open evaluations– VACE: Core Evaluations – CLEAR: Classification of Events, Activities, and Relationships – RT: Rich Transcription– TRECVID: Text Retrieval Conference Video Track– ETISEO: Evaluation du Traitment et de l’Interpretation de
Sequences Video
Open Evaluations and Workshops -- International
Video AnalysisContent Extraction
25
EvaluationExpanded Source Data
Task Sub-conditionConference
MeetingsSeminar Meetings Surveillance
Broadcast News
Studio Poses UAV
3D Sing Per Track Video CHIL
Audio CHIL
Audio+Video CHIL
2D Multi Per Detect VACE VACE
2D Multi Per Track VACE CHIL VACE VACE
Person ID Video CHIL
Audio CHIL
Audio + Video CHIL
2D Face Det VACE VACE
2D Face Track VACE CHIL VACE
Head Pose Est CHIL CHIL
Hand Detect VACE
Hand Track VACE VACE
Text Detect VACE
Text Track VACE
Text Recognition VACE
Vehicle Detect VACE VACE
Vehicle Track VACE VACE
Feature Extract TRECVID
Shot Boundary Detect TRECVID
Search TRECVID
Acoustic Event Detection CHIL
Environment Class CHIL
Video AnalysisContent Extraction
26
Evaluation
2005 2006 2007 2008A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D
Evaluation Schedule VACE Phase 2 VACE Phase 3TRECVID CLEAR RT
NIST/DTO VACE
VACE Phase 2 Core
Evaluation Series 1 (Text, Face in Meeting, BNews) Evaluation Series 2 (Hand in Meeting, Vehicle in UAV)
Evaluation Series 3 (Person in Meeting)
Evaluation Series 4 (Person/Vehicle in Surveillance)
VACE Phase 3 Core
Evaluation Series 5, 6, 7, 8
TRECVID
(Search and Extraction)
CLEAR
(Person/Face/Pose in Meeing)
(Extraction/Localiazation)
Rich Transcription
(English/Arabic Text in BNews)
(Multi-Modal Recognition)
ETISEO
4/30 6/30
5/96/6 11/8Report Results
8/249/23 11/8Report Results
9/810/3 11/8Report Results
6/28Report Results
2/23/2 4/5
Report Results
8/2 9/610/16
Report Results
2/43/114/22
Report Results
7/148/2310/2
Report Results
4/12 9/3011/14
Report Results
4/7 9/20 11/13
Report Results
4/6 9/20 11/13
Report Results
4/7 9/19 11/11
Report Results
2/1 3/94/8Report Results
3/14/25/11Report Results
3/34/4 5/9Report Results
5/4Report Results
4/306/1 7/5Report Results
4/306/6 7/9Report Results
1/2
Cycle 1
5/1 5/8
Cycle 2
11/17 12/8
Seminar
Schedule
Video AnalysisContent Extraction
27
TECH TRANSFER
Purpose: Move technology from lab to operation• Technology Readiness Activity
– An independent repository for test and assessment– Migrate technology out of lab environment– Assess technology maturity – Provide recommendations to DTO and researchers
DTO Test and Assessment Activities
Video AnalysisContent Extraction
28
Level Definitions Entry Condition Contractor Activity
1 Basic principles observed and reported Some peer review of ideas Reporting on basic idea
2 Technology concept and/or application formulated
Target applications are proposed Speculative work; invention
3 Analytical and experimental critical function and/or characteristic proof of concept
Algorithms run in contractor labs and basic testing is possible (internal, some external may be possible)
Doing Analytical studies with weakly integrated components
4 Component/breadboard validation in lab Proof of concept exists; test plans exist; external testing is possible
Low fidelity integration of components
5 Component/breadboard validation in relevant environment
Integrated system functions outside contractor lab; some TRA tests completed
Working with realistic situations
6 System/subsystem model or prototype in demonstration in relevant environment
IC/DoD users identified; target environment defined; simulated testing possible
Demonstrating engineering (software qualities) feasibility
7 System prototype demo in operational environment
Test lab trials in simulated environment completed; installed in operational environment
Completing the product
8 Actual system completed and qualified through test and demonstration
Product completed; Test lab trial completed successfully
Releasing the product; Repairing minor bugs
9 Actual system proven through successful mission operations
Proven value-added in an operational environment
Repairing minor bugs; noting proven operational results
DoD Technology Readiness Levels (TRL)
TECH TRANSFER
Video AnalysisContent Extraction
29
DTOControl
DTOInfluence
DOD Technology Risk Scale
HIGH LOWRISK
Technology Transfer
Use in assessing project’s• Technology maturity• Risk level• Commercialization potential
6 7
8 9
4 5
1 2 3
IC/DODTest
Facility(s)
Info-XTest
FacilityContractorTest
Facility
UNCLASSIFIED
UNCLASSIFIEDCLASSIFIED UNCLASSIFIED
CLASSIFIED
Production
Applying TRL
Video AnalysisContent Extraction
30
Technology Transfer
TRL Assesment
0123456789
Bac
kgro
undS
ubtra
ctio
n
Sta
biliz
atio
n
Obj
ectT
rack
ing_
Sta
tCam
era
Impo
rtanc
eMap
Gen
erat
or
Mul
tilay
erM
osai
cGen
erat
or
Eve
ntE
stim
atio
n
Mar
vel
Info
rmed
ia
STA
RT
Gut
ime
PIQ
UA
NT-
II
MP
SG
Opi
nion
Find
er
Kin
evis
Geo
Tim
e
TRL
Assesment prior to deliveryCurrent assesmentProjected at end of Contract
TRA Maturity Assessments
Video AnalysisContent Extraction
31
Phase III BAA
• Contracting Agency: DOI, Ft. Huachuca, AZ– DOI provides COR– ARDA/DTO retain DoD/IC agency COTR’s and add more
• Currently in Proposal Review Process– Span 3 FY’s and 4 CY’s– Remains open thru 6/30/08
• Funding objective: $30M over program life– Anticipate to grow in FY07 and beyond
• Address the same data source domains as Phase II• Will conduct formal evaluations• Will conduct maturity evaluations and tech transfer
Programmatics
Video AnalysisContent Extraction
32
Phase III BAA
• Emphasis on technology and system approach– Move up technology path where applicable– Stress ubiquity
• Divided into two tiers:– Tier 1: One year base with option year
• Technology focus• Open to all – US and international• More awards for lesser funding
– Tier 2: Two year base with option year(s)• Comprehensive component/system level initiative• Must be US prime• Fewer awards for greater funding
Programmatics
Video AnalysisContent Extraction
33
Phase III BAASchedule2005 2006
Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct
Planning Schedule Milestone Major Task
Activity
Scope Planning
Planning Group Meetings
BAA RFP
First Draft
Second Draft
Third Draft
Final Draft
Contracting Final Review
BAA
Announcement
Questions/Proposals Due
Proposal Evaluation
Source Selection Meeting
Funding Recommendations
Contract Awards
Phase 3 Kickoff Workshop
6/14 6/30 8/31
8/2 9/27 10/18 11/7
DOI
11/22
7/18 2/2
7/18 8/2Six weeks after kickoff
9/1 9/30Six weeks after first draft
10/10 11/4Completed prior to Fall Workshop
11/21 12/2Sent to Contracting Office
12/2Draft Final BAA
12/16 1/8 1/27Final BAA
12/15 6/30
12/15Draft BAA Posting
1/20Brief
2/1Final BAA Posting
1/6Questions/Comments
3/3Proposals
3/6 4/144/25
4/26-27
4/28
5/1 6/30
8/9-10
Video AnalysisContent Extraction
34
Summary
• VACE is interested in:– Solving real problems with risky, radical
approaches– Processing multiple data domains and multimodal
data domains– Developing technology point solutions as well as
component/system solutions– Evaluating technology process– Transferring technology into user’s space
Take-Aways
Video AnalysisContent Extraction
35
Conclusion
• Invitations:– Welcome to participate in VACE Phase III– Welcome to participate in VACE Phase III Evaluations
Potential DTO Collaboration
Video AnalysisContent Extraction
36
Contacts
Dennis Moellman, Program ManagerPhones: 202-231-4453 (Dennis Moellman) 443-479-4365 (Paul Matthews) 301-688-7092 (DTO Office)800-276-3747 (DTO Office)
FAX: 202-231-4242 (Dennis Moellman)301-688-7410 (DTO Office)
E-Mail: [email protected] (Internet Mail) [email protected]
Location: Room 12A69 NBP #1 Suite 6644 9800 Savage RoadFort Meade, MD 20755-6644