Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D....
-
date post
20-Dec-2015 -
Category
Documents
-
view
224 -
download
2
Transcript of Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D....
Closing the Loop for ISPusing Performance Prediction
Dec-05
Greg Arnold, [email protected]
Sensors Directorate
Air Force Research LaboratoryAFRL/SNAT, Bldg 620; 2241 Avionics Circle
WPAFB OH 45433-7321; (937) 255-1115x4388
2
Trilogy of Thoughts / Goals
• Playground- Urban SASO (Security & Stability Ops)• ISP- context is UAV swarms & S-S fusion
– Need multiple sensors– Confirmatory Sensing and Interrogation– Anomaly detection & backtracking– Understand the problem
• Active Vision- manipulate the sensor to improve perf– Offline: ATR-driven sensing– Online: Time-reversal. Active filter. Gotcha,
• ATR Theory- performance prediction is the key!– Reasoning in 3D (requires metrics)– Images are samples from world– General => Specific for robustness
Uninhabited Air Vehicle
3
Target RecognitionLevels of Discrimination
• Detection: the level at which targets are distinguished from non-targets, i.e., clutter objects such as trees, rocks, or image processing artifacts.
• Classification: the level at which target class is resolved, e.g., building, vehicle, or aircraft.
• Recognition: the level at which target subclass is determined, e.g., for a tracked vehicle, tank, APC, or ADU.
• Identification: the level at which the model/make of a target is resolved, e.g., for a tank: M60, M1, or T72.
• Fingerprint: the serial number of a particular instance of a target, i.e. Vince’s Caravan vs. Lori’s Caravan.
Coarse
Fine
4
Target RecognitionLevels of Automation
• Interactive decision aid: – Human and machine work interactively
• Automatic decision aid: – Machine is autonomous from input of data to
output to human– Human makes final decision (Human-in-the-loop)
• Autonomous system: – Machine makes the final decision– Human is NOT in the loop
5
What Does ATR Mean?
Community Function Execution Speed
Intelligence Assist a Human Interpret Data*
Non Real-Time (weeks – years) (hours)
Surveillance Assist a Human Recognize Objects
Non Real-Time (hours-days)
Targeting/Fire Control
Assist a Human Recognize Target
Non Real-Time (seconds – minutes)
Autonomous Guided Munition
Autonomously Acquire Target
Real-Time (milliseconds)
Battle Damage Assessment
Assist a Human Evaluating Effects
Sooner the Better (milliseconds-days)
6
Found Something
Go Get It
ISR Goals
ISR Goals• No Sanctuary• Persistent (PISR)• All Weather• Day / Night• All Terrain (city, country,
forest, desert, ocean)
• Moving & Stationary• Safety !!!
Intelligence, Surveillance, and Reconnaissance
ShotKill Chain
Hit It!
7
Automated Target Recognition (ATR) Insights
• Information Limited: Believe current performance is information limited– Human (Data >> Information)
• Pixels/Pupils ratio• Better SNR, resolution, modalities
– Machine• False Alarms (Google Search)• Finer Discrim./Obscuration (>> higher resolution)
• 3-D: Intuitively understand geometric (3-D) information
• UAVs: UAVs transform the CID problem!
8
Sensors Directorate Structure
• SN Directorate– SNA: Sensor ATR Technology– SNJ: Light (EO) Sensors– SNR: Radio (RF) Sensors– SNZ: Applications
• SNA Division: Mike Bryant, Lori Westerkamp (Ed Zelnio)– SNAA: Evaluation– SNAR: Applications– SNAS: Modeling and Signatures– SNAT: Innovative Algorithms
• SNAT Branch: Dale Nelson, Rob Williams– Generation After Next Technologies & Algorithms (Greg
Arnold)– Tracking and Registration (Devert Wicker)– Vigilance (Kevin Priddy)
9
ATR Thrust Scope
GEOLOCATE
FIND
TRACK
ID M60
FU
NC
TIO
NS
SENSORS
MATURATION PROCESSAlgorith
ms
Signatures
Assessment0 100
1
PD
FAR
10
Challenge ProblemsStandard MetricsATR Theory
Characterized PerformanceHigh Performance ComputingOperational Databases
ASSESSMENT& FOUNDATION
Sensor Data Management System (SDMS)
ATR Thrust Approach - Subthrusts
INNOVATIVE ALGORITHMS
FIND FIX TRACK & ID
Phenomenology ExplorationEM ModelingSynthetic Data
Operational Target Models/Databases
SIGNATURES & MODELING
SignatureCenter
5
10
15
20
10
20
30
40
02.5
5
7.5
10
5
10
15
20
11
12
3-D ATR− 3-D Imaging for RF Floodlight− 3-D for urban context− ATR Theory challenge problem
ATRC
ap
abili
ty /
Dif
fic
ult
y
Near Mid Far
Spiral D
evelopment
We need a generalized pattern recognition capability that will classify things previously unseen, actively manage assets, and predict the intent and actions of combatants.
Adaptive ATR − On-the-fly modeling / reacquisition− Reasoning with uncertainty− Adaptive metrics derived from user
ATR for Anticipatory ISR − Multi-X fusion for PISR/TST− Dynamic GIG Sensor Management− ATR Theory for Anticipation
Goals
13
Assumptions / BeliefsBackground / Framework
• Must Define Problem & EOC’s– Whether or not applying model-based vision– Necessary for testing algorithm capabilities
• Model-Based Vision– More than just CAD models– Characterization of the data
and the system at some level– “If I can’t model it, I don’t understand it”
• Physics-Based Vision– What can we do before appealing to statistics
Real World Dimensionality
DataModel
s
14
Operating Conditions (OCs)
TargetsTargetsSensorsSensors
InteractionsInteractions
EnvironmentEnvironment
OCs: Everything that changes the sensor response. Most OCs have infinite variation
15
Real world variability:Extended Operating Conditions (EOCs)
. . .
20 Target Types
6 DOFPose
Squint & Depression
AngleArticulation
x
y
zObscuration
Variants
Billions20 6524
23
22
61 QQQQQQN Billions20 65
24
23
22
61 QQQQQQN
Configuration
16
Discrimination vs. Robustness
Data Models
Robustness
Disc
rimin
atio
n
Seria
l #
Sam
e Se
nsor
Sens
or T
ype
Synt
hetic
Shape
Reflectivity
Tuni
ng
MSEMSE
Quantized
Quantized
Metrics
Metrics
Binary
Binary
Shape
Shape
Points
Points
17
Using Information More Effectively
More Information
ATR Driven Sensing
Multisensor Approaches
Bio-Inspired Adaptive ATR
Challenge Space
1717
18
Trilogy of Thoughts / Goals
• Playground- Urban SASO• ISP- context is UAV swarms & S-S fusion
– Need multiple sensors– Confirmatory Sensing and Interrogation– Anomaly detection & backtracking– Understand the problem
• Active Vision- manipulate the sensor to improve perf– Offline: ATR-driven sensing– Online: Time-reversal. Active filter. Gotcha,
• ATR Theory- performance prediction is the key!– Reasoning in 3D (requires metrics)– Images are samples from world– General => Specific for robustness
19
Vertically Integrated Sensor Exploitation for Generalized Recce & Instant Prosecution
(VISEGRIP)
20
TheoryTheory
Goal: Quantify the accuracy, completeness, & relevance of information with demonstrable authority. Challenge problem: support counter-WMD
Objective: Theory & algorithm research to incorporate ATR Theory principles into Sensor Mgmt infrastructure modified to implement confirmatory sensing & interrogation
Payoff: Pattern Recognition discipline that is more expressive to assure users that source is authoritative and information is “actionable”
Confirmatory Sensing & Interrogation
AlgorithmsAlgorithms
BackgroundBackground
ATR Theory•Aims to design and predict performance of sensor data exploitation systems•Includes all forms of sensor data exploitation i.e. target detection, tracking, recognition, and fusion
Information TheoryStudies the collection and manipulation of information
Query Generation What question to ask
Query Processing When, How, & Who to ask
Data Fusion Align redundant informationAssess unique or contradictory informationAssimilate valuable information
Evidence Assessment Quantify accuracy and completeness of assertionsPredict a window of opportunity
21
22
• Playground- Urban SASO• ISP- context is UAV swarms & S-S fusion
– Need multiple sensors– Confirmatory Sensing and Interrogation– Anomaly detection & backtracking– Understand the problem
• Active Vision- manipulate the sensor to improve perf– Offline: ATR-driven sensing– Online: Time-reversal. Active filter. Gotcha,
• ATR Theory- performance prediction is the key!– Reasoning in 3D (requires metrics)– Images are samples from world– General => Specific for robustness
Trilogy of Thoughts / Goals
23
ATR-Driven Sensing
Cueing, Prioritization for the Human
~1’10’ ~1’
Short Circuits
Resonant Cross Slots Cavity
Low Dielectric/Low Loss Face Sheet
24
• Playground- Urban SASO• ISP- context is UAV swarms & S-S fusion
– Need multiple sensors– Confirmatory Sensing and Interrogation– Anomaly detection & backtracking– Understand the problem
• Active Vision- manipulate the sensor to improve perf– Offline: ATR-driven sensing– Online: Time-reversal. Active filter. Gotcha,
• ATR Theory- performance prediction is the key!– Reasoning in 3D (requires metrics)– Images are samples from world– General => Specific for robustness
Trilogy of Thoughts / Goals
Uninhabited Air Vehicle
25
What is your Objective Function?
• L-p (L-1, L-2, L-infinity)• Diffusion Distance• Hausdorff• Chamfer• Ali-Silvey• Earth Movers Distance• Chi Squared• Entropy• Kullback-Liebler• Mutual Information• Maximum Likelihood• Renyi
• Pd (Prob. of Detection)• Pcc (correct classification)• Pe (Prob. of Error)• Pfa (False Alarm)• Confusion Matrix• Precision• Recall• ROC Curve
26
‘Clear Box’ View of ATR
0 100
1
PD
FAR
Environment
DetectDetectTrackTrack
GeolocateGeolocateIDID
SensorTargetATR
DecisionsHuman
Decisions
TrainedFeatures Templates Models
Target Knowledge
FeatureExtractor
Discriminator
DecisionRule
Target Knowledge
Target Models &Database
27
ATR/Fusion Processes
Target Models &DatabaseSensor
Model
Environment
DetectDetectTrackTrack
GeolocateGeolocateIDID
Sensor(s)TargetATR
DecisionsHuman
Decisions
Sensor Management
Registration
Environment Model
Performance Model
Adaptation
Behavior Models
Anticipate
28
Performance Model is the Lynchpin
• ATR System is dependent on the Performance Model• Need performance prediction
– Determine where / when to use sensors– Estimate effectiveness of sensors for given task
– Sensor Management– Registration– Learning– …
29
If Somebody Asks…
• Typical DARPA question– “Is it physically possible to do X?”– We’ve invested $K and achieved P% performance, is it
worth investing more?
• Examples– How likely are we to detect a dismount with an HSI
system with 1m spatial resolution? 1ft? 1in?– We spent $40M and achieved 80% of perfection.
Have we reached the knee in the performance curve?– Organization X says it can build a system to do Y.
Does this violate physics?
30
Aspects of ATR Theory Objectives
• Measure the information content of sensor imagery
• Given a set of data and a MOP, determine attainable performance range
• What are the critical design constraints to achieve a desired outcome, using this data?
• Estimate exploitation level of available information
• Establish “feedback loop” between ATR designers and sensor developers
• What are the critical design constraints to achieve a desired outcome at a particular level of confidence?
• Information gain from using models and data adaptively (learning)
• Determine theoretical upper bound on performance of given ATR
• Given an ATR system and a set of data, determine how much information can be exploited
• Determine how close a given system comes to achieving the optimal bound
• What are the critical design constraints to achieve a desired outcome, using this sensor and algorithm?
• What was the benefit of adding ‘this’ (additional data/processing)?
Data Assessment Design System Evaluation
31
Problem Simplification
• Having said all that, let’s examine a problem for which we have some intuition
• 4 or 5 points undergoing rotation, translation, and maybe scale and skew
• 1-D, 2-D, and 3-D• Understand the projection from world to sensor
32
What is Shape?
• Pose and scale invariant, coordinate independent characterization of an arrangement of features.
• Residual geometric relationships that remain between features after “mod-ing out” the transformation group action.
• Captured by a “shape space” where each distinct configuration of features (up to transformation) is represented by a single point.
33
Beyond Invariants
Invariants +
Projection
Object-Image Relations
34
Generalized Weak Perspective
• Projection model applicable to optical images(pinhole camera)
• Approximates full perspective for objects in ‘far field’
• Affine transformations on 3-space, and in the image plane (2-space)
• Denoted GWP
35
Affine Transformations
• In 3D
110003333231
2232221
1131211
z
y
x
taaa
taaa
taaa
(Rotate, Scale, Skew | Translate) (3-D Point)
36
GWP Projection3D to 2D
11...111
...
...
1321
1321
nn
nn
vvvvv
uuuuu
11...111
...
...
...
1000 1321
1321
1321
4
4
3
3
2
2
1
1
nn
nn
nn
zzzzz
yyyyy
xxxxx
b
a
b
a
b
a
b
a
Image Projection Object
37
Object-Image Relation Motivation
Object 1
Object 2Image 1
Image 2
• Image 1 is not equivalent to Image 2 (in 2-D)
• Object 1 is not equivalent to Object 2 (in 3-D)
38
Object - Image Relations Concept
“The relation between objects and imagesexpressed independent of the camera parameters
and transformation group”
(1) Write out the camera equations (geo or photo)
(2) Eliminate the group & camera parameters(3) Recognize the result as a relation between
the object and image invariants.
But pure elimination is VERY difficult even for polynomials.
39
Weak PerspectiveObject - Image Relations
(Generalized)
• Parallel things remain parallel• The object size is 1/10 the distance from the camera• (Standard Position Method)
40
Weak Perspective Camera
3-D Model Pi={xi,yi,zi} N-points (3N DOF)Rotate,Translate,Scale,Shear (12 Constraints)
3N-12 Absolute Invariants
2-D Image qi={ui,vi} N-points (2N DOF)Rotate,Translate,Scale,Shear (6 Constraints)
2N-6 Absolute Invariants
Camera ModelN-points (2N DOF)Union 2-D & 3-D (8 Constraints)
2N-8 relations
Need 5 corresponded points (minimum)q1 q2
q3
q4 q5
P1 P2
P3
P4 P5
41
||
||,
||
||,
||
||
4321
5432
4321
5431
4321
5321
PPPP
PPPP
PPPP
PPPP
PPPP
PPPP
3-D Invariants
P1 P2
P3
P4 P5 3-D Model Pi={xi,yi,zi,1} 5-pointsGL3+Translation (12 Constraints)
3N-12 Absolute Invariants
Invariant is a function of the Ratio of Determinants:
A useful standard position is:
11111100001000010
53
52
51
III
42
||
||,
||
||,
||
||,
||
||
321
532
321
432
321
521
321
421
qqq
qqq
qqq
qqq
qqq
qqq
qqq
qqq
11111100010
5242
5141
iiii
2-D Invariants
2-D Image qi={ui,vi,1} 5-pointsGL2+Translation (6 Constraints)
2N-6 Absolute Invariants
Invariant is a function of the Ratio of Determinants:
A useful standard position is:
q1 q2
q3
q4 q5
43
11111100001000010
1000010001
11111100010
53
52
51
42
41
5242
5141
III
ii
iiii
Object - Image RelationGeneralized Weak Perspective Camera
(2-D Standard Position) = (Camera Transform) (3-D Standard Position)
The camera transforms the first 4 object point to image points,the remaining points satisfy the object - image relation iff:
5342525253415151 ; IiIiIiIi
11111
1000
0100
0010
100011111
100
010
53
52
51
0232221
0131211
5242
5141
I
I
I
vttt
uttt
ii
ii
Eliminate camera transform parameters:
44
Object-Image Relation Abstraction
All objects that could have
produced the image.
All images of the object.
x
uObject-Image
Relations
GX n /
Space ShapeObject 3 GU n ˆ/
Space Shape Image2
45
GWP Shape Spaces
• The shape spaces in the GWP case are Grassmann manifolds
• In 3D– Gr(n-4,H) or dually the Schubert cycle of 4-planes in
Gr(4,n) which contain (1,….,1)– Manifold has dimension 3n-12
• In 2D– Gr(n-3,H) or dually the Schubert cycle of 3-planes in
Gr(3,n) which contain (1,….,1)– Manifold has dimension 2n-6
H is the subspace of n-space orthogonal to the vector (1,…,1)
46
Why
• We associate to our object data, viewed as a linear transformation from n-space to 4-space, its null space K of dimension n-4.
• Likewise to our image data in 2D we associate the null space L of dimension n-3.
11...111
...
...
...
1321
1321
1321
nn
nn
nn
zzzzz
yyyyy
xxxxx
47
Global Shape Coordinates
• Better than local invariants
• Come from an isometric embedding of the shape space in either Euclidean space or projective space.
• Matching expressed in these coordinates will gracefully degrade
48
Example in GWP
• 3D, n = 5 feature points
• Global shape coordinates are the Plucker coordinates (or dual Plucker coordinates) of the 4xn object data matrix or the 3xn image data matrix.
1111
det],,,[,,,lkji
lkji
lkji
lkji zzzz
yyyy
xxxx
lkjiM
111
det],,[,, tsr
tsr
tsr vvv
uuu
tsrN
49
Global Object-Image Relations
• General– If and only If conditions– Overdetermined set of equations
• GWP– To match, K must be contained in L (iff)– This incidence condition can be expressed in terms of the
global shape coordinates– For n=5, 10 (non-independent) relations that look like:
[1234][125]-[1235][124]+[1245][123]
Locally only 2 of the 10 are independent, because the locus V of matching pairs (object shape, image shape) in the 7 dimensional product space XxY has dimension 5, codimension 2.
50
Beyond Object-Image Relations
Object-Image Relations
+
Matching
Object-Image Metrics
51
• We intuitively know that if we want to measure something we need a metric… ATR is no different.
• How far apart are these points?
1. The triangle inequality provides efficient match searching
2. Reliable & predictable Unknowns rejection3. Theoretical performance prediction
Why Metrics?
52
The Triangle Inequality Advantage
u: image, x*: prototype object, xk: object from group
Shape Space
u
Measure the distance from the image to each shape object?
Search the group iff the distance to the prototype is less than the sum of the max intragroup distance and noise threshold.
X*
X*
X*
u
Measure the distance from the image to each shape prototype!
Group 1
Group 2
Group 3
53
Using the Triangle Inequality
*
*
noise*
Offline,
*
tMeasuremen
*
**
**
Threshold],[
],[max],[],[
],[],[],[
],[],[],[
x
d
jKj
k
kk
kk
dxud
xxdxudxud
xxdxudxud
xxdxudxud
x
Equivalent
Grouping
Decision
u: image, x*: prototype object, xk: object from group
Search the group iff the distance to the prototype is less than the sum of
the max intragroup distance and noise threshold.
XkX*
uShape Space
Reject beyond Thresholdnoise
54
What are Shape and Distance?
• Shape: What is left after translation & rotation are removed (more generally, the group)
• This is the (Partial) Procrustes definition of distance– R represents rotation and T represents translation– Procrustes normalizes the size of the objects
2222111
,,,21 )ˆ()ˆ(inf),(
2211
TORTOROOdTRTR
p
55
New Metrics?
Any ol’ metric just won’t do…
1. Invariant to translation & rotation of 3-D object (+ more)
2. Invariant to the camera projection (+ discretization)
• This leads to the concept of Object-Image Relations (O-IR’s)
• Incomplete
– O-IR’s are only surrogate metrics
– =0 iff the object and image features are consistent
• Object-Image Metrics satisfy all the metric properties
• Shape Space is NOT Euclidean!• There is some evidence that human similarity perception is not always
metric
56
Metrics on the Shape Spaces
• How to compare objects to images!
• We want a natural shape matching metric– Invariant to transformations of the 3D or 2D data,– e.g. Rotations, translations, or scale of the object or image
• Generalize Weak Perspective– We use the natural Riemannian metric on the Grassmannian
to measure distances between object shapes and image shapes
– This involves the so called principal angles between subspaces and is easily computed from the original data matrices via QR decomposition and SVD.
57
Object-Image Metrics
Two ways to compute an “object to image” distance
1. Object SpaceCompute the minimum distance in object space from the given object to the set of all objects capable of producing the given image
2. Image SpaceCompute the minimum distance from the given image to the set of all images produced by that object
58
Object-Image Metrics & Duality
],[inf],[Obj xxdxud uEuxu
],[inf],[Img xExu
uudxudx
Duality Theorem: ],[],[ ImgObj xudxud
xu: all objects that could have produced the
image.
ux: all images of the object.
x
uObject-Image
Relations
Matching can (in principle)Matching can (in principle) be performed in either object be performed in either object or image space without loss of performance !or image space without loss of performance !
GX n /3 GU n ˆ/2Object Shape Space Image Shape Space
59
Duality
• Theorem - with suitable normalization
These metrics are the same!
In the GWP case this distance turns out to
be the distance between two subspaces of
different dimension defined again by using
principal angles.
60
Image Geodesics
•2 random images
•Geodesic between them
•Not Linear
•Not the projection of a line
•Not even coplanar•Geodesics on the this cone have the same length as the calculated image distance!
61
Orthographic Shape Space3 Points in 1-D & 2-D
• 3 points modulo translation, rotation, reflection yields…
• 1-D: Surface of a 30o cone w/ axis along {1,1,1}
• 2-D: Interior of the cone
• {0,0,0} object @ origin• {a,a,b} objects partition cone• Scale: lines through origin
• Geodesics on the this cone have the same length as the calculated image distance!
-2
0
2
-2
0
2
-4
-3
-2
-1
0
-2
0
2
-2
0
2
Rotation on the ‘wrong’ side aboutthe centroid rotates the cone (isotropy condition).
62
Object-Image Relations (1)
•Fix an Object
•Set of Images it can produce
•Always circumscribe the cone
•Not conic sections!
•Equilateral triangle produces
a slightly smaller circle
•‘Image’ produces line to origin
63
Object-Image Relations (2)
• Fix an Image• Set of objects that could
produce the given image• “Bent over cone”• Touches along line through
origin and image• Eventually converges to the
cone surface– Large objects must be
nearly collinear to produce the image
64
Epsilon Balls & Noise Analysis
• The set of all shapes of distance 1 from the given shape (image)
• The image + Gaussian IID noise added to each image point location(std. dev. 0.5)
• Still working the analytic model of noise in the shape space
65
Intrinsic Separability
• How many different shapes can I hope to identify?– Shape space as a unit volume– Epsilon balls defined by metric– Noise balls generated by Gaussian noise
• 5 points in 3D (Generalized Weak Perspective)– Epsilon in [0, 0.73] (max radius of ball)– P(Random Shape in Epsilon Ball)=1.37*Epsilon
• Example: Epsilon=0.01– P(Random Shape in Epsilon Ball)=0.014– Requires as least 73 balls to cover shape space– (Could be more or less efficient coverings)– Separability on a gross level
66
Summary
• Integration of Sensing and Processing• Active Vision• ATR Theory
• These are all connected & overlapping areas• Provides a rich field of problems and applications
ATR Theory
Active VisionISP
67
Open Problems in Inf. Exploitation
• Open Problems: ATR Theory– Classification before Recognition before ID– Representation
• Modeling free forms spanning discrete-continuous• Uncertainty in models (i.e. target variability)?• Cross-sensor phenomenology (registration / fusion)• Correspondence
– Unmixing: automated methods for separating foreground / background in various scenarios (ID before segmentation or pose estimation)
– Intrinsic Separability• How objects separate in quotient space as a function of sensor• Confusers• Unknowns: How to separate knowns from unknowns
– (Object-Image) Metrics• Efficient Search of Large Databases • How to choose the metrics based on the expected noise model & the type of
quotient space derived from the choice of metric• Long Poles:
– Probabilities for Fusion / Reasoning with Uncertainty – Recognition By Components (including construction/decomposition) – Non-Gaussian, Nonlinear Analysis (i.e. most models and algorithms assume
these two properties) – Adaptive systems– How to modulate the prior probabilities on-the-fly