Multiple target detection in video using quadratic multi...
Transcript of Multiple target detection in video using quadratic multi...
1
Multiple target detection in video using quadratic multi-frame
correlation filtering
Ryan KerekesOak Ridge National Laboratory
B. V. K. Vijaya KumarCarnegie Mellon University
March 17, 2008
2
Outline
PART I• Correlation filtering overview• Database
PART II• Multi-frame correlation theory• Multi-frame simulation results
3
CF
Why use correlation filters?
• Shift invariance: useful when we don’t know:– Target locations in the scene– Number of targets in the scene
• Distortion tolerance: CF’s can be trained to tolerate some distortion and reject false targets
• Graceful degradation: noise, occlusion
False patterns
Scaled versionsTraining pattern
4
Correlation peak metric
• Often we will measure peak sharpness– Peak-to-sidelobe ratio (PSR)
5
Correlation peak metric
• Often we will measure peak sharpness– Peak-to-sidelobe ratio (PSR) metric– invariance to overall image brightness
PSR = 7.46
6
FFT
Correlation filter usage
• Typically computed in the frequency domain– 2 FFTs– complexity reduction
• PSR computed at every point in frequency domain– 4 more FFTs
IFFT
input
FT of filter
correlation array
FT of PSR window
4FFTs
PSR array
*
7
Linear filter vs. quadratic filter
• Consider thresholding the output– results in a discriminant at each shift
pixel 2
pixe
l 3
pixel 2
pixe
l 3
threshold
class 1class 2
example: elliptical boundary
threshold
linear filter quadratic filter
8
Turntable database
• Captured rotational imagery of three targets• Depression angles used: 17 , 19 , 21 , 23• Green background used to aid in segmentation
17°
211cmtarget
camera
turntable
green background
9
Filter training
• Each filter trained on scaled and out-of-plane-rotated images of true- and false-class targets
0° 3° 30°
…
…
…
…
Azimuth
… …Scale (PHTW)
True
False (see
n)
Missile
Leopard
False (unsee
n)Abrams
16
20
10
Filter comparisons
• Showed that RQQCF filters outperform others– each filter bank trained to divide up 360 azimuth range– different number of filters/bank, equal computation
filter design
LEGEND
EOTSDF Eigen-filter
UOTSDF Unconstrained OTSDF
EMACH Extended MACH
CPCF/UPCF
Polynomial CFs
RQQCF/SSQSDF
Quadratic filters
11
PART IIMulti-frame correlation filtering
MFCF
12
CF in video sequences
• Three types of approaches to the problem
CF
CF
CF
Frame 1
Frame 2
Frame 3
…
THRESHOLD
THRESHOLD
THRESHOLD
Correlation planes
Test images
DetectionsTYPE 1: SINGLE-FRAME
13
CF in video sequences
• Three types of approaches to the problem
CF
CF
CF
Frame 1
Frame 2
Frame 3
…
THRESHOLD
THRESHOLD
THRESHOLD
Correlation planes
Test images
DetectionsTYPE 2: DETECTION-FIRST (CLASSICAL TRACKING)
Relate information
14
CF in video sequences
• Three types of approaches to the problem
CF
CF
CF
Frame 1
Frame 2
Frame 3
…
THRESHOLD
THRESHOLD
THRESHOLD
Correlation planes
Test images
DetectionsTYPE 2: DETECTION-FIRST (CLASSICAL TRACKING)
Relate information
Information loss due to thresholding
step
Data association
problem due to false alarms
15
CF’s in video sequences
• Three types of approaches to the problem
CF
CF
CF
Frame 1
Frame 2
Frame 3
…
THRESHOLD
THRESHOLD
THRESHOLD
Correlation planes
Test images
DetectionsTYPE 3: RELATION-THEN-DETECTION (our work)
Relate information
16
Efficient output combination
• We want to combine outputs while preserving information– Avoid preliminary thresholding step – Avoid assumptions on number of targets– Avoid a large computational increase
• We have developed a probabilistic approach– We can derive defensible theory– We can confidently utilize all available information
( )Pr | 1, 2,target frame frame …
17
Typical detection schematic
18
Probability mapping
• We map each correlation value to probability of target presence using all available information
PSR array probability array
mapping
high likelihood of target (PSR ≈ 10)
reasonable likelihood of target (PSR ≈ 4)
19
PSR score distributions
• Must assume distributions for target scores and non-target scores for each filter– (we used Gaussians)
Example from a FLIR sequencetarget non-target (clutter)
20
Target motion model
• Must assume transition probability function between adjacent frames– can take velocity into account
• Function of displacement (position-independent)– (we used centered, rotationally symmetric functions)
Gaussian exponential uniform
21
Posterior probability array
• We can derive a pixel-wise mapping function for the posterior probability array
“enhanced output”
“knowledge” from Frames 1
and 2
map
Frame 3 prior probs.
Frame 3 output
Frame 3 posterior probs.
information from Frame 3
22
i-th frame posterior probability array
i-th frame prior probability array
score pdf of filter l on target k
Mapping function
• Various properties of the mapping function
Uses prior arrays from all filters
Uses PSR scores from all filters
Already normalized “filter communication”
sub-class (filter) index
frame index
23
Mapping function
• Example of a mapping in a two-class problemPSR value target probability
24
Prior probability array
• We can derive the prior probability array for the next frame: based on convolution
Frame 3 posterior probs.
“predictor” for Frame 4
motion model priorformula
“knowledge” from Frames 1,
2, and 3 Frame 4 prior probs.
25
i-th frame posterior array
i-th frame prior array
Motion model function
Emergence/disappearance probability functions
Subclass transition probs.
,
Prior array formula
• Various properties of the prior array formula
probability of target emergence
ε(x)
non-zero probability
region
zero probability region
26
Prior array formula
• Various properties of the prior array formula
probability of target disappearance
δ(x)
non-zero probability
region
zero probability region
i-th frame posterior array
i-th frame prior array
Motion model function
Emergence/disappearance probability functions
Subclass transition probs.
,
27
Prior array formula
• Various properties of the prior array formula
only one convolution per filter
* i-th frame posterior array
i-th frame prior array
Motion model function
Emergence/disappearance probability functions
Subclass transition probs.
,
28
Prior array formula
• Various properties of the prior array formulauses posterior arrays from all
filters
subclass transition probability
“filter communication”
i-th frame posterior array
i-th frame prior array
Motion model function
Emergence/disappearance probability functions
Subclass transition probs.
,
29
Multi-frame schematic
mapFrame i
Frame ioutput
(to next iteration)
THRESHOLD
Frame i priors
Frame iposterior
probabilitiesMotion model
Frame i+1 priorspriorformula
detections
30
Multi-frame schematic
mapFrame i
Frame ioutput
(to next iteration)
THRESHOLD
Frame i priors
Frame iposterior
probabilitiesMotion model
Frame i+1 priorsbig
formula
detections
Table lookup
2 FFTs
31
Synthetic sequence demo
Sequence
Prior probability
arrayOriginal output
Posterior probability
array
log-likelihood values
PSR values
32
Synthetic sequence demo
• MFCF can handle multiple targets– Example: sequence with 3 true-class targets, 3 false-class
MFCF detections
single-frame
MFCF
33
Synthetic sequence demo
• False alarms reduced in target and non-target areas@ 90% detection rate
target paths
single-frame MFCF
34
FLIR sequence results• Two FLIR sequences tested • Two types of noise each• MFCF offers greater performance improvement in
white noise than in compression noise
H.264 compression noise (65:1)
AWGN, SNR=20dB
Sequence L2117
MFCF
35
Noise vs. clutter in MFCF
• MFCF favors filters with better clutter rejection– Example: sequence Synthetic3, noisy (SNR = 20dB)– Varied number of eigenvalues (Ne) retained in filter design
single-frame multi-frameworse than
single-frame
36
Summary of main findings
MFCF algorithm:• MFCF is best-suited for handling temporally
uncorrelated noise• Clutter rejection should be handled by the filter(s)• Filter communication can help reject false targets• MFCF degrades gracefully under parameter
changes
37
Future work
• Multi-view correlation filtering (multiple cameras)
scene
View 1 View 2