Image Fusion with some Emphasis on CWD - Lehigh ECE R.S. Blum - Lehigh University 16 NMDB Image...
Transcript of Image Fusion with some Emphasis on CWD - Lehigh ECE R.S. Blum - Lehigh University 16 NMDB Image...
Image Fusion with some Emphasis on CWD
R. S. [email protected]
ECE Dept., Lehigh University
This material is based on work supported by the U. S. Army Research Office under grant number DAAD19-00-1-0431. The content of the information does not necessarily reflect the position or the policy of the federal government, and no official endorsement should be inferred.
Several Graduate Students of the SPCRL Lab at Lehigh contributed greatly to this work:
Zhong Zhang Zhiyun Xue
Jinzhong YangFor more information and papers please see
our website at: http://www.ece.lehigh.edu/SPCRL/spcrl.htm
06/24/2002 R.S. Blum - Lehigh University 3
Outline
Introduction to Image FusionMethods for Image FusionPerformance Testing for CWD A Signal Processing Approach Other Lehigh Contributions Conclusions
06/24/2002 R.S. Blum - Lehigh University 4
Image SensorsOptical cameras, millimeter wave (MMW) cameras, infrared (IR) cameras, x-ray imagers, radar imagersOptimized for different operating conditionsDifferent characteristics of the generated image data
Introduction
06/24/2002 R.S. Blum - Lehigh University 5
Concept of Image FusionA process of combining information from multiple images to generate a single image that contains a more accurate description of the scene than any of the individual source images
Introduction
06/24/2002 R.S. Blum - Lehigh University 6
Advantages of Image FusionMore completeMore accurateMore robustIn less timeAt a lower costImprove human visual perceptionImprove automatic computer analysis
Introduction
06/24/2002 R.S. Blum - Lehigh University 7
Applications of Image FusionConcealed weapon detectionDigital Camera Remote sensingIntelligent robotsMedical diagnosisDefect inspectionsurveillance
Introduction
06/24/2002 R.S. Blum - Lehigh University 8
(a) Image from CCD (b) Image from MMW (c) Fused image
Concealed Weapon Detection
*The source images were obtained from Thermotex Corporation
Our current focus (see our website)
06/24/2002 R.S. Blum - Lehigh University 9
Digital Camera
(a) Focus on the left (b) Focus on the right (c) Fused image (all-focus)
A past focus (see our website)
06/24/2002 R.S. Blum - Lehigh University 10
IntroductionCategory of Fusion
Signal-level fusionPixel-level fusion = Image Fusion (focus)Feature-level fusionDecision-level fusion (see our website: we have an extensive set of papers.)
06/24/2002 R.S. Blum - Lehigh University 11
IntroductionMethods of Image Fusion
Non-multiscale-decomposition-based (NMDB) methodsMultiscale-decomposition-based (MDB) methods
06/24/2002 R.S. Blum - Lehigh University 12
Methods of Image FusionNon-Multiscale-Decomposition-Based (NMDB) Methods
Pixel-level weighted averagingNonlinear methodOpponent Color Processing Artificial neural networkEstimation theory based methods
06/24/2002 R.S. Blum - Lehigh University 13
Pixel-level weighted averagingMethodology
To take the weighted average of the pixel intensity of the two source images
ExamplePrinciple component analysis (PCA): O. Rockinger, and T. Fechner, "Pixel-level image fusion: The case of image sequences", Proceedings of SPIE, vol. 3374, 1998Adaptive weight averaging (AWA): E. Lallier and M. Farooq, “A real time pixel-level based image fusion via adaptive weight averaging”, ISIF 2000
NMDB Fusion Methods
*Example methods chosen selected for CWD tests shown later.
06/24/2002 R.S. Blum - Lehigh University 14
NMDB Fusion MethodsNonlinear Method
MethodologySeparate images into low-pass & high-pass componentsadaptively modify each component, fuse, then addlow-pass: enhance each local luminance mean & fuse the by nonlinear mappinghigh-pass: fuse by weighted averaging
ExampleC. W. Therrien and W. K. Krebs, “An adaptive technique for the enhanced fusion of low-light visible with uncooledthermal infrared imagery”, ICIP 1997
06/24/2002 R.S. Blum - Lehigh University 15
NMDB Image Fusion MethodsOpponent Color Processing
MethodologyUse biological models of opponent-color processing to fuse low-light visible and thermal IR imagery, and render it in color
ExampleA. M. Waxman, M. Aguilar, R. A. Baxter, et.al. “Opponent-color fusion of multi-sensor imagery: visible, IR and SAR”, Proceedings of IRIS Passive Sensors, vol.1, pp.43-61, 1998
06/24/2002 R.S. Blum - Lehigh University 16
NMDB Image Fusion MethodsArtificial Neural Networks
MethodologyMotivated by the fusion of different sensor signals in biological systemsMulti-layer perceptron neural networks and pulse-coupled neural networks
ExampleT. Fechner and G. Godlewski, “Optimal fusion of TV and infrared images using artificial neural networks”, Proceedings of SPIE, vol.2492, pp.919-925, 1995J. M. Kinser, “Pulse-coupled image fusion”, Optical Engineering, vol.36, no.3, pp.737-742, 1997
06/24/2002 R.S. Blum - Lehigh University 17
NMDB Image Fusion MethodsEstimated Theory Based Methods
MethodologyImage formation model and the prior modelsMAP estimate, ML estimate, others. Bayes formula (s=scene, z=observations)
ExampleR. K. Sharma, T. K. Leen, and M. Pavel, “Bayesian sensor image fusion using local linear generative models”, Optical Engineering, vol.40, no.7, pp.1364-1376, July 2001
)()()|(
)|(zp
spszpzsp =
06/24/2002 R.S. Blum - Lehigh University 18
Methods of Image FusionMultiscale Decomposition-Based (MDB) Methods
IMST
MST
MST Image 1
Image 2
Fused Image
Fusion
Images/coefficents for different scalesMST = Multiscale transform
06/24/2002 R.S. Blum - Lehigh University 19
Multiscale-Decomposition-Based (MDB) Methods
Methods of Image Fusion
Z. Zhang, and R. S. Blum, “A categorization of multiscale-decomposition-based image fusion schemes with a performance study for a digital camera application”, Proceedings of IEEE, vol. 87, no. 8, pp. 1315-1326, 1999.
06/24/2002 R.S. Blum - Lehigh University 20
Laplacian Pyramid TransformDecomposition
Reconstruction
MDB Fusion Methods
IG =0
[ ] 21 ↓−∗= kk GG ω 1,...,0 −= Nk[ ] 214 ↑+∗−= kkk GGL ω 1,...,0 −= Nk
NN GG =ˆ
[ ] 21ˆ4ˆ
↑+∗+= kkk GLG ω
0ˆˆ GI =
1,...,0 −= Nk
06/24/2002 R.S. Blum - Lehigh University 21
MDB Fusion MethodsDiscrete Wavelet Transform
One stage of a 2-D DWT decomposition
06/24/2002 R.S. Blum - Lehigh University 22
MDB Fusion MethodsDiscrete Wavelet Transform
One stage of a 2-D DWT reconstruction
06/24/2002 R.S. Blum - Lehigh University 25
Non-Multiscale-Decomposition-Based (NMDB) Methods
Principle component analysis (PCA): O. Rockinger, and T. Fechner, "Pixel-level image fusion: The case of image sequences", Proceedings of SPIE, vol. 3374, 1998Adaptive weight averaging (AWA): E. Lallier and M. Farooq, “A real time pixel-level based image fusion via adaptive weight averaging”, ISIF 2000Pixel-level choosing maximum (MAX): simply to take the maximum value of the source images pixel by pixelNonlinear method (NONL): C. W. Therrien and W. K. Krebs, “An adaptive technique for the enhanced fusion of low-light visible with uncooled thermal infrared imagery”, ICIP 1997
Performance Testing for CWD :Visual and IR images
Pixel-level weighted averaging
NL
06/24/2002 R.S. Blum - Lehigh University 26
Multiscale-Decomposition-Based (MDB) Methods (similar with Laplacian & DWF)
DWT-1 : DWT, coefficient-based, single-scale grouping, choose max, no verification (2-level, average for the last LL band)DWT-2 : DWT, window-based (max), no grouping, choose max, no verification (2-level, average for the last LL band)DWT-3 : DWT, window-based (weighted average), no grouping, choose max, no verification (2-level, average for the last LL band)DWT-4 : DWT, window-based (weighted average), no grouping, weighted average, no verification (2-level, average for the last LL band)DWT-5: DWT, window-based (max), no grouping, choose max, consistency verification (2-level, average for the last LL band)
Performance Testing for CWD
06/24/2002 R.S. Blum - Lehigh University 27
Performance Testing for CWDDWF-1: DWF, window-based, modified Burt’s method, no grouping, consistency verification (DWT and Laplacian also)
Modified Burt’s method (A=visual image)
Attempt to maintain high resolution in the fused image by preserving the high-resolution in the visual image except where the two images are dissimilar
),,(),,(
),,(),,(2),,(
),,(),,(
;),,,(),,(),,(,),,,(
),,(),,(,),,,(
),,(
2
kjiSkjiS
knjmiCknjmiCkjiM
knjmiCkjiS
MifkjiCkjiSkjiSandMifkjiC
kjiSkjiSandMifkjiC
kjiC
BA
m nBA
AB
m n
ABA
BAABB
BAABA
c
+
++⋅++=
++=
><≤≥≤
=
∑∑∑∑
ααα
06/24/2002 R.S. Blum - Lehigh University 28
Evaluation Methods Requiring a Reference Image
Root Mean Square Error (RMSE)
Correlation
Peak Signal to Noise Ratio (PSNR)
2
1 1),(),(
1 ∑ ∑= =−=
N
i
M
jjiZjiR
NMRMSE
∑∑∑∑∑∑= == == =
===+
=N
i
M
jZR
N
i
M
jZ
N
i
M
jR
ZR
ZR jiZjiRRjiZRjiRRRR
RCORR
1 1,
1 1
2
1 1
2, ),(),(,),(,),(,2
−=
∑ ∑= =
2
1 1
2
10
),(),(1log10
N
i
M
jjiZjiR
NM
LPSNR
Evaluation Methods
06/24/2002 R.S. Blum - Lehigh University 29
Evaluation Methods Not Requiring a Reference Image
Standard Deviation (SD)
Entropy
Overall Cross Entropy (CE)
( ) ∑∑==
×=−=L
i
L
i
ihiiihii00
2)(,)(σ
∑=
−=L
i
ihihH0
2 )(log)(
∑=
=+
=L
i Z
XX ih
ihihZXCE
ZYCEZXCEZYXCE
02 )(
)(log)();(,
2);();(
);,(
Evaluation Methods
06/24/2002 R.S. Blum - Lehigh University 30
Experimental Tests
Test Images
img1 – Visual img2 – Visual img3 – Visual img4 – Visual img5 – Visual
img1 – IR img2 – IR img3 – IR img4 – IR img5 – IR
06/24/2002 R.S. Blum - Lehigh University 31
Experimental Tests
Test Images
img6 – Visual img7 – Visual img8 – Visual img9 – Visual
img6 – IR img7 – IR img8 – IR img9 – IR
06/24/2002 R.S. Blum - Lehigh University 32
Experimental Tests
Image Fusion Results (Image-1)
Visual IR Reference
PCA AWA Max Nonlinear Laplacian
06/24/2002 R.S. Blum - Lehigh University 33
Experimental Tests
Image Fusion Results (Image-1)
DWT-2 DWT-3 DWT-4 DWT-5 DWF-1
FSD Contrast Gradient Morphological DWT-1
06/24/2002 R.S. Blum - Lehigh University 34
RMSE
00.050.1
0.150.2
0.250.3
0.350.4
0.45
A B C D E F G H I J K L M N O VI IR
CORR
00.10.20.30.40.50.60.70.80.9
1
A B C D E F G H I J K L M N O VI IR
PSNR
02468
101214161820
A B C D E F G H I J K L M N O VI IR
Image - 1SD
05
101520253035404550
A B C D E F G H I J K L M N O VI IR
Entropy
6.6
6.7
6.8
6.9
7
7.1
7.2
7.3
7.4
A B C D E F G H I J K L M N O VI IR
CE
0
0.2
0.4
0.6
0.8
1
1.2
A B C D E F G H I J K L M N O
06/24/2002 R.S. Blum - Lehigh University 35
Visual EvaluationFor most of the images, all of the methods are acceptable except method j (DWT-1)Method c (Pixel-level maximum) and method o (DWF-1) are best for producing a detailed view of the person’s face and a recognizable view of the object of interest (the gun)
Discussion
06/24/2002 R.S. Blum - Lehigh University 36
Discussion
Quantitative Evaluation
Often differs greatly from the visual evaluation making the use of these measures questionable. More work needed on automatic evaluation methods.
06/24/2002 R.S. Blum - Lehigh University 37
A Signal Processing Approach
Motivation
Image formation model
EM fusion algorithm
Experiments and results
Summary
06/24/2002 R.S. Blum - Lehigh University 38
MotivationAttempt to capitalize on the well developed theories of statistics and estimation theoryEstimate true scene from images with different sensor typesDifficult to provide a well-justified, fixed statistical model for imagesCan solve this with an adaptive signal processing approach for a general modelCan get a particularly efficient processing structure
06/24/2002 R.S. Blum - Lehigh University 39
Image Formation ModelImage Formation Model
indexes the sensors denotes the pyramid
coefficient, is the pixel coordinate, m is the level z is the sensor images is the true scene image
is the sensor selectivity factor
is the non-Gaussian distortion
nFor ease of explanation we present a simple model.nWe have also studied more complicated models.
MST Pyramid
m=M
m=2
m=1
y
x
)()()()( jjsjjz iii εβ +=
qi ,...,1=),,( myxj ≡
),( yx
0,1±=β
ε
06/24/2002 R.S. Blum - Lehigh University 40
Image Formation Model (Cont.)is the random distortion modeled using a K-
term mixture of Gaussian probability density functions (pdfs) as
Can represent Gaussian or non-Gaussian distortion
1λ
21σ
2λ
22σ
Two terms
v , get Gaussian
vGroup data into two sets
v = Prob (sample from set i)
11 =λ 02 =λ
iλ
)( jiε
∑=
−=
K
k ik
i
ik
ikij jj
jjjf
i1
2,
2
2,
,)( )(2)(
exp)(2
1)())((
σε
πσλεε
06/24/2002 R.S. Blum - Lehigh University 41
EM Fusion AlgorithmEM used to estimate
Start with an initial set of estimates ( , …) and observed data ( , …). Then produce updated estimates ( , …) each iteration.Estimates converge to maximum likelihood estimates (typically need only 3 to 5 iterations).
)1(1z
jijjjjjjs iKiiKii ,allfor)(),...,();(),...,();();( 2,
2,1,,1 σσλλβ
)1(s
)1('s
06/24/2002 R.S. Blum - Lehigh University 42
Using a Window of dataTo estimate parameters for coefficient j, use a window of coefficients around j
Coefficient j
window
h
h
vPyramid level m
vWindow contains coefficients
v Number coefficient in window
v and the parameters of distortion pdf constant over the window
hhL ×=
Ll ,...,1=
β
06/24/2002 R.S. Blum - Lehigh University 43
Developing Iterative Estimation Approach
EM algorithm produces a sequence of estimates that increase in likelihoodAt each step you must find the estimate that maximizes a “likelihood-like” cost functionSkip details to give the iterative algorithm results Use SAGE algorithm: one variable at a time
06/24/2002 R.S. Blum - Lehigh University 44
Iterative Fusion ProcedureFirst compute (classifier)
Update . Choose from set 0,-1,+1 to maximize
Q is the “likelihood-like” function we need to maximize. Note Q is computed using the window
This procedure gives estimates for coefficient j at the center of window.
( )
( )LlqiKk
lslz
lslz
lzgK
p ip
ii
ip
ip
ik
ii
ik
ik
iilk ,...,1;,...,1;,...,1,
2)()(
exp2
2)()(
exp2
)]([
12
,
2
2,
,
2,
2
2,
,
, ===
−−
−−
=
∑= σ
β
πσ
λ
σβ
πσ
λ
iβ
( ) ( ) [ ]∑∑∑= = =
⋅
−+−=
q
i
L
l
K
kiilk
ik
iiik lzg
lslzQ
1 1 1,2
,
2'2, )(
)()(ln
21
σβ
σ
06/24/2002 R.S. Blum - Lehigh University 45
EM Fusion Procedure (Cont.)Update true scene (Generalization of Gaussian when )
Update and
Processing similar to radial basis function neural networks but comes from estimation theory
1)]([, ≠lzg iilk)(ls
ik ,λ 2,ikσ
[ ]
[ ] Lllzg
lzglz
lsq
i
K
k ik
iilki
q
i
K
k ik
iilkii
,...,1,)(
'
)(')(
)('
1 12,
,2
1 12,
,
==
∑∑
∑∑
= =
= =
σβ
σβ
[ ]∑=
===L
liilkik qiKklzg
L 1,, ,...,1;,...,1,)(
1'λ
( ) [ ]
[ ]qiKk
lzg
lzglslz
L
liilk
L
liilkii
ik ,...,1;,...,1,)(
)()('')('
1,
1,
2
2, ==
−=
∑
∑
=
=
βσ
06/24/2002 R.S. Blum - Lehigh University 46
Initialization for EM Fusion Algorithm
Initialize as the weighted average of sensor images:
are determined by salience measure:
The salience measure:
v is the weight for each coefficient around
∑=
==q
iii Lllzlwls
1
,...,1)()()(
)(lwi
qilllwq
jjii ,...,1)(/)()(
1
=ΩΩ= ∑=
qimyyxxzyxplx y
ii ,...,1),','()','()(2
2'
2
2'
2 =++=Ω ∑ ∑−= −=
),,( myxl =
)(ls
)','( yxp
06/24/2002 R.S. Blum - Lehigh University 47
Initialization for EM Fusion Algorithm (Cont.)
InitializeInitialize and
Initialize
qii ,...,1,1 ==β
qiKiKi ,...,1),1/(2.0,,2 =−=== λλ L
[ ]∑∑==
−=L
li
K
kikik Llslz
1
2
1
2,, /)()(σλso that
8.0,1 =iλ
Kkqiikik ,...,2;,...,1,2,1
2, === −γσσ
06/24/2002 R.S. Blum - Lehigh University 48
Force of coefficient j to be consistent with its
neighboring coefficients ( l = 1,…, L) in the same
pyramid level m
Define and
Pick to minimize
Can minimize component by component to simplify
Consistency Verification
∑=
−L
llj
1
ßßjß
β
( )jqjj ββ ,...,1=ß ( ) Lllqll ,...,1,,...,1 == ββß
06/24/2002 R.S. Blum - Lehigh University 49
EM Fusion for CWD (1)
Visual Image MMW Image
EM Fusion Averaging Selecting Maximum Laplacian Fusion
vImage size: 256×256
vNumber of parameter levels: 5
vGaussian mixture terms: 2
vWindow size: 3×3
06/24/2002 R.S. Blum - Lehigh University 50
Visual Image IR Image
EM Fusion Averaging Selecting Maximum Laplacian Fusion
vImage size: 256×256
vNumber of parameter levels: 7
vGaussian mixture terms: 2
vWindow size: 5×5
EM Fusion for CWD (2)
06/24/2002 R.S. Blum - Lehigh University 51
EM Fusion for autonomous landing guidance (ALG)
Long Wave Medium Wave
EM Fusion Averaging Selecting Maximum Laplacian Fusion
vImage size: 233×233
vNumber of parameter levels: 3
vGaussian mixture terms: 2
vWindow size: 5×5
06/24/2002 R.S. Blum - Lehigh University 52
Research on Optimizing the Procedure
Number of MST pyramid levels has slight effect on the fused result Window size should be chosen carefully: 3×3 or 5×5 is best in most applicationsCan improve fusion using multi-frame sensor images
06/24/2002 R.S. Blum - Lehigh University 53
SummaryPresented a statistical signal processing based image fusion methodApplied the EM fusion method to concealed weapon detection and autonomous landing guidance applications with good results
06/24/2002 R.S. Blum - Lehigh University 54
New Image Registration Algorithm for Image Fusion
Details in:
Z. Zhang and R. S. Blum, ``A Hybrid Image Registration Technique for a Digital Camera Image Fusion Application'', Information Fusion, pp. 135-149, June 2001.
Z. Zhang and R. S. Blum, ``Image registration for Multi-focus image fusion'', SPIE AeroSense, Conference on Battlefield Digitization and Network Centric Warfare (4396-39), Orlando, FL, April 2001.
06/24/2002 R.S. Blum - Lehigh University 55
Conclusion
We hope this talk was useful.We are seeking out collaborators. Working with people who have practical knowledge of real world applications of image fusion is highly desirable to us.Please contact us to initiate collaboration: [email protected]
06/24/2002 R.S. Blum - Lehigh University 56
ReferencesJ. Yang, R. S. Blum,`` A Statistical Signal Processing Approach to image fusion for concealed weapon detection'', IEEE International Conference on Image Processing, Rochester, NY, Sept. 2002.Z. Xue, R. S. Blum and Y. Li , ``Fusion of visual and IR images for concealed weapon detection'',
International Conference on Information Fusion (Fusion 2002), Annapolis, Maryland, July 2002. Z. Zhang and R. S. Blum, ``A Hybrid Image Registration Technique for a Digital Camera Image Fusion Application'', Information Fusion, pp. 135-149, June 2001.Z. Zhang and R. S. Blum, ``A hybrid image registration technique for a digital camera image fusion application'', SPIE AeroSense, Conference on Battlefield Digitization and Network Centric Warfare (4396-39), Orlando, FL, April 2001. Z. Zhang, and R. S. Blum, “A categorization of multiscale-decomposition-based image fusion schemes with a performance study for a digital camera application”, Proceedings of IEEE, vol. 87, no. 8, pp. 1315-1326, 1999.Z. Zhong and R. S. Blum, ``Extraction of 3-D coordinates from fusion of Omnicamera images,'' Asilomar Conference on Signals, Systems, and Computers, pp. 397-401, Monterey, CA, Nov. 1999. Z. Zhang and R. S. Blum, "Image fusion for a digital camera application" Asilomar Conference on Signals, Systems, and Computers, pp. 603-607, Monterey, CA, Nov. 1998.R. S. Blum, ``Decision and data fusion research at Lehigh University,'' National Symposium on Data Fusion (sponsored by AFOSR) and held at Georgia Tech. Atlanta, Georgia, March 1998. Z. Zhong and R. S. Blum, ``A region-based image fusion scheme for concealed weapon detection,'‘ 30th Annual Conference on Information Sciences andSystems, Johns Hopkins University, pp. 168-173, Baltimore, MD, March 1997.Z. Zhang and R. S. Blum, ``On estimating the quality of noisy images,'' 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 2897-2901, Seattle, WA, May 1998.Z. Zhang and R. S. Blum, "Multisensor image fusion using a region-based wavelet transform approach", Proc. of the DARPA IUW, pp. 1447-1451, 1997.