IMAGE FUSION PERFORMANCE ANALYSIS BY ANOVA · image fusion process is to create a good...
Transcript of IMAGE FUSION PERFORMANCE ANALYSIS BY ANOVA · image fusion process is to create a good...
IMAGE FUSION PERFORMANCE ANALYSIS BY ANOVA
NATALIA R. DOS SANTOS
Divisão de Engenharia Mecânica, Departamento de Mecatrônica, Instituto Tecnológico de Aeronáutica,
12228-900 São José dos Campos, SP, Brasil
E-mails: [email protected]
ELDER M. HEMERLY
Divisão de Engenharia Eletrônica, Departamento de Sistemas e Controle, Instituto Tecnológico de Aeronáutica,
12228-900 São José dos Campos, SP, Brasil
E-mails:[email protected]
Abstract Image fusion techniques play an important role in several fields of knowledge, due to its ability to concentrate re-
dundant and complementary information, without distorting them, into one single image, which is more appropriate for human or
machine processing. There are many ways to perform the image fusion procedure, and several variables to be considered in their
design. The goal of this work is to present a systematic performance analysis of image fusion methods, with statistical justifica-tion. A consistent method, based on ANOVA (analysis of variance) concepts, is employed to compare results from different in-
puts of the fusion procedure, by using objective performance criteria for the fused image.
Keywords image fusion, wavelet transform, ANOVA, image fusion performance criteria, multi scale decomposition.
Resumo Técnicas de fusão de imagem cumprem um papel importante em diversas áreas do conhecimento, devido à premissa
de concentrarem informações redundantes ou complementares de duas ou mais fontes, sem que haja distorção, a fim de produzir
uma imagem única que seja mais apropriada ao processamento por seres humanos ou máquinas. Há muitas maneiras de se efetuar fusão de imagens, e diversas variáveis a serem consideradas no projeto. O objetivo deste trabalho está em sistematizar a análise
dos resultados de fusões, usando justificativa estatística, ao definir um método consistente baseado nos conceitos da ANOVA
(análise de variância), para compará-los quando gerados por diferentes variáveis de projeto de fusão, usando critérios objetivos de desempenho para a imagem final.
Palavras-chave fusão de imagens, transformada wavelet, ANOVA, critérios de desempenho para fusão de imagens, decompo-
sição em múltiplas escalas.
1 Introduction
Image fusion methods have been consistently
used in several applications, such as in geology
(disaster monitoring), medicine (PET/CT machines),
remote sensing (satellite imaging), military (target
recognition), safety &security (law enforcement,
surveillance), industrial (quality inspection) (Li, et
al., 1994) (Cheng, et al., 2008) (Yang & Blum,
2002). This wide range of applicability has attracted
much interest to this area, and several researches
have been conducted in the last decades, aimed at
finding techniques to improve the fusion results
(Choi, et al., 2010).
The main purpose of the image fusion techniques
is to produce a single image, using complementary
or redundant information from two or more sources,
which is more suitable for human or machine per-
ception and treatment (Zhang & Blum, 1999). The
expected results are usually the improvement in the
quantity and/or quality of information and simulta-
neously the decrease in the amount of data to be
analyzed (Petrovié & Xydeas, 2004), manually or
automatically. The images to be used in the fusion
processing may come from different sensors (multi-
sensor), or from the same sensor (multifocus appli-
cations for instance), but containing different fea-
tures. Practical examples are shown in Figure 1-3.
Figure 1: Example of medical application. Primary tumor is not well seen on noncontrast CT (A) but is clearly delineated on PET
(B) and PET/CT fusion (C) images (JNM, 2014)
Figure 2: Example of military application in target recognition:
a) IR sensor showing physical edges; b) EO sensor-objects cov-
ered by smoke and c) Fused Image – physical features and smoke represented. (SPIE, 2014)
Anais do XX Congresso Brasileiro de Automática Belo Horizonte, MG, 20 a 24 de Setembro de 2014
181
Figure 3: Multifocus source images fusion result. a) focus on left;
b) focus on right and c) fused image.
The Discrete Wavelet Transform (DWT) is a
suitable tool for the image fusion application and is
the most used technique in recent works (Amolins,
et al., 2007). The procedure involves decomposing
the source images in their multiresolution wavelet
representations, and then manipulating the obtained
data to generate the final image. The key to any
image fusion process is to create a good
multiresolution decomposition by choosing its
adequate variables (wavelet type and number of
decomposition levels) and then defining a fusion
rule that performs well. See Figure 4 for details.
Figure 4: Basic scheme for DWT image fusion.
Results are usually registered in tables and the
assessment of the data is commonly done by visual
comparison of the results (Yang, 2011) (Zhang &
Blum, 1999) (Gonzalo & de la Cruz, 2004) (Zeng, et
al., 2012) (Luo, et al., 2012) or by a simple graphical
representation without any further mathematical
background. In several other fields of knowledge,
this background is a must and the systematization of
analysis results using statistical justification is the
default procedure, especially when dealing with
biological samples. This can be seen in many medi-
cal studies (Koehler, et al., 2014).
This paper introduces the statistical parameteri-
zation on the data analysis. An experimental
procedure for evaluating the performance of image
fusion methods is proposed, by considering several
combinations of wavelet types, number of decompo-
sition levels, source images and fusion rule methods.
This data analysis relies on the computational im-
plementation of the well-known ANOVA
(Montgomery, 2001) concepts for performing the
required analysis. The main goal of this paper is
then to define a consistent method for analyzing
image fusion results. An automated statistical ap-
proach is used for generating and evaluating the
outputs produced by several changes in input pa-
rameters of a DWT based fusion. This approach
allows quick and reliable selection of methods and
parameters for the desired fusion application, and
permits incremental changes to cope with new de-
velopments. This flexibility is due to the fact that
new performance criteria and processing methods
themselves can be adapted as variables to the analy-
sis. This way a bigger set of data can be analyzed at
once and the results statistically compared and justi-
fied.
2. Image fusion and statistical analysis
2.1 Wavelet based image fusion
The DWT is nowadays the most used tool for
applications in image fusion and image compres-
sion. Images are two dimension objects, computa-
tionally represented by matrices where each element
defines a pixel whose properties are associated to
this element value.
The basic principle is to perform the signal
analysis by decomposing it into its frequency com-
ponents, like it is done when transformed methods
are used, such as the Fourier Transform. The main
difference lies on the reference used (Gonzalo & de
la Cruz, 2004). In contrast to the Fourier methods,
where the basic functions are sinusoids, the basic
element are now small waves (the wavelets), whose
main characteristics are variable frequency, finite
duration, mean value equal to zero and usually
assymetry.
When compared to the usual Fourier transform,
the wavelet transform has a striking advantage: it is
capable of mapping both the frequency and temporal
position of the original signal, so that every resolu-
tion is tied to its scale. In short, the temporal infor-
mation is not lost in the transformation process.
This approach has notable advantages for signals
that contain several peaks and discontinuities, which
is the case of images. An illustration for this proper-
ty is provided in Figure 5.
Figure 5: Time-frequency/scale representation of the Wavelet
Transform. (MATLAB, R2010A)
Anais do XX Congresso Brasileiro de Automática Belo Horizonte, MG, 20 a 24 de Setembro de 2014
182
The use of the Wavelet Transform for image
processing is based on the MRA methods (Multi
Resolution Analysis), technique in which the signal
is decomposed into different resolutions. This ap-
proach follows the principle that distinct elements
have their representation more suited to a specific
resolution level. The level represents a certain reso-
lution and the decomposition creates 4 new groups
on every level, being one for the approximation (low
frequency) coefficients and another 3 for the detail
(high frequency) coefficients, but each one operates
on a different direction (horizontal, vertical and
diagonal) (Gonzalez & Woods, 2002). See Figure 6.
Figure 6: Multiresolution decomposition structure for a 3 level decomposition.
The high frequency bands of an image provide
the details of the scene. The corresponding coeffi-
cients depict strong intensity variations and preserve
salient information on the images, and are character-
ized by high absolute values. On the other hand, the
low frequency bands of an image supply the basic
and coarse information of the depicted scene. The
corresponding coefficients representation is equiva-
lent to a low resolution view of that image and gen-
erally has small absolute values, being usually re-
ferred to as “approximation” coefficients.
The fusion method using DWT relies on three
main steps: 1) Decompose the source images into
their multi resolution scales, generating a matrix of
coefficients for each of the sources. This is done by
filtering and down sampling the initial data. This
first step supplies the high and low frequency coef-
ficients mentioned before. 2) The second step is the
subject which attracts the largest attention in the
literature: it deals with the rules for actually creating
a composite image, by using algorithms to merge
or choose the coefficients from each image source
and 3) The new and single coefficients matrix that
was created goes through and inverse wavelet trans-
form, that finally generates the fused image.
2.2 ANOVA Analysis
According to statistics premises, one result is
only different from another if, after analyzing the
variance of the samples, considering or not the in-
fluence factors as random effects, the difference
between the mean values is within a pre-set proba-
bility of happening, which is called statistically
significant. (Montgomery, 2001).
3 Methods
Tests were done changing the MSD variables
(wavelet family and number of levels) and the fu-
sion method. Grey level images were used, so as to
simplify the experiment. The chosen performance
metrics were RMSE (root mean square error), entro-
py and overall cross entropy. See Table 1 for details.
Table 1: Variables tested on the experiment
Factors Levels Values
Sample 1 Cameraman.tiff
Wavelet 5 'db8', 'db2', 'haar', 'sym2',
'sym6'
Fusion
Rule
2 Joint (mean-mean),
Individual (Yang, 2011)
Number
of levels
4 2, 3, 5, 8
In this work the fusion procedure was imple-
mented in MATLAB (MATLAB, R2010) and the
core function used was “wfusimg”, from the image
processing package. This function uses as variables
all the factors that were exercised in this experiment.
Two fusion rules were investigated: 1) a joint rule
for all wavelet decomposition coefficients , based on
averaging them, here called mean-mean method,
which is very simple and intuitive, and 2) frequency
band individualized rules as proposed by (Yang,
2011), which is quite promising, since it is able to
capture relevant information from different signal
frequency ranges.
The mean-mean rule is a method where the co-
efficients of both MSD matrixes are averaged in
order to compose the fused matrix.
The rule proposed by (Yang, 2011) considers
high and low frequency information given by (1)
and (2) respectively
( ) { ( ) ( ) ( )
( ) ( ) ( ) ( )
( ) ( ) ( ) ( ) ( ) ( )
where Dx and Dy are coefficients on the multiscale
representation for the source images and Dz for the
fused image. Dz is chosen based on the maximum
variance in a window for each pixel in (1) and is
selected by Wx and Wy in (2) whose values can be
1or 0 according to maximum edge detection rule.
Anais do XX Congresso Brasileiro de Automática Belo Horizonte, MG, 20 a 24 de Setembro de 2014
183
3.1 The source image
Multifocus source images refer to the images
taken from a unique scene using the same sensor,
but with different quality in each portion of the
image, due to changes in the focus. This is equiva-
lent to what happens when shots are taken by a cam-
era, with focus in two parts of the image. On the
other hand, multisensor images refer to a scene
captured by different sensors such as infra-red, elec-
tro-optical, magnetic resonance, etc.
In multifocus applications, it is possible to con-
catenate the portions of the source images whose
focus is adequate and generate a final image with
desired quality. This is an artisanal process though,
which can be easily simulated by blurring different
parts of a single image. The advantage of using a
single reference image lies on the fact that it pro-
vides a “control group”, also called ground truth,
which simplifies performance evaluation.
In this work input images were created by blur-
ring the right and left sides of the original image, by
using a symmetric Gaussian low pass filter. Two
well know images were used: the Lena and Camer-
aman, both of size 256 x 256, gray level on TIFF
format.
3.2 Performance Criteria
In the past the results of image fusion were
commonly evaluated by visual assessment (Li, et al.,
1994). The quantitative evaluation of performance is
a complicated issue due to the fact that the ideal
result is usually unknown (Zhang & Blum, 1999),
especially when working with multisensor image
fusion which appears in most real applications
(Zheng & Qin, 2009). Multifocus applications and
its simulations, as done in this work, can be better
evaluated because it allows comparison to the
ground-truth reference. Performance criteria are also
classified according to these reference requirements.
Three complementary performance indices are used
here: RMSE, Entropy and Overall Cross Entropy
(OCE) shown in Equations 3-6.
(
∑ ∑( ( ) ( ))
)
( )
where xR is the reference image (ground-truth),
xF is the fused image and M and N represent its size.
The lower the RMSE value, the better the fusion.
∑ ( )
where P is the probability of a particular gray
level occurrence on that image, obtained from its
histogram, and L the number of gray levels. The
Entropy represents an index for reflecting the
amount of information in the image, which is statis-
tically measured by the randomness of the infor-
mation. The higher, the better.
( ) ( )
( )
∑
( )
where P and Q are histograms of source and fused
image. A,B and F are the source and fused images,
respectively. This index is a measure for differences
to the final image, therefore, the smaller, the better.
RMSE is a good performance indicator for stud-
ies with simulated databases because it takes the
advantage of using the reference image, being a very
straight way of comparing them. The entropy and
overall cross entropy methods are based on the
physical meaning of the image, thus, not requiring a
reference. The drawback is that they are susceptible
to particularities of each sample, such as amount of
details on image.
4. Results
The plots in Figure 7-9 show the mean error values
for the three performance methods chosen in section
3.2.
Figure 7:RMSE versus each tested factor
2.0
2.5
3.0
3.5
4.0
Means of Factors x RMSE
Factors
RM
SE
db2 db8
haar
sym2 sym6
individual
joint
2
3 5 8
Wavelet Fusion_Rule Nb_Levels
Anais do XX Congresso Brasileiro de Automática Belo Horizonte, MG, 20 a 24 de Setembro de 2014
184
Figure 8: Overall Cross Entropy versus each tested factor
Figure 9: Entropy versus each tested factor
The means showing divergent results for each
performance index indicate that they actually meas-
ure different things. This is an important issue that
expresses the importance of adequately choosing a
performance criterion to the desired application of
the fused image.
The ANOVA also showed different results for
each performance metric as seen in Table 2. Howev-
er, some factors are clearly significant in all three
analyses: the fusion rule and wavelet type, as ex-
pected. These results are also very useful to under-
stand the behavior of interactions among the factors,
which informs the effects of a certain parameter
beyond its own influence and ultimately indicates
how difficult it is to optimize the response under the
parameter variation.
It should be stressed, however, that the main
goal here is not to select the best method, but actual-
ly how to compare performance of different meth-
ods, with statistical significance.
Table 2
Factors RMSE Entropy OCE
Wavelet *** *** ***
Fusion Rule *** *** ***
# Levels *** ** ***
Fus Rule : #Levels *** * ***
Wavelet : #Levels *** ** x
Wavelet : Fus Rule x ** ***
Model Adequacy 99.9% 98.7% 97.7%
Significance codes:
0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘x ’ 1
The evaluation of the ANOVA premises of in-
dependency, normality of distribution and homosce-
dasticity of the residuals indicate that the analysis is
valid, since none of them is considered to be violat-
ed, according to the graphical evaluations shown in
Figure 10.
Figure 10: ANOVA premises analysis
A closer analysis of the RMSE results, by look-
ing at the interaction plots in Figure 11, indicates
considerably different results for each fusion rule as
the number of levels increases. There is no effect for
the averaging rule and a clear tendency of improving
the results until 5 levels of decomposition for the
individual fusion rule. This graphical analysis also
leads to the conclusion that the significant response
of the levels found in ANOVA is only present in
individual fusion rule, which finally indicates an
interaction between these two factors (fusion rule
and number of levels).
7.0
1
7.0
2
7.0
3
7.0
4
Means of Factors x Entropy
Factors
En
tro
py
db2
db8 haar
sym2
sym6
individual
joint
2 3
5 8
Wavelet Fusion_Rule Nb_Levels
7.8
84
7.8
86
7.8
88
7.8
90
Means of Factors x Overall Cross Entropy
Factors
Ove
rall
Cro
ss E
ntr
op
y
db2
db8
haar
sym2
sym6
individual
joint
2
3
5 8
Wavelet Fusion_Rule Nb_Levels
0 10 20 30 40
-0.0
6
0.0
2
Independency
Order
Resid
ua
ls
-2 -1 0 1 2
-0.0
6
0.0
2
q-q plot-Normality
Predicted
Resid
ua
ls
Histogram-Normality
Residuals
Fre
que
ncy
-0.05 0.00 0.05
0
4
8
12
1.5 2.5 3.5
-0.0
6
0.0
2
Homoscedasticity
Fitted Residuals
Resid
ua
ls
Anais do XX Congresso Brasileiro de Automática Belo Horizonte, MG, 20 a 24 de Setembro de 2014
185
Figure 11: Interaction plot of number of levels and Fusion rule
5. Conclusion
A statistical procedure was proposed in this
work for comparing performance of image fusion
methods. The ANOVA technique was implemented
and showed consistent results: it supplies technical
and mathematical backgrounds for the performance
analysis of fused images. The influence of several
design parameters can be investigated, thus reducing
considerably the amount of data necessary to obtain
conclusive and practical results, with statistical
meaning.
6. References
Amolins, K., Zhang, Y. & Dare, P., 2007. Wavelet
based image fusion techniques-An
introduction,review and comparison. ISPRS
Journal of Photogrammetry & Remote Sensing.
Cheng, S., He, J. & Lv, Z., 2008. Medical Image of
PET/CT Weighted Fusion Based on Wavelet
Transform. IEEE 978-1-4244.
Choi, K., Kim, C. & Beon Ra, J., 2010. Infrared
Image enhancement based on aligned high
resolution visible. IEEE-ICIP.
Gonzalez, R. C. & Woods, R. E., 2002. Digital
Image Processing. Second ed. New Jersey:
Prentice-Hall Inc.
Gonzalo, P. & de la Cruz, J. M., 2004. A wavelet-
based image fusion tutorial. 0031-3203 Patttern
Recognition Society.
JNM, T. j. o. N. M., 2014. jnm.snmjournals.org.
[Online]
Available at:
http://jnm.snmjournals.org/content/47/5/755.fig
ures-only
[Accessed 20 March 2014].
Koehler, G. et al., 2014. Insulin degludec is not
associated with a delayed or diminished
response to hypoglycaemia compared with
insulin glargine in type 1 diabetes: a double-
blind randomised crossover study.
Diabetologia, pp. 40-49.
Lehigh University, B., 2014. ece.lehigh.edu.
[Online]
Available at:
http://www.ece.lehigh.edu/SPCRL/IF/image_fu
sion.htm
[Accessed 20 March 2014].
Li, H., Manjunat, B. & Mitra, S., 1994. MULTI-
SENSOR IMAGE FUSION USING THE
WAVELET TRANSFORM. IEEE.
Luo, B. et al., 2012. Decision-based fusion for
pasharpening of remote sensing images. IEEE
Explore.
Mallat, S. G., 1989. A Theory for Multiresolution
Signal Decomposition: The Wavelet
Representation. IEEE Transactions on Pattern
Analysis and Machine intelligence.
Montgomery, D. C., 2001. DESIGN AND
ANALYSIS OF EXPERIMENTS. Fifth Edition
ed. s.l.:s.n.
Petrovié, V. S. & Xydeas, C. S., 2004. Gradient-
Based Multiresolution Image Fusion. IEEE
Transactions on Image Processing.
Soliman, R., Amin, M. & Abd El-Samie, F., 2012.
Enhanced Fusion for Infrared and Visible
Images. Cairo, s.n.
SPIE, I. S. f. O. a. P., 2014. spie.org. [Online]
Available at: http://spie.org/x32828.xml
[Accessed 20 March 2014].
Yang, J. & Blum, R., 2002. A stattistical signal
processing approach to image fusion for
concealed weapon detection. IEEE ICIP.
Yang, Y., 2011. Multiresolution Image Fusion
Based on Wavelet Transform by Using a Novel
technique for Selection Coefficients. Journal of
Multimedia.
Zeng, Y., van Genderen, J., Zhang, J. & Wang, G.,
2012. A22Evaluation Criteria for Image Fusion
Performance in Different Applications. 2012
International Conference on Industrial Control
and Electronics Engineerig.
Zhang, Z. & Blum, R. S., 1999. A Categorization of
Multiscale-Decomposition-Based Image Fusion
Schemes with a Performance Study for a Digital
Camera Application. PROCEEDINGS OF THE
IEEE, August.
Zheng, Y. & Qin, Z., 2009. Objective image fusion
quality evaluation using structural similarity.
Tsighua science and te chnology.
1.5
2
.0
2.5
3
.0
3.5
4
.0
Factors Interaction Plot
Number of Levels
RM
SE
2 3 5 8
Fusion_Rule
joint individual
Anais do XX Congresso Brasileiro de Automática Belo Horizonte, MG, 20 a 24 de Setembro de 2014
186