4 satellite image fusion using fast discrete

Satellite Image Fusion using Fast DiscreteCurvelet Transforms

C.V.Rao, J.Malleswara Rao, A.Senthil Kumar, D.S.Jain, V.K.DadhwalNational Remote Sensing Centre, Indian Space Research Organization, Hyderabad-500037, India

Email: rao [email protected]

Abstract—Image fusion based on the Fourier and wavelettransform methods retain rich multispectral details but lessspatial details from source images. Wavelets perform well onlyat linear features but not at non linear discontinuities becausethey do not use the geometric properties of structures. Curvelettransforms overcome such difficulties in feature representation. Inthis paper, we define a novel fusion rule via high pass modulationusing Local Magnitude Ratio (LMR) in Fast Discrete CurveletTransforms (FDCT) domain. For experimental study of thismethod Indian Remote Sensing (IRS) Resourcesat-1 LISS IVsatellite sensor image of spatial resolution of 5.8m is used as lowresolution (LR) multispectral image and Cartosat-1 Panchromatic(Pan) of spatial resolution 2.5m is used as high resolution (HR)Pan image. This fusion rule generates HR multispectral image at2.5m spatial resolution. This method is quantitatively comparedwith Wavelet, Principal component analysis (PCA), High passfiltering(HPF), Modified Intensity-Hue-Saturation (M.IHS) andGrams-Schmidth fusion methods. Proposed method spatiallyoutperform the other methods and retains rich multispectraldetails.

Keywords—Image Fusion, Fast Discrete Curvelet Transforms,Local Magnitude Ratio (LMR).

I. INTRODUCTION

Image fusion integrates the multisensor data to create afused image containing high spatial, spectral and radiometricresolutions. In remote sensing, image fusion is most valuabletechnique for utilization of multisensor, multispectral at variousresolutions of earth observation satellites [10]. Spatial resolu-tion plays a vital role to delineate the objects in the remotesensing image. It is easy to interpret the features with highspatial resolution [20] image with multispectral informationthan the single high resolution Pan image. Image fusionenhance the spatial, spectral and radiometric [11] resolutionsof images.

There are several satellite image fusion techniques butspatial and spectral details retention simultaneously is a tradeoff. F.Nencini et al. [5] proposed a fusion method based oninterband structure model (IBSM) in first generation curvelettransform domain. The method uses Quick-bird and Ikonosmultispectral and Pan images. The experimental results shownthat, the method slightly better than the Atrous WaveletTransform (ATWT) and outperform Grams-Schmidt spec-tral sharpening method. Ying Li et al. [6] proposed a FastDiscrete Curvelet Transform (FDCT) based remote sensingimage fusion. The method uses Synthetic Aperture Radar(SAR) and Thematic Mapper(TM) images for fusion. Thiswork concluded that FDCT based fusion method retain goodspatial details and simultaneously preserve the rich spectralcontent compared with Discrete Wavelet Transform (DWT)

and Intensity-Hue-saturation (IHS). Arash Golibag et al. [7]focuses on region based image fusion using linear dependencydecision rule based on Wronskian determinant. The methoduses multispectral Landsat and IRS Pan images for fusion.Shutao Li et al. [8] have done multifocus image fusion bycombining curvelet and wavelet transform. Cai Xi [9] proposedFDCT based image fusion by using Support Vector Machines(SVM) and Pulse Coupled Neural Network (PCNN) to narratethe good sensitivity to boundaries of objects, concluded thatthe method has better performance than FDCT.

Fast Fourier and wavelet transform based image fusionmethods retain better spectral characteristics but represent poorspatial details in fused images. Objective of this paper isto develop a method, which retains better characteristics ofboth spatial and spectral qualities of source images. Wavelettransforms do not represent the curved objects as in HRPan image. Curvelet transforms overcome such difficulties ofwavelet. Over a period, curvelet transforms are evolved intwo generations, such as first generation curvelet transformsand second generation curvelet transforms named as Fast Dis-crete Curvelet Transforms(FDCT).First genereation curvelettransforms computational complexity is more to computethe curvelet coefficients [5]. To overcome these difficultiesEmmanuel J.Candes [1] developed FDCT. FDCT representslinear-edges and curves accurately than any other mathematicaltransforms.

II. FAST DISCRETE CURVELET TRANSFORMS (FDCT)

Multi resolution ideas like wavelet [1] and related ideasled to convenient tools to navigate through large datasets,to transmit compressed data rapidly, to remove noise fromsignals and images, and to identify crucial transient featuresin such datasets. Wavelets perform well only at representingpoint singularities, since they ignore the geometric propertiesof structures and do not exploit the regularity of edges.Curvelet transform was proposed [1] in order to overcome thedrawbacks of conventional two-dimensional discrete wavelettransform. In the two-dimensional (2D) case, the curvelettransform allows almost optimal sparse representation of ob-jects with C2 singularities.

Emmanuel.J.Candes et.al.,[1] developed and described thecontinuous curvelet transform as follows. Let aj = 2−j , j ≥ 0

be the scale, θj,l = πl2−�j/2�2 with l = 0, 1, ..., 4.2�j/2� − 1

be the equidistant sequence of rotation angles. Here �x�denotes the smallest integer being greater than or equal to x,bj,lk = bj,lk1,k2

= R−1θj,l

(k1

2j ,k2

2j/2)T , k1, k2 ∈ Z and where Rθ

denotes the rotation matrix with angle θ . This choice lead

952978-1-4799-2572-8/14/$31.00 c©2014 IEEE

to a discrete curvelet system that forms a tight frame, i.e.,every function f ∈ L2(R2) is represented by a curvelt seriesand hence discrete curvelet transform is invertible. The scaledwindows are defined by using the Meyer windows [14] inpolar coordinates of frequency domain as

Uj(r, w) = 2−3j/4W (2−jr)V ( 2.2�j/2�wpi )

= 2−3j/4W (2−jr)V (w

θj,1), j ∈ N (1)

Where W(r) and V(t) are radial and angular windows respec-tively. The basic curvelet is defined by the Fourier transformof Uj(r, w).

φ̂j,0,0(ξ) = Uj(ξ) (2)

and the family of curvelet functions is given by translation androtation of φj,0,0 ∈ L2(R2).

φj,k,l(x) = φj,0,0(Rθj,l(x− bj,lk )) (3)

The family of curvelets is obtained by the Fourier transform of

φj,k,l(x) and it is denoted by φ̂j,k,l(ξ) = e−i〈bj,lk

,ξ〉Uj(Rθj,lξ)

= e−i〈bj,lk

,ξ〉2−3j/4W (2−jr)V (w + θj,lθj,1

) (4)

They are supported inside the polar wedge with radius 2j−1 ≤r ≤ 2j+1 and angle

2−�j/2�π(−1−l)2 ≤ w ≤ 2−�j/2�π(1−l)

2 foreach scale j and orientation l .

Finally some coarser scale curvelet elements for low fre-quencies is given by φ−1,k,0 = φ−1(x − k), k ∈ Z2 where

φ̂−1(ξ) = W0(|ξ|) with W0(r)2 = 1 − ∑

j≥0 W (2−jr)2.

The system of curvelets {φ−1,k,0 : k ∈ Z2} ∪ {φj,k,l : j ∈N, l = 0, 1, ...4.2�j/2� − 1, k ∈ Z2} satisfies a tight frameproperty. Every function f ∈ L2(R2) is represented by acurvelet series and hence discrete curvelet transform is in-vertible f =

∑j,k,l〈f, φj,k,l〉φj,k,l and the Parseval identity∑

j,k,l |〈f, φj,k,l〉|2 = ‖f‖2L2(R2), for all f ∈ L2(R2) holds.The curvelet coefficients are obtained by using the Plancherel’stheorem for j ≥ 0,

cj,k,l = 〈f, φj,k,l〉 =∫R2

f(x)φj,k,l(x)dx =

∫R2

f(ξ)φ̂j,k,l(ξ)dξ

=

∫R2

f(ξ)Uj(Rθj,lξ)ei〈bj,l

k,ξ〉dξ (5)

Where cj,k,l is the curvelet coefficients at scale j and inthe direction l at location k

The low frequency (coarse scale) coefficients are shown atthe center of the display in Fig. 1. The cartesian concentriccorona show the coefficients at different scales. The outercorona corresponds to high frequencies. There are four stripsassociated to each corona in north, south, east and westdirection, these are further subdivided in angular panels. Eachpanel represents coefficients at a specified scale and orientationsuggested by the position of the panel.

Fig. 1. Curvelet coefficents at scale j=0 and at scale j=1, 2. in multipledirections

III. PROPOSED APPROACH

In a directional sub-band, bigger curvelet coefficients ofHR Pan image and LR multispectral image represent sharplocal feature [19]. In this paper, we define a Local MagnitudeRatio (LMR) to inject high frequency details of the local imagefeature into the fused image. LMR is defined as follows.

Let us suppose that cj,l(M) , cj,l(P ) are the sub-bandcurvelet coefficients at scale j in a direction l of the multispec-tral band M and panchromatic image P at higher frequenciesrespectively.

LMRj,l(x, y) =|cj,l(M(x, y))||cj,l(P (x, y))| (6)

Where LMRj,l(x, y) is the sub-band curvelet coefficients atscale j in direction l at location (x,y).

If LMRj,l(x, y) ≤ 1 then cj,l(P (x, y)) represents goodlocal feature. If LMRj,l(x, y) > 1 then cj,l(M(x, y)) rep-resents good local feature. Fusion rule to inject high spatialdetails from HR panchromatic image into LR multispectralimage bands is defined using LMR of curvelet coefficients inthe directional high frequency sub-bands

IV. IMAGE FUSION ALGORITHM USING FDCT

Spatial resolution ratio between HR Pan image and LRmultispectral image is 2. Input images size must be power of 2for coherent multi resolution decomposition in FDCT domain.To obtain HR multispectral image, high frequency details areinjected into each LR multispectral band in FDCT domain. Thefusion rule based on the LMR in FDCT domain is defined asfollows.

1) LR multispectral image is resampled to the scale ofHR Pan image in image coregistration. i.e., both theimages must be at identical geometry and of samesize.

2014 IEEE International Advance Computing Conference (IACC) 953

cj,l(F (x, y)) =

{cj,l(P (x, y)). ∗ LMRj,l(x, y) if LMRj,l(x, y) > 1;

cj,l(P (x, y)) if LMRj,l(x, y) ≤ 1;(7)

Fig. 2. Image fusion scheme for LR multispectral and HR Panchromatic images using fast discrete curvelet transforms

2) The multispectral data in Green, Red and near-infrared bands are extracted band wise.

3) Apply fast discrete curvelet transform (FDCT) tomultispectral band M and Panchromatic image P.The input images are decomposed into four levelsin multiple directions. Number of directions dependson the image size and decomposition levels.

LMS = {c3,l(M), c2,l(M), c1,l(M), a0(M)}HPan = {c3,l(P ), c2,l(P ), c1,l(P ), a0(P )}

where LMS is the set of curvelet coefficients forlow resolution multispectral band, where HPan isthe set of curvelet coefficients for high resolutionpanchromatic image and a0(M) is the coarser scalecoefficents of the multispectral band M, similarlya0(P ) for the panchromatic image P .

4) Fusion rule 1 is defined for the curvelet coefficientsat lower frequencies (coarser scale coefficients).Construct coarser scale coefficients for fusedimage F from LR multispectral band M such thata0(F ) = a0(M)

5) Fusion rule 2 is defined for the curvelet coefficientsat higher frequencies based on high pass modulation.Construct the multidirectional multiresolutioncurvelet coefficients cj,l(F ) by using Equation (7)for fused image.

6) Construct the set of curvelet planes for fused imageas

HFus = {c3,l(F ), c2,l(F ), c1,l(F ), a0(F )}and apply the Inverse Fast Discrete CurveletTransforms (IFDCT).

7) Apply steps (3) to (6) for each multispectral band.

8) Combination of three resultant fused bands providethe HR multispectral fused image

V. EXPERIMENTAL RESULTS AND COMPARISONS

Resourcesat -1 LISS IV image is taken as LR multipectralimage which is ortho rectified and resampled to 5m spatialresolution. Cartosat-1 data taken as an HR Pan image ofresolution 2.5m and these images are in 1:2 scale ratio. HRPan image size is 2048×2048 and LR multispectral image sizeis 1024 × 1024. For clear visualization, subset images of thefused images of different techniques are shown in Fig. 3. Fig.3(a) is the original HR Pan Cartosat-1 image and Fig. 3(b) isthe resampled LR multispectral image. Fig. 3(c) is the HR mul-tispectral image obtained by new fusion rule based on FDCT.Fig. 3(d) - 3(h) are obtained by the wavelet transform, PCA,HPF, Modified IHS and Grams-Schmidth fusion techniquesrespectively implemented in Earth Resources Data AnalysisSystem (ERDAS 2013) satellite image processing software.Quality of the fused images is evaluated with both spatial andspectral quality measures.

A. Spatial Quality Evaluation

Each MS band in a fused image is compared to the HRPan image for spatial quality evaluation.

1) Entropy: Entropy is a measure to directly concludethe performance of image fusion. The Entropy can show theaverage information included in the image and reflect the detailinformation of the fused image. Commonly, the greater theentropy of the fused image is, the more abundant informationincluded in it, and the greater the quality of the fusion.According to the information theory of Shannon, the entropy

954 2014 IEEE International Advance Computing Conference (IACC)

(a) Cartosat-1 (b) LISS-IV

(c) FDCT (d) Wavelet (e) PCA

(f) HPF (g) M.IHS (h) Gramschmidth

Fig. 3. Color compositions of full-scale fusion results for the reported detail (NIR, Red, and Green bands as R, G, and B channels)

of image is defined as

E = −n∑

i=0

pi log2 pi

Where E is the entropy of image and pi is the probability ofi in the image , here pi is the frequency of pixel values from0 to n in the image. We normalized the HR Pan data and LRMS data radiometric resolutions. Entropy values bandwise areshown in Table I.

TABLE I. ENTROPY

Band FDCT Wavelet PCA HPF MIHS GS1 6.22 5.91 5.66 5.89 6.09 5.852 6.78 6.55 6.81 6.68 6.70 6.793 6.29 6.15 6.20 6.12 6.12 6.11

Average 6.43 6.21 6.23 6.23 6.30 6.25

2) Correlation coefficient of high pass filtered images:High frequency details from the Pan image are compared to

the high frequency details from each band of the fused imagesusing a method proposed by Zhou et al.[16]. To extract thehigh frequency data, apply the following convolution kernel tothe images

mask =

[ −1 −1 −1−1 8 −1−1 −1 −1

]

The correlation coefficients between the high-pass filteredfused bands and the high-pass filtered Pan image is used as anindex of the spatial quality [17]. The principle is that the spatialinformation unique in Pan image is mostly concentrated in thehigh frequency component of Pan image has been injected intothe fusion. Correlation coefficient of high pass filtered imagesband-wise are shown in Table II.


TABLE II. CORRELATION COEFFICIENT OF HIGH PASS FILTERED

IMAGES


Average 0.97 0.18 0.77 0.86 0.90 0.64

3) Average gradient: Spatial quality of fused image f byaverage gradient can be calculated by using the equation

ag =1

(M − 1)(N − 1)

M−1∑x=1

N−1∑y=1

√|∂f(x,y)∂x |2 + |∂f(x,y)∂y |2

2

Where f(x, y) is the pixel value of the fused image atposition (x, y). The average gradient reflects the clarity of thefused image. It can be used to measure the spatial resolution ofthe fused image, i.e., a larger average gradient indicates higherthe spatial resolution. Average gradient values band-wise areshown in Table III.

TABLE III. AVERAGE GRADIENT


Average 4.49 3.98 4.40 3.71 4.01 4.13

B. Spectral Quality Evaluation

Resampled multispectral bands of LISS-IV sensor imageand corresponding bands in the fused image are compared forspectral quality evaluation.

1) Spectral Angle Mapper(SAM): Let v and v̂ be twovectors having l components of resampled multispectral LISSIV sensor band and the corresponding band in the fused imagerespectively. Spectral angle mapper (SAM) is the absolutevalue of the angle between the two vectors [5].

SAM(v, v̂) = cos−1

( 〈v, v̂〉‖v‖2.‖v̂‖2

)SAM is measured in either degrees or radians and is usuallyaveraged over the whole image to yield a global measurementof spectral distortion. SAM values equal to zero denotes theabsence of spectral distortion. Table IV shows the SAM valuesfor each fused band.

TABLE IV. SPECTRAL ANGLE MAPPER (IN DEGREES)


Average 0.04 0.07 0.13 0.04 0.08 0.10

2) Universal Image Quality Index (UIQI): Let A and B bethe resampled MS LISS IV sensor band and the correspondingband in the fused image respectively. The Universal ImageQuality Index [18] is defined by Wang and Bovik (2002) as

Q =σAB

σAσB.

2mAmB

(mA)2 + (mB)2.

2σAσB

(σA)2 + (σB)2.

Where σ2A, σ

2B are the variances of images A and B respec-

tively, σAB is the covariance of the images A and B , mA,mB

are mean of the images A and B respectively. Higher value ofUIQI indicates the better fusion method. UIQI values for eachband are shown in Table V. FDCT has better score than othermethods.

TABLE V. UNIVERSAL IMAGE QUALITY INDEX


Average 0.15 0.08 0.08 0.10 0.11 0.09

VI. CONCLUSION

We have described new fusion method based on fastdiscrete curvelet transforms (FDCT). Two fusion rules aredefined, fusion rule 1 is for curvelet coefficients at lowerfrequencies and fusion rule 2 is for the curvelet coefficients athigher frequencies. Fusion rule 1 substitute the coarser scalecoefficients of LR multispectral bands into the coarser scalecoefficients of HR Pan image. Fusion rule 2 is based on thehigh pass modulation using Local Magnitude Ratio (LMR) ofthe curvelet coefficients in each orientation and scale. Biggercurvelet coefficients of HR Pan image and LR multispectralimage represent sharp local feature. LMR directs the injectionof high frequency details of the local image feature in HRPan image into fused image. For experimental study of thismethod Indian Remote Sensing (IRS) Resourcesat-1 LISS IVsatellite sensor image of spatial resolution of 5.8m is usedas low resolution (LR) multispectral image and Cartosat-1Panchromatic (Pan) of spatial resolution 2.5m is used as highresolution (HR) Pan image. This fusion rule generates HRmultispectral image at 2.5m spatial resolution. This methodis quantitatively compared with Wavelet, Principal componentanalysis (PCA), High pass filtering (HPF), Modified Intensity-Hue-Saturation (M.IHS), and Grams-Schmidt fusion methods.Proposed method spatially outperforms the other methods andretains rich multispectral details.

VII. ACKNOWLEDGMENTS

This research was supported under Technology Develop-ment Project (TDP) at National Remote Sensing Centre, ISRO,Department of Space, Government of India through a project,code: 0606313AE901.

REFERENCES

[1] Emmanuel.J. Candes, L. Demanet,D.Donoho and L.Ying.” Fast DiscreteCurvelet Transforms(FDCT)”,Caltech, Pasadena, CA, March 2006.

[2] L. Boubchir, J.M.Fadili,” Multivariate statistical modeling of imagesin wavelet and curvelet domain using the Bessel K form densities”’,Caen,France,2005.

[3] L. Boubchir, R.Boumaza,B. Pumo. Multivariate statistical modeling ofimage with the curvelet transform. InICIP,Cairo,Egypt,November 2009.

[4] J. Luc, Emmanuel.J. Candes, and D. Donoho . Curvelettransform forimage denoising. IEE Tran. on imageprocessing, 11(6):670-684, June2002.

[5] F. Nencini, A. Garzelli,S. Baronti,L. Alparone ,” Remote sensing imagefusion”, Informaion Fusion, 8:143-156, May 2006.

[6] Ying. Li, Xing. Xu,Ben-Du. Bai,Y.N. Zhang, ” Remote sensing imagefusion based on Fast Discrete Curvelet Transform”, In Internationalconference on machine learning and cybernetics, Kumming,China, July2008.

956 2014 IEEE International Advance Computing Conference (IACC)

[7] A. Golibagh, Mehran. Yazdi, ” A novel image fusion method usingcurvelet transform based on linear dependency test”, In Internationalconference on digital image processing, pp:351-354, Bangkok, Thai-land, March 2009.

[8] Shutao. Li, Bin. Yang,” Multi focus image fusion based on wavelet andcurvelet transform”,Pattern recognition letters, 29:1295-1301, February2008.

[9] Cai. Xi, Zhao. Wei,Gao. Fei, ”Image fusion algorithm based on adaptivepulse coupled neural networks in curvelet domain”, In Internationalconference on software processing(ICSP), Paderborn,Germany, July2010.

[10] C.V. Rao, P.S. Reddy, D.S. Jain, K.M.M. Rao,” Quantitative valueaddition analysis of multisensory data fusion” , The ICFAI journal ofearth sciences, 1(1):82-96, March 2007.

[11] M.F. Yakhdani,A. Azizi, ”Quality assessment of image fusion tech-niques for multi sensor high resolution satellite images(cas study: IRSP5and IRSP6 Satellite images)”, In ISPRS TC VII Symposium, pp:205-209,Vienna, Austria,July 2010.

[12] Jun. Li. ”Spatial quality evaluation of fusion of different resolutionimages”, In International archives of photogrammetry and remotesensing, Vol,XXXIII,Part B2, pp:339-346,Amsterdam,2000.

[13] C.V. Rao,K.M.M. Rao P.S. Reddy, G. Pujari, ” A novel method for en-hancement of radiometric resolution using image fusion”, Internationaljournal of applied earth observation and geoinformation, 10:165-174,January 2008.

[14] Jianwei. Ma, Gerlind. Plonka, ”A Review of curvelet and recentapplications”, Caltech, Pasadena, CA, March 2006.

[15] P.P. Gajjar,M.V. Joshi,” New learning base super resolution: use of DWTand IGMRF prior”, IEEE Tran. on image processing, 19(5):1201-1213,May,2010.

[16] J. Zhou,D.L. Civico, J.A. Silander,”A Wavelet Transform method tomrege Landsat TM and SPOT Panchromatic data”, InternationalJournal of Remote Sensing, 19(4):743-757,1998.

[17] G. Hong,Y. Zhang, ”Comparison and improvement of Wavelet-basedimage fusion”, International Journal of Remote Sensing, 29(3):673-691,Dec,2007.

[18] Wang, Bovik, “Universal Image Quality Index.”, IEEE Singal Process-ing Letters, vol.9,pp:81-84,2002.

[19] Chengzhi Deng, Hanqiang Cao, Chao Cao, ShengqianWang,”Multisensor image fusion using fast discrete curvelet transform”,Remote Sensing, GIS Data Processing and Applications, Proc. of SPIEVol. 6790 679004,2007.

[20] C.V. Rao, J. Malleswara Rao, A. Senthil Kumar, and A. S. Manjunath.”Restoration of high frequency details while constructing the highresolution image.” In India Conference (INDICON), 2011 Annual IEEE,pp. 1-5. IEEE, 2011.


4 satellite image fusion using fast discrete

Engineering

Transcript of 4 satellite image fusion using fast discrete