Multiple resolution image restoration

A.W.C. Liew N.F. Law D.T. Nguyen

Indexing terms: Image processing, Wavelet transform

Abstract: A new algorithm for solving the deconvolution problem is proposed. This algorithm uses the wavelet transform to induce a multiresolution approach to deconvolve a blurred signalhmage. The low resolution part of a signal/ image is restored first and then high resolution information is added successively into the estimation process. Two different ways to incorporate the image space positivity constraint, namely loosely and strictly, are discussed. In contrast to most restoration algorithms, the positivity constraint is applied directly in the transformed domain. The performance of the algorithm in the presence of noise is also investigated.

1 Introduction

Deconvolution is a common problem encountered in many different disciplines, such as in astronomical imaging and medical imaging. It refers to the problem of recovering the original object from the recorded image which is distorted by an unwanted blurring func- tion and is often corrupted by noise. In many cases, it is possible to treat the blur as approximately spatially invariant [l]. The whole process can then be regarded as a convolution and the distortion model is described by

g = H f + c (1) where the vectors g , f and c represent the lexicographi- cally ordered noisy blurred recorded signalhmage, the original signalhmage and the additive noise, respec- tively. The matrix H represents the blurring function. Our objective is to recover f given g and H and the knowledge about the noise process.

One popular approach to solving this deconvolution problem is by the regularisation technique. The decon- volution problem is reformulated as minimising an error function defined as

E = 119 - Hf1I2 + Wfl12 (2) 0 IEE, 1997 IEE Proceedings online no. 19971045 Paper first received 14th May and in revised form 7th November 1996 A.W.C. Liew was with the University of Tasmania and is now with the Department of System Engineering and Engineering Management, The Chinese University of Hong Kong, Shatin, NT, Hong Kong N.F. Law was with the University of Tasmania and is now with the Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong D.T. Nguyen is with the Department of Electrical and Electronic Engineering, University of Tasmania, GPO Box 252C, Hobart, Tas 7001, Australia

where //x// denotes the norm of x, a is the regularisation parameter and C represents in general a highpass filter to prevent noise amplification. The minimisation can be done by using the steepest descent algorithm, which is iterative in nature. As pointed out by Wang et al. [2], one difficulty encountered in this minimisation problem is the slow convergence rate. It becomes particularly serious when the size of the problem is large. A way to improve the convergence rate is to employ multigrid processing as proposed by Wang et al. [2].

Wavelet transform based techniques have been applied in many different areas such as image coding and object recognition. One desirable feature of the wavelet transform is that it provides a multiresolution picture of an object and thus enables multigrid process- ing. We propose to apply the wavelet transform to solve the restoration problem.

Most signals in practice have their energy concen- trated at the low frequency region rather than the high frequency region. In contrast, noise usually spreads over the whole frequency range. Hence the signal-to- noise ratio in the low frequency region is higher than that in the high frequency region. As we have more confidence in the low frequency region, we therefore propose a method of restoring the low frequency part of signal first before adding the high frequency infor- mation successively into the estimation. This is in con- trast to the approaches of Wang et al. [2] and Banham et al. [3]. Our method would provide a successively bet- ter approximation to the true signalhmage as we progress from the lower resolution level to the higher resolution level.

2 Multiresolution analysis and orthonormal wavelet transform

The wavelet transform is a mathematical tool that cuts up data into different frequency components and then analyses each component with a resolution matched to its scale. It therefore provides a multiresolution approach to signal analysis and processing. Using the orthonormal wavelet transform, a multiresolution anal- ysis of a signal can be performed [4].

The idea of multiresolution analysis is to write an L2(R) function f(t) as a limit of successive approxima- tions, each of which is a smoothed version offit). Mul- tiresolution analysis decomposes the L2(R) space into a ladder of subspaces V, with the following properties:

(4)

IEE Proc.-Vis. Image Signal Process.. Vol. 144, No. 4, August 1997 199

f ( t ) E vm * f ( 2 " t ) E vo (6)

V k , = vm cE3 wm, VmlWm (7)

63 wm = P ( R ) ( 8 )

Let Wm be the orthogonal bandpass complement of Vm in Vm-, such that

Then it follows that

m E Z The space Vo has associated with it an orthonormal

basis {#(t - I Z ) } , ~ = . The lowpass function $(t) is called the scaling function. By eqn. 6, the set of functions {$m,n(t))m,nEZ, where

& J t ) = 2-"/24(2-"t - n) (9) then forms an orthonormal basis for V,. Similarly, the space Wm has associated with it an orthonormal basis W m , n ( t ) ) m , n E Z , where

$Im,&) = 2-"/2$I(2-"t - n) (10) The bandpass function q(t) is called the wavelet function. The functions $(t) and q(t) extract the low frequency and high frequency information in a signal respectively. The decomposition of f ( t ) E V0 into a lowpass component in Vl and a bandpass component in Wl can therefore be viewed as splitting the frequency spectrum of f ( t ) into a lowpass and a bandpass component. The lowpass component of the frequency spectrum can again be split into a lowpass and a bandpass component, belonging to V2 and W,, respectively, during the next level of decomposition. Pictorially, the situation is as shown in Fig. 1.

Fig. 1 analysi:

frequency Pictorial view of division of frequency spectrum in multiresolution

L+pq-+(g- - -

Fig. 2 F a t orthonormal wavelet transform

Mallat [4] showed that the orthonormal wavelet transform can be implemented in an efficient manner using the pyramid algorithm. Let H and G be the dis- crete lowpass and highpass filters that are used to implement the scaling ftmction and the wavelet, respec- tively. Then the J-level fast orthonormal wavelet trans- form and the J-level fast orthonormal inverse wavelet transform can be implemented by the pyramid scheme shown in Figs. 2 and 3, respectively. The sequence so in Fig. 2 is the discrete input signal at resolution VO. The

200

sequence of detail signals d,, d,, ..., dJ is the orthogonal projection of so onto W,, W,, ..., W,, respectively, and the sequence sJ is the coarsest lowpass component of so at resolution V,.

'J+ Od-- f 2 Fig. 3 Fast orthonormal inverse wavelet trandorm

3 Proposed method

The Mallat's pyramid algorithm [4] is used to imple- ment the orthonormal wavelet transform. The decon- volution problem can be reformulated in the wavelet domain as

wg = W H W - W f + we (11) where W is a wavelet operator. The term WHW-' can be obtained by first applying 1-D transformation to each row of H and then to each column of the result. It can, however, be obtained efficiently in the Fourier domain as it has the semiblock circular structure [3].

To illustrate the method, let us consider one-level decomposition of an N-dimensional signal. After one level decomposition, we have

where the subscripts L and H denote the low frequency and high frequency parts of the wavelet transform, respectively. T I , T2, T3 and T4 are the submatrices of the matrix WHW-l and can be obtained easily in the Fourier domain. The proposed algorithm consists of two steps for a single level decomposition. As the energy of most real world images is concentrated in relatively low frequenciesiresolutions, and the energy of the additive white noise is equally distributed in all frequenciesiresolutions, the signal-to-noise ratio is lower in high frequenciedresolutions, and is higher in low frequencies/resolutions. The problem can then be reformulated as first minimising

with respect to f L . In eqn. 13, a is the regularisation parameter and f H L is the highpass component offL. It should be noted that there are only NI2 unknowns in eqn. 13 which is half the total number of unknowns. Minimising eqn. 13 allows the low frequency part of the signal, f L , to be estimated which provides a low res- olution view of the true signal. Using this low fre- quency part, which is close to the true signal, as an initial estimate off, we then minimise

with respect to f. In this way, the low resolution esti- mate is improved by adding high resolution infonna- tion into the algorithm. This iterative procedure can be easily extended to a multiple resolution restoration.

The choice of the regularisation parameter in eqns. 13 and 14 is an important issue as it controls the tradeoff between the fidelity of the estimation and the smoothness of the solution. In the approach of Wang et al. [2], the regularisation parameter is chosen on a

IEE Proc-Vis. Image Signal Process., Vol. 144, No. 4, August 1997

trial-and-error basis. However, we calculate this parameter iteratively in our approach. For eqn. 13, an appropriate choice of a is [SI

rJ

(15)

where 2 is the noise variance in the subband and 6 is a compensation factor to keep IlfHLII + 6 away from zero. Our multiple resolution restoration method allows the noise variance to be adapted to each subband. For example, if the noise is assumed to be white, then 2 is chosen to be half the total noise variance as eqn. 13 has only half the unknowns. As llfHLII changes from itera- tion to iteration, 6 needs to be changed as well. While choosing an appropriate value for 6 is not easy, we could treat 6 as an unknown and update it iteratively with fL .

The minimisation of eqn. 13 or eqn. 14 is performed using the conjugate gradient routine [6]. The conjugate gradient routine is chosen because of its efficiency in terms of both convergence rate and memory allocation when the number of parameters is of the order of the number of pixels in the image. To use this routine, the analytical expression of the gradient of the error has to be found. For eqn. 13, the gradient of the first part of the error is standard and is repeated as [7]

For the second part, wllfHL112, the gradient is found to be

a(aIlfHL1l2) - 2 ~ ~ 6 0 -

8 f L ( l l f + HLll + 6)3

where

D = W T U (18) and U is the same as the wavelet transform of f L but with allf,, replaced by zero.

4 Loose a priori constraints

The smoothness constraint in eqn. 14 can be regarded as one type of loose a priori constraint applied. As the signal-to-noise ratio is lower in high frequencies/ resolutions, this term helps to prevent extrapolating the values off beyond a certain frequency range at which noise is dominant. This is commonly known as superresolution and should be avoided especially in severe noise conditions [8, 91.

Although it is possible to obtain a solution using the proposed method discussed in Section 3, the solution and the convergence rate can be further improved by imposing extra constraints on the algorithm. Positivity is another a priori constraint that can be used to further improve the estimation. As image intensity is positive, it imposes that each element of the estimated image has to be positive. Following a least-squares approach, we can achieve this by defining an error term as the quadratic sum of pixel intensities at which the constraint is violated, i.e.

wheref, is the ith element off and kV;) is defined as

0 if f z is positive c otherwise K f z ) = {

and c is a positive parameter used to set the weighting of the error term Ef with respect to the other error terms. A large value of c will emphasise the importance of the positivity constraint.

The positivity constraint is an image space constraint and is usually applied in the image space domain [lo]. In our problem, it will be inefficient to transform the low resolution part of the image in the wavelet domain to the image space every time the positivity constraint is imposed. Therefore, we propose to impose the positivity constraint directly in the wavelet domain by the choice of an appropriate scaling function. Specifically, as wavelet transform provides flexibility in choosing the wavelet and the scaling function, by choosing a scaling function that is non-negative at all points, the low resolution part of the image in the wavelet domain must also be non-negative. In this way eqns. 19 and 20 can be applied directly in the wavelet domain.

5 Strict a priori constraints

While the constraints are loosely enforced, some devia- tion from these constraints is tolerated and the result- ant estimate is a compromise solution between different parts of the error function. In some cases, however, the a priori constraint is known is to be exactly verified. The positivity constraint is one such example. To apply the positivity constraint more strictly, one can choose a higher weight for this loose a priori constraint, i.e. choose a bigger value for c. This, however, presents a disadvantage as the final solution would depend on the choice of the weight for the loose constraint, which makes the algorithm nonrobust [l 11. Since image inten- sity is non-negative, the positivity constraint can be applied strictly.

To have a positive f , each element off is written as

f z = (uJ2 (21) Then the gradient of the error function E with respect to the new set of parameters can be written as

dE - dE d f z du, a f t duz

~ ___ -

dE = 2uz-

a f i

It should be noted that this formulation can also be applied directly in the wavelet domain by choosing a non-negative scaling function in the wavelet decompo- sition as discussed in Section 4.

6 Performance measures

In Peng and Stark’s paper [12], they suggest that a pos- sible way to quantify the performance of the algorithm is to use the similarity metric. It is defined as the corre- lation between the original object and the reconstructed object normalised by their respective norms, i.e.

They claim that the similarity metric S is a measure of the similarity of the shapes of fand f . Therefore, it is

20 1

2

IEE Proc.-Vis. Image Signal Process., Vol. 144, No. 4, August 1997

more closely related to the visual quality of the reconstruction than the mean square distance. Another possible metric used is the normalised mean square error suggested by Miura et al. [13]. It is defined as

The normalised mean square error can be used to pro- vide an average error for the estimation. In fact, it is easy to show that S and MSE are related by

M S E = 1 - S2 (25) A plot of S against MSE is shown in Fig. 4. If the estimation is of good quality, then MSE provides more discriminating power than S. Conversely, S should be used when the quality of the reconstructed image is poor. Both measures are used in our numerical experiments.

0' 0 0.2 0 . 1 0.6 0.8

normalised mean sauare e r r o r Fig.4 ..

Plot of similarity metric against normalised mean square error ... 45 deg. straight line

Fig.5

202

Original object used in simulation

7 Effect of superresolution

In practical implementation, we should watch out for the superresolution effect [9] for each minimisation step as represented by eqns. 13 and 14. Superresolution means extrapolating the values o f f in the frequency domain beyond a particular frequency limit determined by noise. As noted by Sementilli et al. [14], superresolu- tion, even when the point spread function is known, is prone to a null image artifact if iterations are carried too far. According to Lannes et al. [8], only a smooth version of the original object can be recovered in the severe noise contamination situation as noise and the object of interest in the high frequency region are indis- tinguishable. Minimisation over a point limited by noise would make the recovered object noisy due to incorrect estimation of the high frequency components.

~

Fig. 6 Circular point spread function with seven pixels in diameter

Fig. 7 Resultant blurred image

To illustrate this point, we consider an example shown in Figs. 5-7. Fig. 5 shows a 64 x 64 pixel image. It is blurred by a circular point spread function of seven pixels in diameter shown in Fig. 6. The resultant blurred image is shown in Fig. 7. Additive white noise is added to the blurred image to give a 20dB SNR image. Here, the signal-to-noise ratio is defined as lOlog(o:/o>), where 02 and 02 are the variances of the blurred image (HA and the noise, respectively.

IEE Proc -Vis Image Signal Process, Vol 144, No 4, August 1997

Fig. 8 shows a plot of the convolution error, IJg - H’I2, and the high frequency error, IK1l2, against the number of iterations. As the number of iterations increases, the convolution error decreases but the high frequency error increases. Plots of similarity and MSE against the number of iterations are shown in Figs. 9 and 10, respectively. The curve of similarity achieves a peak value at the iteration when the convolution error is approximately equal to the noise variance and then decreases monotonically for the rest of the number of iterations. A similar performance is also shown in the curve of MSE.

5

6

5

Y! P ISI

1

1

\

10 20 30 number of iterations

Fig.8 of iterations ~ convolution error

Plot of convolution error and high frequency error against number

. . . . . . . . . . . high frequency errnr

0

0

A 4-

L 0 ._

.- E ._ VI

0

0

I I

I

, 10 20 30 LO number of iterations

Fig.9 Plot of similarity against number of iterations

IEE Proc.-Vis. Image Signal Process., Vol. 144, No. 4, August 1997

Hence, in implementing the minimisation using our proposed method, we limit each minimisation step to a particular value determined by the noise variance to prevent the effect of superresolution. If the error passes through this particular point, the minimisation process is stopped and the algorithm proceeds to the next reso- lution level.

1

0

VI

2 a,

$ 0 U VI C ar E

0

0

\ \

10 20 30 LO number of iterations

Fig. 10 tions

Plot of normalised mean square error against number of itera-

8 Numerical examples

The proposed algorithm can simply be applied to deb- lur Fig. 7 without incorporating further a priori con- straint. It is denoted as algorithm 1. On the other hand, some a priori constraints can be applied to constrain the estimation to have a particular form. As discussed in Sections 4 and 5, the a priori knowledge can be applied either loosely or strictly. Algorithms 2 and 3 refer to the loose and strict application of the positivity constraints, respectively.

To apply the positivity constraint directly in the wavelet domain, a non-negative scaling function is cho- sen. We choose the Haar wavelet and use Mallat’s pyr- amid algorithm to implement the orthonormal wavelet transformation. Thus, in this case, W is defined as

W =

1 - __ ?

.. . . . . Jz

. . . . . .

We set the level of decomposition to three, which implies that there are four steps in the restoration algorithm. The weighting constant c in eqn. 20 for algorithm 2 is chosen to be 1.

203

of noise contamination in this example, algorithm 3 performs the best. This is also true for all other images we have tested. We also observed that, in all our exper- iments, algorithm 2 performed better than algorithm 1 in high SNR and they are comparable in low SNR. The reconstructed images for algorithms 1, 2 and 3 are shown in Figs. 11 and 12, 13 and 14. and 15 and 16, respectively.

Fig. 11 Reconstructed imuge for algorithm I for S N R of 20 dB

Fig. 13 Reconstructed image for ulgorithni 2 for S N R of 20 dB

Fig.12 Reconstructed iinugefor ulgorithm 1 for S N R of 40 dB

Table 1: Mean square error for different algorithn presence of different noise levels

Algorithm 1 Algorithm 2 Algorithm MSE MSE MSE Noise level

M 0.0374 0.054 0.0004

40 0.0659 0.0254 0.0012

30 0.1853 0.1606 0.0094

20 0.4001 0.401 1 0.3181

10 0.4903 0.5049 0.4325

i s in

- 3

Fig. 14 Reconstructed irnuge for ulgorithm 2 for S N R of 40 dB

Table 2: Value of similarity for different algorithms in presence of different noise levels

Algorithm 1 Algorithm 2 Algorithm 3 S S S Noise level

M 0.981 1 0.9973 0.9998

40 0.9665 0.9872 0.9994

30 0.9026 0.9162 0.9953

20 0.7745 0.7739 0.8258

10 0.7139 0.7036 0.7533

Tables 1 and 2 summarise the results for algorithms 1, 2 and 3 in the presence of different levels of noise. Generally, S decreases and MSE increases when the signal-to-noise ratio decreases. For all different levels Fig. 15 Recoiistritcfed imuge for ulgorithnl 3 for S N R of 30 dB

204 IEE Proc.-Vis. Imuge Signal Process., Vol. 144. No. 4, August 1997

It can be seen from Figs. 11-16 that the images reconstructed by algorithm 3 are the cleanest and have the sharpest contrast amongst the images reconstructed by the three algorithms. This is due to the strict appli- cation of the positivity constraint into the restoration algorithm.

We apply algorithm 3 to the ‘Truck’ image shown in Fig. 17. The blurred image shown in Fig. 18 is blurred by a circular point spread function of Fig. 6. The deconvolved image is shown in Fig. 19. It can be seen that the reconstructed image is of fairly good quality, although artifacts can be seen. It may be due to the fact that the Haar wavelet does not match well to the original ‘Truck’ image. Further improvement may be possible by using a smoother wavelet.

Fig. 16 Reconstnrcted image for algorithm 3 jor S N R of40 dB I,

#m A M ’ “ , . “ . , r , ,nw (hn,

Fig. 19 Deconrolved ’Tnrck‘ imuge

9 Conclusions

Fig. 17 ’Truck’ image

Fig. 18 spread function

IEE Proc-Vis. hnuge Signal Proce.w. Vol. 144, No. I, August I997

Blurred image obtained from convolving with a circirlar point

We have proposed a multiresolution approach to solve the deconvolution problem. The ability of the wavelet transform to provide a multiresolution view of an object and the flexibility in choosing a wavelet function to match a particular application have made it useful for solving the multiresolution deconvolution problem.

Based on the facts that most real world signals/ images have most of their energy concentrated in the low frequency region and the signal-to-noise ratio of the signaldimages is higher at low frequency than at high frequency, we have proposed to restore the low frequencyhesolution part of a signalhmage first and then successively refine the estimation by adding high frequency information into the estimation process.

In practice, we usually have some knowledge about the signalhmage we are trying to recover. One such piece of n priori information is the positivity constraint. We have proposed to incorporate this n priori informa- tion either loosely or strictly into the estimation proc- ess. Numerical results show that the incorporation of some a priori information can significantly improve the estimation.

10 References

BATES, R.H.T., and McDONNEL, M.J.: ‘Image restoration and reconstruction’ (Clarendon Press, 1986) WANG. G., ZHANG. J., and PAN, G.W.: ‘Solution of inverse problems in image processing by wavelet expansion’, ZEEE Truzs. Image Process.. 1995, 4, pp. 519-593 BANHAM. M., GALATSANOS, N., GONZALEZ. H., and KATSAGGELOS, A.: ‘Multichannel restoration of single chan- nel images using a wavelet-based subband decomposition’, ZEEE Trans. Imuge Process., 1994, 3, pp. 821-833 MALLAT, S.: ‘A theory for multiresolution signal decomposi- tion: the wavelet representation’. ZEEE Trans. Pattern A n d Much. Zntell., 1989. 11, pp. 674-693

205

5 KANG, M., and KATSAGGELOS, A.: ‘Simultaneous iterative image restoration and evaluation of the regularisation parameter’, IEEE Trans. Signal Process., 1992, 40, pp. 2329-2334

6 PRESS, W., TEUKOLSKY, W., VATTERLINE, W., and FLANNERY, B.: ‘Numeral recipes in C: the art of scientific computing’ (Cambridge University Press, 1992)

7 YANG, Y., GALATSANOS, N.P., and STARK, H.: ‘Projection- based blind deconvolution’, J. Opt. Soc. Am. A, 1994, 11, (9), pp. 2401-2409 LANNES, A., ROQUES, S., and CASANOVE, M.J.: ‘Resolu- tion and robustness in image processing: a new regularization principle’, J. Opt. Soc. Am., 1987, 4, pp. 189-199 LAW, N.F., and LANE, R.G.: ‘Blind deconvolution using least squares minimisation’, Opt. Commun., 1996, 124, pp. 341-352

8

9

10 LAW, N.F., and NGUYEN, D.T.: ‘Multiple frame projection based blind deconvolution’, Electvon. Lett., 1995, 31, (20), pp. 1734-1735

11 THIEBAUT, E., and CONAN, J.M.: ‘Strict a priori constraints for maximum likelihood blind deconvolution’, J. Opt. Soc. Am. A, 1995, 12, (3), pp. 485492

12 PENG, H., and STARK, H.: ‘Signal recovery with similarity con- straints’, J. Opt. Soc. Am. A, 1989, 6 , (6) , pp. 844-851

13 MIURA, N., KUWAMURA, S., BABA, N., ISOBE, S., and NOGUCHI, M.: ‘Parallel scheme of the iterative blind deconvo- lution method for stellar object reconstruction’, Appl. Opt., 1993, 32, (32), pp. 6514-6520

14 SEMENTILLI, P.J., HUNT, B.R., and NADAR, M.S.: ‘Analy- sis of the limit to superresolution in incoherent imaging’, J. Opt. Soc. Am. A, 1993, 10, pp. 2265-2216

206 IEE Proc.-Vis. Image Signal Process., Vol. 144, No. 4, August 1997