ImageDeblurringbyFine-Granularityandspatially ... · ImageDeblurringbyFine-Granularityandspatially...

INTERNATIONAL JOURNAL OF EMERGING TRENDS IN TECHNOLOGY AND SCIENCES(ISSN: 2348-0246(online)), VOLUME-1, ISSUE-1, JANUARY-2014

106www.ijetts.org

Image Deblurring by Fine-Granularity and spatiallyadaptive regularization.Nallamothu Pranoy Roy1, P.Muniguravaiah2

1M.Tech ,Bharat Institute of Engineering & Technology2Assoc.Prof.,Bharat Institute of Engineering & Technology

Abstract—This paper studies two classes of regularizationstrate-gies to achieve an improved tradeoff between imagerecovery and noise suppression in projection-based imagedeblurring. The first is based on a simple fact that -timesLandweber iteration leads to a fixed level of regularization,which allows us to achieve fine-gran-ularity control ofprojection-based iterative deblurring by varying the value. The regularization behavior is explained by using the

theory of Lagrangian multiplier for variational schemes.The second class of regularization strategy is based on theobservation that various regularized filters can be viewed asnonexpansive mappings in the metric space. A deeperunderstanding about different regularization filters can begained by probing into their asymptotic behavior—the fixedpoint of nonexpansive mappings. By making an analogy tothe states of matter in statistical physics, we can observethat different image structures (smooth regions, regularedges and textures) correspond to different fixed points ofnonexpansive mappings when thetemperature(regularization) parameter varies. Such ananalogy motivates us to propose a de-terministic annealingbased approach toward spatial adaptation inprojection-based image deblurring. Significantperformance improvements over the currentstate-of-the-art schemes have been observed in ourexperiments, which substantiates the effectiveness of theproposed regularization strategies.

Index Terms—Deterministic annealing,fine-granularity regular-ization, projection-baseddeblurring, nonexpansive mapping, spa-tial adaptation.

INTRODUCTION

The problem of image deblurring, namely therecovery of an image from its noisy and blurredversion, has been extensively studied in the pastdecades. Several classical itera-tive solutionalgorithms exist in the literature—e.g., Kaczmarz’smethod [1], Cimmino’s method [2] and Van Cittert’smethod [3] (also called Landweber method [4]),maximum-entropy method [5] and Lucy-Richardsonmethod [6]. Due to the ill-posed nature of deblurringproblem, regularization techniques, which attempt toincorporate the a priori knowledge “about either thetrue so-lution or the noise into the solutionalgorithm” [7], have shown to be highly effective.Under the context of image deblurring,there are two popular frameworks for implementing

regulariza-tion: projection-based and variational. Inprojection-based iter-ative deblurring, the knowledgeabout true solution can be in-corporated into the

so-called prior constraint set; noisy blurred image alongwith the degradation model specify the other set calledobservation constraint set. In variational iterativedeblur-ring, there is a so-called regularization functional(also called stabilizing functional in [7]) conveying theprior information.Historically, both projection-based and variational

ap-proaches have been extensively studied in1970s–1980s. Youla’s pioneering work in 1978 [8] hasled to a flurry of works on projection-based imagedeblurring [9]–[13]. Classical regu-larization techniquesfor variational image restoration include constrainedLeast-Square [3], Tikhonov-Miller method [14]. Sincethe invention of total-variation (TV)-basedregularization [15] in early 1990s, variational imagedeblurring [16] and its extensions (e.g., half-quadraticapproximation [17], majoriza-tion-minimization [18],variational Bayesian [19], nonlocal TV [20]) havereceived much more attention. The other class ofcompeting schemes in the past decade weresparsity-based regularization (e.g., wavelet-based[21]–[24] and DCT-based [25]). The relationshipbetween TV and wavelet-based regular-ization hasalready been established in the literature [26], [27]. Forexample, the thresholding parameter in waveletshrinkage plays the dual role of Lagrangian multiplier invariational schemes.The tantalizing question is: what happened to the

class of pro-jection-based approaches? They might havelost the momentum in the race; but we argue thatprojection-based framework is still powerful in that itallows us to unify various regularization tech-niques. Infact, it is known that both TV diffusion and waveletshrinkage can be interpreted as the projection onto someconvex set [28]. Moreover, it is possible to comparedifferent linear or nonlinear projection operators withoutan analytic form of the underlying variational functionalby borrowing tools from the fixed-point theory [29].What has been limiting the power of projection-basedapproaches is a lack of flexible relaxation control (a.k.a.regularization) strategies, whose importance is clearlyappreciated in early works such as [30] but has not beenseriously addressed by the image processingcommunity. There-fore, it is the objective of this workto fill in this gap and hope-fully revive the interest in


107www.ijetts.org

studying projections for image de-blurring. Under theframework of projection-based image de-blurring, wemake the following new contributions in this work:• Fine-granularity regularization. Inspired by theequiva-lence between Landweber iteration andprojection onto

convex set [13], we propose to concatenate -timesLandweber iterations to achieve fine-granularityregular-ization in projection-based deblurring. Theparameter plays the role of relaxation control initerative deblurring1. Under a variational framework,we show that the regular-ization parameter isequivalent to a weighted deblurring functional.Eigenvalue analysis is used to illustrate how suchweighting strategy on deblurring functional canachieve a better tradeoff between image recovery andnoise suppression.• Spatially adaptive regularization. Inspired by

recent works on deterministic annealing basedsparsity optimization [31], [32], we propose togradually decrease the regular-ization parameter toachieve spatial adaptation. Interesting analogybetween statistical physics and image processing isused to illustrate the asymptotic behavior of bothlocal and nonlocal regularized filters. It is shownthat regu-larization parameter in image processingplays a similar role of temperature parameter instatistical physics; and different structures (e.g.,smooth areas, regular edges and textures) in imageprocessing correspond to the states of matter instatistical physics. The technique of deter-ministicannealing—popular for nonconvex optimizationproblems—is shown effective on achieving spatialadap-tation.

To give a quick view of the performance achieved byour new regularization strategies, we have includedthe comparison of improved SNR (ISNR) resultsamong seven competing methods in Table I. It is easyto observe that BM3D-based nonlocal deblurring,powered by the fine-granularity and spa-tial adaptiveregularization strategies proposed in this work, hasachieved significant improvement (around 1.6 dB)

over current state-of-the-art techniques in thisspecific experiment (cameraman image, 9 9uniform blur and dB). Such promisingexperimental findings support an idea advo-cated byEkeland over 30 years ago—“it will be foundrelevant to the study of more general variationalproblems, where one does not seek a minimum, butsome kind of saddle-point” [29]. We expect theinterplay of image processing and nonconvexminimization is going to be more fruitful in thefuture. In summary, projection-based imagedeblurring with fine-granu-larity and spatiallyadaptive regularization is an old folk song (finding acommon point of constraint sets) sung to a new tune(saddle-point seeking).

1In variational deblurring, it is known that -times Landweberiteration is equivalent to the scaling of Lagrangian multiplier by afactor of . The rest of the paper is structured as follows. InSection II, we briefly review variational imagedeblurring to introduce nec-essary notations and variousregularization techniques from a projection perspective.Section III introduces a new weighted deblurringfunctional, which allows us to control the degree ofregularization to fine granularity. Using variationalcalculus, we show how it generalizes the existingLandweber iteration. Section IV.A covers the study ofasymptotic behavior of regu-larized filtering and focuseson the spatial adaptive capability of nonlocal filters. InSection IV.B, we present a complete iterative imagedeblurring algorithm with fine-granularity and spatiallyadaptive regularization. Implementation details andextensive experimental results are reported in Section Vto demonstrate the effectiveness of the proposedalgorithm. Some open issues related to annealingschedule and perspectives toward the future of imagedeblurring are included in Section VI.

ITERATIVE REGULARIZED IMAGE DEBLURRING: APROJECTION-BASED PERSPECTIVE

The degradation model in image deblurringproblem can be written as [7]

where are lexicographically stacked representationof un-blurred/blurred(observed) images, is theblurring operator and denotes the additive whiteGaussian noise with zero mean and variance .Temporally ignoring the interference from ad-ditivenoise, we note that each row of the linear blurringsystem defines a hyperplane

where is the inner product and


108www.ijetts.org

is the spatial coordinate. Such hyperplane-basedgeometric per-spective has motivated a collection ofprojection-based methods to solve the above linearsystem. For example, one of the ear-liest iterativealgorithms proposed by Kaczmarz in 1937 [1] opts tosequentially find the orthogonal projection onto those

hyperplanes; alternatively if we do a reflectiveprojection onto those hyperplanes in parallel andtake their average, we ob-tain so-called Cimmino’smethod [2]. Various iterative methods developed latercan be viewed as the extensions of these two basicalgorithms. Among them, it has been shown in [13],[34] that Landweber iteration [4] (a.k.a. Van Cittert’smethod) admits a projection-based interpretation (pleaserefer to Appendix A for more details).

PROJECTION-BASED IMAGE DEBLURRING

One common shortcoming of these conventionalprojection methods is their sensitivity to noiseinterference. Image deblur-ring belongs to the class ofinverse problems in which small er-rors in theobservation data could result an arbitrarily largeamount od derivation of from the true solution [7].Many in-verse problems of such flavor including imagedeblurring, com-puter tomography and inversescattering are more widely known as ill-posed problems[14]. Regularization techniques represent a class ofmethods which facilitates the finding of physicallymeaningful answer to an ill-posed problem by pursuinga related well-posed problem. Identifying the constraintset of the true solution is a popular approach towardsregularization. Histori-cally, early attempts on imageregularization are mostly based on heuristicobservations about image signals (e.g., non-nega-tiveand power spectral bound sets [12]). They have showntoo weak to characterize the images of our interest.More successful development of image regularization

tech-niques has been largely influenced by the so-calledvariational principle—i.e., one comes up with someregularization func-tional first (it reflects our a priorknowledge about ); then invokes variational calculus toobtain projection operator onto the prior constraint setby setting (e.g., the projection algorithmfor TV minimization [35]). Under the con-text of imagedeblurring, one can seek to minimize the deblur-ringfunctional subject to some constraint on thereg-ularization functional. Using Lagrangian multiplier,the above constrained optimization can be transformedinto the following unconstrained optimization problem(the standard variational formulation)(3)

where Lagrangian multiplier is often calledregularization pa-rameter because it controls thetradeoff between image recovery and noisesuppression. The major difference between variousvariational deblurring techniques lies in the selectionof regu-larization functional (e.g., TV-based [15]versus wavelet-based [22]). We note that both TVdiffusion and wavelet shrinkage can be interpreted asthe projection onto some convex set as shown in [28].In our recent work [36], we have shown how to

understand and compare different regularized filters byviewing them as nonlinear projection operators andinspecting their asymptotic behavior. The basic idea isto switch from an energy-first ap-proach (i.e., defining

first) to a projection-first one (i.e., designingprojection operator first). Such transition is based on theobservation that unlike physical sciences, it is ofteneasier to come up with a filtering algorithm based onheuristics in-stead of principles in image processing. Infact, it is often more challenging to analytically obtain

from the corresponding projection operator than theother way around However, what really matters in thepractice of image processing is how close is theunknown image of our interest to the minimum of .Therefore, we proposed to approach the minimum of

by successively applying projection operators basedon the connec-tion between Caristi’s fixed pointtheorem [37] and Ekeland’s variational principle [29](please refer to Appendix B for some backgroundinformation).

To summarize this review section, we have seen bothde-blurring and regularization can be interpreted under aprojec-tion-based framework. The value of a unifiedprojection-based framework is that it allows us tounderstand various image de-blurring algorithms in aprincipled manner. From this perspec-tive, manydeblurring algorithms based on different heuristics arein fact connected and can be viewed as a combination ofse-quential/parallel projection strategies. The completeprocedure including the order and choice of projectionoperators under-lying any iterative image deblurringalgorithm has been called relaxation control [30], whichis conceptually equivalent to reg-ularization in our ownopinion. The objective of this paper is to study twoclasses of regularization strategies, namelyfine-granularity and spatial adaptation under theframework of pro-jection-based iterative imagedeblurring algorithms and experi-mentally demonstratetheir effectiveness.

III. FINE-GRANULARITY REGULARIZATIONWITHWEIGHTEDDEBLURRING FUNCTIONAL


109www.ijetts.org

In variational image deblurring, the degree ofregularization is controlled by the regularizationparameter [38]. Such flexi-bility is not inherited byprojection-based image deblurring be-cause it is notalways feasible to analytically derive from aprojection operator (the inverse of variationalformulation). Therefore, it is desirable to find someother way of control-ling the degree of regularization tofine-granularity for projec-tion-based schemes. It turnsout that one possible solution is based on a scalingargument about regularization parameter [39]— -times Landweber-iteration is equivalent to scaling theregularization parameter by a factor of . Suchobservation suggests a convenient way of controllingthe degree of regular-ization in projection-baseddeblurring through varying instead of . Morerigorously, the above idea can be implemented bymodifying the deblurring functional in (3). We proposeto min-imize the following weighted deblurringfunctional.

.

Sample fixed points of different projection operators used inleading image deblurring algorithms: (a) SA-DCT [25]; (b) TV[17]; (c) BM3D [33]; (d) NLTV [41]. always feasibleespecially in the presence of degraded obser-vationdata. A more promising approach is to target at jointconvergence—i.e., with a sequence of decreasingrameter in thresholding-based projection) or (paramin minimization-based projection) [30], we often observethat both and would converge as.

The comparison of asymptotic behavior amongdifferent regularized filters is relevant to imagedeblurring because it allows us to inspect the solutionspecified by the prior constraint set only (i.e., withoutany interference from the observation constraint setdefined by and degradation model). Fig. 1compares the fixed-point of various local andnonlocal projection operators which have been usedin several leading deblurring algorithms:shape-adaptive DCT (SA-DCT [25], total-variation(TV) [17]), block-matching 3-D (BM3D) [33] and

nonlocal TV [41]. The schedule ofrelaxation control is forSADCT/BM3D, for TV and

for NLTV. The startingpoint of each experiment is random noise and we cansee that different starting points lead to varyingfix-points of the same class. It can also be observedthat nonlocal projection operators andare capable to producing a richer collec-tion of

mixed image structures than local ones. Suchcapability of “synthesizing” spatially varyingstructures is an important advantage of nonlocaloperators for the class of photographic images. B.Spatial Adaptation via Deterministic Annealing

One issue that has not been fully addressed in nonlocalde-blurring schemes is the choice of regularizationparameter in NLTV [41] or threshold parameter inBM3D [33]. Such issue is related to the convexity ofnonlocal regularization functional or prior constraint set(note the duality between convex sets and

Fig. 2. Sample fixed points of BM3D-based (top) and NLTV-based(bottom) projection operator with different temperature values ( for

and for ).

convex functions [45]). It is easy to verify thenon-convexity of prior constraint set because thecollection of image patches of the same size inphotographic images form a low dimen-sional manifold[46]. One way of conceptually visualizing such imagemanifold is that it is decomposed of severalconstella-tions: smooth regions, regular edges andtextures. Different con-stellations on image manifold areconceptually similar to dif-ferent states of matter (solid,liquid and gas) in natural phys-ical systems. It isenlightening to empirically observe how theasymptotic behavior of nonlocal projection operatorssuch as NLTV and BM3D evolve from one phase toanother as the reg-ularization parameter varies (referto Fig. 2). We note that such phenomenon of phasetransition has not been observed for con-ventionallocal image models including Markov-Random-Field[47].


110www.ijetts.org

The duality between regularization parameter inimage deblurring and temperature parameter instatistical physics suggests a deterministic annealingbased approach toward the finding of common point atthe intersection of several constraint sets, which mightbe non-convex. The basic idea of annealing has knownfor long—e.g., simulated annealing for optimization[48], deterministic annealing for clustering andcompression [49], iterative regularization for inverseproblems [50]. Here we propose to gradually decreasethe regularization parameter in nonlocal TV orthresholding parameter in BM3D and provide aninterpretation from the perspective of spatial adaptation:• Starting with a large parameter (hightemperature), it is rel-atively easier to find areliable local optimum because the constellationof smooth regions is approximately convex(note that convexity is a sufficient condition forall local minima to be global minima [51]);

• As the temperature parameter gradually decreases,the cost functional gets more rugged and lessconvex. After some critical temperature, thesystem starts to enter a new phase (theconstellation of edges) and its associatedregulariza-tion constraint set starts to click with thedata constraint;

• As the temperature keeps dropping and reachesthe new phase (the constellation of textures),fine-scale details sup-pressed at highertemperatures become admissible (please refer toFig. 2).

Fig. 3. Impact of regularization parameter ( value) on ISNRprofiles of Alg. 1 at different noise levels (cameraman, , 9 2 9uniform blur).

Fig. 4. ISNR profiles of Alg. 1 with same parameter setting (1 dB) but different projection operators for cameraman (solid) and

(dashed) images: —‘ ’; —‘ ’; —‘ ’; —x’.seen that: 1) for cameraman image, both local andnonlocal regularization work effectively and thedifference is mainly on the convergence speed (note that

has not reached its con-vergence yet due to earlytermination); 2) for barb256 image, nonlocalregularization noticeably outperforms local one—this isnot surprising because texture patterns can not beeffectively handled by models describing localvariations. The gap between and ispartially due to the choice of regulariza-tion parameter

which favors over (can achieve ISNR of over 5 dB with a more

appropriate.

Third, we use experimental results to substantiate thebenefits of annealing-based spatial adaptation. Fig. 5includes the com-parison of ISNR profiles of Alg. 1with and without annealing at the BSNR of 40 dB. Thefollowing parameter setting is used in this experiment:

and (w/o annealing) versus(with annealing). We can observe

Fig. 5. ISNR profiles of Alg. 1 ( dB) on cameraman image with(‘ ’) and without (‘ ’) deterministic annealing for (solid) and

(dashed).

that for both and , Alg. 1 withdeterministic an-nealing eventually achieves higherISNR. However, it is impor-tant to note thatdeterministic annealing could enter a negative loopwhen is below so-called noise floor [54] (as aconse-quence of non-stable clustering). That is why wehave adopted an updating energy-based stoppingcriterion in Alg. 1. As soon as the term starts to


111www.ijetts.org

fluctuate (the larger value is set, the more fluctuationis allowed), the algorithm is terminated.

B. Comparison With Other Competing MethodsThree recently published works on image deblurring,

namely TV based on majorization-minimization(TVMM) [18], shape-adaptive DCT [25] and iterativeshrinkage/thresholding (IST) [24], have been chosen asthe benchmark schemes in our study. Their source codesor executables are released by the original authorsfollowing the principle of reproducible research [55].Default parameter settings provided by their demoprogram are used and could vary from experiment toexperiment (e.g., SA-DCT used different sets ofregularization parameters for different blurring kernels).For the reason of fairness, we have kept the sameparameter setting of Alg. 1 in all experiments (nooptimality is claimed):

.

Fig. 6. Comparison of original cameraman image, noisy blurredimage (

dB, 9 2 9 uniform blur) and deblurred images by differentalgorithms):

(a) original;(b) noisy blurred; (c) TVMM (dB); (d) SADCT (dB); (e)IST (dB); (f) this work (dB).

schemes in all six cases. It should be noted that thediscrepancy between the ISNR values reported in TablesI and II for cam-eraman image is due to different stepsizein the annealing sched-ules. The small difference (9.96dB versus 10.10 dB) caused by different also showsthat Alg. 1 is reasonably robust to the an-nealing strategy(note that the ISNR results of deblurring experi-mentwith the same parameter setting would not be constantdue to the randomness of additive noise—the range offluctuation is about dB according to our experience).

Figs. 6 and 7 compare the original image, the noisyblurred image and deblurred images by four competingalgorithms. For the cameraman image, the subjectivequality of deblurred image by Alg. 1 is apparentlysuperior to that of other com-peting schemes—e.g., oursdoes not contain any artifact around the tripod regionlike others do. The recovery of objects in thebackground at the distance is also noticeably better. Forthe barb256 image, Alg. 1 and SADCT noticeablyoutperform the other two. Between Alg. 1 and SADCT,the 1 dB ISNR difference mainly comes from the morefaithful reconstruction of some low-contrast textureareas such as the scarf areas with shading and in theshadow. Both Alg. 1 and SADCT still suffer fromaliasing-related artifacts around the bottom-middleregion of the image, which deserve further investigation.Next, we report the deblurring results for a

nonuniform blur-ring kernel:and two different test images (house and lena256).

PROJECTION-BASED IMAGE DEBLURRING

Fig. 7. Comparison of original image, noisy blurred image ( dB, 9 2

9 uniform blur) and deblurred images by different algorithms): (a)original; (b) noisy blurred; (c) TVMM( dB); (d) SADCT ( dB); (e)IST ( dB); (f) this work ( dB).

VI. CONCLUDING REMARKS

In this paper, we have revisited the problem ofiterative image deblurring under a projection-basedframework. Our theoretic contributions are two-fold. Onone hand, we generalize existing variational formulationby showing it is possible to define pro-jection operator


112www.ijetts.org

first instead of defining the energy. The lesson we havelearned is that images with complex structures do notnecessarily correspond to the global minimum butsaddle point of some energy functional. Suchsaddle-point perspective fa-vors projection-based imagedeblurring because it admits the solution to anonconvex minimization problem. On the other hand,we propose to understand nonlocal regularizedfiltering by probing into their asymptotic behaviorbased on fixed-point theory. Phase transition ofBM3D and NLTV filtering motivates a deterministicannealing approach toward spatial adaptation, whoseeffectiveness has been experimentally verified bysim-ulation results. Significant improvements overcurrent state-of-the-art reported in the open literaturehave been observed in our experiments with twoblurring kernels, three noise levels and four testimages.There are several open issues that remain to be

addressed in the future. Among them, the choice ofrelaxation parameter has been shown critical to theperformance but only the rule of thumb has beenobtained through experiments: lighter noisecor-responds to larger values. It will be desirable toderive some analytical result between andsimilar to what is known in variational deblurring.The annealing schedule has sec-ondary impacton the deblurring performance and its automaticselection calls for further technical investigation.There are also several possible extensions along thisline of research—e.g., in this work we haveexclusively considered non-blind scenarios; itspotential implications into blind image deconvolution[56] remains unexplored. Among popularapplications of blind de-convolution, motiondeblurring has attracted increasingly more attentionin recent works (e.g., [57]). Whether the proposednon-convex optimization framework can be appliedto better solve motion deblurring problem remainsunknown.

REFERENCES

[1] E. Cands, “Compressive sampling,” in Proc. Int.Congr. Mathematics,Madrid, Spain, 2010, pp.1433–1452.WU et al.: LOW BIT-RATE IMAGECOMPRESSION VIA ADAPTIVEDOWN-SAMPLING 561

[2] X.Wu,K.U. Barthel, and W. Zhang, “Piecewise2-D autoregression forpredictive image coding,” in

Proc. IEEE Int. Conf. Image Processing,Chicago, IL,Oct. 1998, vol. 3, pp. 901–904.

[3] X. Li and M. T. Orchard, “Edge-diretedprediction for lossless compressionof naturalimages,” IEEE Trans. Image Process., vol. 10, no.6,pp. 813–817, Jun. 2005.

[4] D. Santa-Cruz, R. Grosbois, and T. Ebrahimi,“Jpeg 2000 performanceevaluation and assessment,”Signal Process.: Image Commun., vol. [6] Y. Tsaig,M. Elad, and P. Milanfar, “Variable projection fornear-optimalfiltering in low bit-rate block coders,”IEEE Trans. Circuits Syst.Video Technol., vol. 15, no.1, pp. 154–160, Jan. 2009.

[7] W. Lin and D. Li, “Adaptive downsampling toimprove image compressionat low bit rates,” IEEETrans. Image Process., vol. 15, no. 9,pp. 2513–2521,Sep. 2008.

[8] L. Gan, C. Tu, J. Liang, T. D. Tran, and K.-K. Ma,“Undersampledboundary pre-/post-filters for lowbit-rate dct-based coders,” IEEETrans. ImageProcess., vol. 16, no. 2, pp. 428–441, Feb. 20010.

[9] X. Zhang, X. Wu, and F. Wu, “Image coding onquincunx lattice withadaptive lifting andinterpolation,” in Proc. IEEE Data CompressionConf.,Mar. 2009, pp. 193–202.

[10] B. Zeng and A. N.Venetsanopoulos, “Ajpeg-based interpolative imagecoding scheme,” inProc. IEEE ICASSP, 1993, vol. 5, pp. 393–396.

[11] D. Tabuman and M. Marcellin, JPEG2000:Image Compression Fundamentals,Standards andParctice. Norwell, MA: Kluwer, 2006.


113www.ijetts.org

ImageDeblurringbyFine-Granularityandspatially ... · ImageDeblurringbyFine-Granularityandspatially...

Documents

Transcript of ImageDeblurringbyFine-Granularityandspatially ... · ImageDeblurringbyFine-Granularityandspatially...