LOCALIZED AND COMPUTATIONALLY EFFICIENT APPROACH TO

SHIFT-VARIANT IMAGE DEBLURRING

ICIP, Oct. 2008, San Diego

Murali SubbaRao, Youn-sik Kang, Satyaki Dutta*, Xue Tu

{murali , yskang, tuxue}@ece.sunysb.edu,*sunny@math.sunysb.edu

Dept. of Electrical and Computer Engineering*Dept. of Mathematics

State University of New York at Stony Brook, NY 11794

Problem: Shift/Space-Variant Image Deblurring

• Example: Actual Image of a slanted planar object captured by a cell-phone camera

• Blurring Point Spread Function (PSF) is different at each pixel.• Computationally efficient shift-variant image deblurring is

needed.edgeleft near 55.2}Max{ ≈σ

Image Captured by a Cellular phone Image Restored by RT

areacenter near 1}min{ ≈σ

Causes and Applications

• Causes of Shift-Variant Blur– 3D shape of objects and limited depth of field of cameras result in

shift-variant defocus blur.– Motion of 3D objects causes shift-variant motion blur.– Aberrations of Optical Imaging Systems: e.g.: defocus, coma,

astigmatism, and spherical.– atmospheric turbulence in astronomical telescopes.

• Applications of Shift-Variant Deblurring– Digital Image/Video Cameras and Cellphone cameras: Image

deblurring.– Machine Vision: Shape-from-Defocus, Inverse Imaging– Medical Imaging: Microscopes, Endoscopes– Scientific: Telescopes (atmospheric turbulence causes shift-

variant blur in astronomy)– Solution of Integral and Differential Equations.

COMPARISON• Previous approaches

– Partial (not full) localization: e.g. divide a large image into 32x32 blocks and carry-out shift variant deblurring in each block separately.

– Often does not exploit the locality and limited support domain of the blurring kernel PSF,

– Often does not exploit the spatial smoothness of focused images (formulation includes redundant unknown parameters).

– Computationally exorbitant and less accurate.

• New Approach– Avoids the disadvantages listed above for the previous

approaches.– Represents a fundamental theoretical and computational advance.– relevant to solving integral/differential equations and shape from

defocus.– Extension of S Transform for shift-invariant deblurring proposed

by Subbarao in 1989.

Shift-Variant Blurring Model

Conventional Model: Fredholm Integral Equation of the First Kind

f ( ): Focused imagek( ) : Shift-Variant PSF (blurring kernel)g( ) : blurred image

Problems: – k( ) is in global form.– inverting the equation to obtain f( ) is difficult.– Discrete form of the above equation is often used to solve for f( )

in small (e.g. 32x32) image blocks using a Singular Value Decomposition (SVD) technique (e.g. spectral filtering).

– Computational cost is exorbitant, and accuracy may be limited.

( ) ( ) ( )b d

a cg x y k x y u v f u v d u d v, = , , , , .∫ ∫

Rao Transform (RT) Theory and Algorithm

Define a completely localized kernel

Reformulate the problem in an equivalent form that definesRao Transform (RT):

In this form, it becomes possible to invert this integral transform locally under the assumption of analyticity of images.

f( ) is expanded in a Taylor-series at f(x,y), and order or integration and summation are interchanged.

Various order derivatives of g(x,y) are consedered.

( , , , ) ( , , )h x y u v k x u y v x y= + , +

( ) ( ) ( )x a y c

x b y dg x y h x u y v u v f x u y v du dv

− −

− −, = − , − , , − , − .∫ ∫

RT Theory and Algorithm

Blurred image can be expressed as the weighted sum of the derivatives of the focused images where the weights are determined by the given shift-variant PSF (SV-PSF):

By considering the derivatives of g( ), the following linear system of equations can be obtained:

( )

0 0( )

N nn i i

n in i

g x y S f − ,,

= =

, ≈ ∑ ∑

( 0 , 0 ) ( 0 , 0 )0 0 0 1

( 1 , 0 ) ( 1 , 0 )1 0 1 1

( 0 , ) ( 0 , )N N

g fr rr rg f

g f

⎡ ⎤ ⎡ ⎤⎡ ⎤⎢ ⎥ ⎢ ⎥⎢ ⎥⎢ ⎥ ⎢ ⎥⎢ ⎥=⎢ ⎥ ⎢ ⎥⎢ ⎥⎢ ⎥ ⎢ ⎥⎢ ⎥⎢ ⎥ ⎢ ⎥⎣ ⎦⎣ ⎦ ⎣ ⎦

, , ,x y x y x y=g R f

Inverse RT

Forward RT:

Inverse RT:

Focused Image:

A regularization technique (e.g. SVD based spectral filtering) can be used to obtain a better estimate of inverse RT while inverting . This yields a new localized regularization technique.

, , ,x y x y x y=g R f

, , ,x y x y x y′=f R g 1, ,( )x y x y

−′ =R R

( )

0 0( )

N nn i i

n in i

f x y gS − ,,

= =

, = ′∑∑

,x yR

RT/IRT Example: Symmetric SV-PSFBlurred image:

Closed-form solution to Focused Image:

We are not aware of any previous work that presents such a localized closed-form solution, or one that has similar

computational advantages.

( 0 0 ) ( 0 0 ) ( 1 0 ) ( 1 0 ) ( 0 1 ) ( 0 1 )2 0 0 2

( 2 0 ) ( 0 0 ) ( 0 2 ) ( 0 0 )2 0 0 2

1 1 . . .2 2

g f f h f h

f h f h H O T s

, , , , , ,, ,

, , , ,, ,

= + +

+ + +

( 0 0 ) ( 0 0 ) ( 1 0 ) ( 1 0 ) ( 0 1 ) ( 0 1 )2 0 0 2

( 2 0 ) ( 1 0 ) 2 ( 0 1 ) ( 0 1 ) ( 0 0 )2 0 0 2 2 0 2 0

( 0 1 ) 2 ( 1 0 ) ( 1 0 ) ( 0 0 )( 0 2 )0 2 2 0 0 2 0 2

3 1 1( )2 2 2

3 1 1( )2 2 2

. . . ( H i g h e r O r d e r T e r m s

f g g h g h

g h h h h

g h h h h

H O T s

, , , , , ,, ,

⎛ ⎞⎜ ⎟, , , , ,⎜ ⎟, , , ,⎜ ⎟⎝ ⎠

⎛ ⎞⎜ ⎟, , , ,,⎜ ⎟, , , ,⎜ ⎟⎝ ⎠

= − −

+ + −

+ + −

+ )

Analysis of RT/IRTIf higher order derivatives of g( ) used in the computation of IRT

are erroneous (due to noise, usually so), then, solution for f ( )will have errors, even if a localized regularization technique is used.

The region of significant support of the blurring kernel often satisfies the condition (e.g. blur circle radius or sigma)

The error will be low if, within the region of significant support of the blurring kernel, Taylor-series expansion of f( ) in RT provides a good approximation to the actual product under the integral:

Further research has provided some improved solutions to theseproblems. They will be presented in a future publication.

( ) ( )h x u y v u v f x u y v− , − , , − , −

( ) 0 for | | or | |h x u y v u v u R v R− , − , , ≈ > >

Analysis of Computational Complexity

For a conventional SVD method based on spectral filtering, a 32x32 image requires inverting a matrix, i.e. O(32^6)=O(2^30).

In the RT approach, computations are dominated by the inversionof NxN size matrix , once at each of the 32^2 pixels. N is the order up to which the derivatives of g( ) are used.Setting N=4, the computational complexity becomes O(16^3 X 32^2)=O(2^22).

In this example, approximate computational advantage is a factorof 2^8=256. In our MATLAB implementation, without code optimization, we have observed a speed up of 2 to 4.

2 232 32×

,x yR

Experiment on 1-D RT: Simulated blur

Input: sine wave,PSF: Gaussian, sigma =0.5+0.1x , x_max =10, N=8, M=2

Input Shift variant blurred input

Restored output

Experiment on 2-D RT: Simulated blur

• Input image size :469x188,• PSF: Gaussian, • N=4 M=2• Sigma_max =2.8,

Sigma_min = 0.2

Input focused image

Result of image restoration by RT

Shift-variant Gaussian blurred image

Experiment on 2-D RT: Real Image

Input image size :640x480PSF: Gaussian, N=4 M=2Sigma_max =2.5, Sigma_min = 0.5

Result of image restoration by RTShift-variant real input image

RT on Cell-phone image

• Image size: 640 x 480 • Shift Variant PSF

plate of edgeright in 7.2}Max{ ≈σ

Image captured by a Cell phone Image restored by RT

area left wallin 35.1}min{ ≈σ

RT on Cell-phone image

• Image size: 640 x 480 • Shift Variant PSF

edgeleft near 55.2}Max{ ≈σ

Image captured by a Cell phone Image restored by RT

areacenter near 1}min{ ≈σ

Example of Ongoing Research: Extending RT for Aberration PSFs.

Aberration PSF: Coma

Cylindrical PSF: Defocus Gaussian PSF: Defocus

Aberration PSF: Astigmatism

Conclusion• A novel approach has been presented for shift-variant image

deblurring. It naturally exploits the spatial locality of blurring PSFs and smoothness of underlying focused images.

• A central idea is the formulation of the problem in a completelylocalized form. It represents a fundamental theoretical and computational advance.

• A closed-form solution is provided that is computationally efficient and usually accurate.

• The RT formulation provides new insights, and a new local regularization technique.

• RT provides a new method of solving integral equations (and therefore differential equations).

• This approach can be extended to shape-from-defocus techniques in the case of shift-variant blur.

• This approach is being extended in several ways through further research.