Fast Non-uniform Deblurring using Constrained Camera Pose ...

Fast Non-uniform Deblurring

Fast Non-uniform Deblurring usingConstrained Camera Pose Subspace

Zhe Hu and Ming-Hsuan Yang

Electrical Engineering and Computer ScienceUniversity of California at Merced

Merced, CA 95343

1 / 41

Fast Non-uniform Deblurring Intro Related Work Proposed Method Results Conclusions

Non-uniform Image Deblurring

Static scene

Long exposure time/dim light condition

Different blur effects on different pixels

2 / 41


Non-uniform Deblurring Algorithms

Segment-wise Uniform Deblur

[Bardsley et al. Optics06], [Cho et al. ICCV07]

Additional Image Sequence/Hybrid Camera System

[Tai et al. CVPR08], [Joshi et al. SIGGRAPH10]

Single Image Deblurring with a Global Model

[Whyte et al. CVPR10], [Gupta et al. ECCV10], [Harmeling et al.NIPS10], [Hirsch et al. ICCV11]

3 / 41


Problem Formulation

Model the blurred input B as the integration of intermediateimages captured at different poses of a motion trajectory [Whyteet al. CVPR10],

B =

∫f (Hθ, L)w(θ)dθ + n (1)

L: latent image, n: noiseθ: camera pose, Hθ: homography induced by camera posef (Hθ, L): transformed copy of the sharp imagew(θ): exposure time at each pose

4 / 41


Discretized Formulation

B =∑θ∈S

f (Hθ, L)wθ + n =∑θ∈S

wθ(KθL) + n (2)

Kθ: matrix that warps latent image L to the transformed copyS : camera pose space

Optimization Problem

min(L,W )

||(∑θ∈S

wθKθ)L− B||2 + Φ1(L) + Φ2(W ) (3)

Φ1, Φ2: regularization terms on L and W

5 / 41


Main Challenges

Lack of good initialization technique of camera motion

High computational load

CPU execution timeof a 441× 611 image

Whyte et al. CVPR10 about 3 hours

Gupta et al. ECCV10 about 1 hour

Hirsch et al. ICCV11 about 1500 secs

6 / 41


Main Steps

1 Initialization

2 Estimating W by fixing L

3 Recovering L by fixing W

7 / 41


1. Initialization

Importance of Initialization

Avoid numerous solutions at local minimum

Fast convergence

Our Approach

Motivated by backprojection in image processing:reconstruct 2D signal from its 1D projections

8 / 41


Initialization Using Backprojection

Computed tomography (CT) views. Each sample acquired in aview is equal to the sum of the image values along a ray pointingto that sample.

9 / 41



A backprojection is formed by duplicating each view through theimage in the direction it was originally acquired.

10 / 41


Initialization

Our Approach

Motivated by backprojection in image processing:reconstruct 2D signal from its 1D projections

Backproject local blur kernels (PSFs) to initialize cameramotion: reconstruct 3D camera motion from 2D blur kernels

11 / 41



Blurred image.

Ground-truth camera motion(in-plane rotation only).

x ,y axis: in-plane translationsz axis: in-plane rotation

12 / 41



Selected regions used to estimatelocal blur kernels.

Corresponding estimated blur kernels.

13 / 41



Blur kernel to be backprojected.

Backprojected camera motion.

14 / 41





15 / 41





16 / 41





17 / 41





18 / 41



Blur kernel kl : projection of camera motion trajectory,

kl =∑θ

wθp(θ, l) (4)

p(θ, l): how camera pose θ affects pixel l

Backprojection: approximate camera motion

bp(kl , l) =∑i

∑{θ|p(θ,l)=i}

kl(i)Γ(θ) (5)

i : element index of blur kernel klΓ(θ): indicator function, 1 for pose θ and 0 for others

Initialization by backprojection:

W =1

n

∑l

bp(kl , l) (6)

19 / 41


Main Steps

1 Initialization



20 / 41


2. Weight Estimation on Constrained Pose Subspace

Camera motion estimation: weight estimation

minW||∑θ∈S

wθ(KθL)− B||2 + Φ2(W ) (7)

Kθ: the matrix that warps latent image L to the transformed copyL, B: column vectors

21 / 41


Weight Estimation on Constrained Pose Subspace

Consider Kθ

Large matrix mn ×mn (image size m × n)

High computational load if |S | is large

Active Set

Motion trajectory is sparse in the pose space

Find a subset A ∈ S to cover the poses lie on the motiontrajectory

Estimate weights on A

22 / 41



How to determine the active set?

Initialize A by thresholding the backprojected camera motion

At each iteration, discard the poses with small weights andadd in poses by sampling

Sampling based on the previous A using a Gaussiandistribution

23 / 41



Active set from last iteration

24 / 41



Discard poses with small weights

25 / 41



Add in new poses by sampling and form the new active set

26 / 41



Original formulation

minW||∑θ∈S

wθ(KθL)− B||2 + Φ2(W ) (8)

Using active set

minW||∑θ∈A

wθ(KθL)− B||2 + Φ2(W ) (9)

Pose numberused per iteration

Execution timeper iteration

warping matrixto be computed

Original Thousands 2 mins ThousandsUsing active set Less than 100 Less than 1 min Around 200 - 300

27 / 41


Main Steps

1 Initialization



28 / 41


3. Recovering Latent Image

Estimating latent image L by fixing weights W as [Gupta et al.ECCV10],

minL||KL− B||2 + Φ1(L) (10)

where K =∑

θ∈A wθKθ is the blur matrix within the active set A

29 / 41


Experimental Results

MATLAB implementation: 600 seconds to process a blurryimage of 441× 611 pixels (3.40 GHz CPU and 16 GB RAM)

10-15 iterations to converge

Results and code are available onhttp://eng.ucmerced.edu/people/zhu/

30 / 41

http://eng.ucmerced.edu/people/zhu/



Input blurred image

31 / 41



[Gupta et al. ECCV10]

32 / 41



[Hirsch et al. ICCV11]

33 / 41



Our method

34 / 41



35 / 41



Input blurred image

36 / 41



[Harmeling et al. NIPS10]

37 / 41



[Hirsch et al. ICCV11]

38 / 41



Our method

39 / 41



40 / 41


Concluding Remarks

Propose a algorithm to remove non-uniform blur using one image

Initialize camera pose effectively using backprojection

Estimate pose weights efficiently by restricting solution spacewith the active set

Perform favorably against the state-of-the-art methods

41 / 41

Fast Non-uniform Deblurring using Constrained Camera Pose ...

Documents

Transcript of Fast Non-uniform Deblurring using Constrained Camera Pose ...