Video Completion Using Block Matching for Video Stabilization

Seunghwan Yoo and Aggelos K.

Katsaggelos Department of Electrical Engineering

and Computer Science Northwestern University, Evanston, IL

60208, USA seunghwanyoo2013@u.northwestern.edu,

aggk@eecs.northwestern.edu

Gwanghyun Jo1, Eunjung Chae1, Hejin Cheong1, Semi Jeon2, Minseo Kim2,

and Joonki Paik1,2

1Department of Image, 2School of Integrative Engineering,

Chung-Ang University, Seoul, Korea jghjgh09@cau.ac.kr, nia38@cau.ac.kr,

hejin87@cau.ac.kr, semi2530@gmail.com, kmse706@cau.ac.kr, paikj@cau.ac.kr

Chanyong Park SIC R&D Laboratory LG Electronics Inc.

Seoul, Korea chanyong.park@lge.com

Abstract— This paper presents a video completion algorithm

using block matching for video stabilization. In order to fill in missing pixels, the proposed algorithm consists of three steps: i) mosaicking for covering the missing static, planar regions, ii) estimation of local motion vectors using the hierarchical Lucas-Kanade optical flow method, and iii) selection of the most similar patch in both spatial and temporal neighbors. The proposed video completion algorithm can be applied in the wide areas of consumer electronics including camcorders, smart phone cameras, tablet cameras, and smart glasses.

Keywords—video stabilization; video completion; block matching; motion inpainting

I. INTRODUCTION

Video stabilization is a fundamental technique to enhance the visual quality of video that contains motion blur and image coordinate jitter by relative motion between the camera and objects. The state-of-the-art completion method for video stabilization fills in missing pixels using motion inpainting [1]. This method propagates the local motion field into missing pixels, and the propagated motion field is used to stitch image patches. Although it enforces both spatial and temporal consistency, accurate estimation of the local motion is difficult or even impossible when objects move faster than the frame rate of the video. This results in erroneous inpainting of the missing regions.

In order to solve this problem, the proposed method estimates the transformation matrix between frames, based on which it performs video stabilization. Missing regions are first generated using the motion field as in [1]. The restoration of the damaged video is then performed using video completion [2]. Specifically, the proposed algorithm extracts patches around the motion vector, and selects the most similar patch in temporally adjacent frames. Missing pixels are then replaced by the selected patch.

II. VIDEO COMPLETION USING BLOCK MATCHING FOR

VIDEO STABILIZATION

The block diagram of the proposed video completion method is shown in Fig. 1.

Fig. 1: The proposed video completion algorithm using block matching

The mosaicking block fills in the static missing region. The variance of the k-th frame is computed as

{ }2

1 1

1( , , ) ( , , ) ( , , )

2

k n M N

s k n x y

v x y k I x y k I x y kn

+

= − = =

= − , (1)

1 1

1( , , ) ( , , )

2 1

k n M N

s k n x y

I x y k I x y kn

+

= − = =

=+

, (2)

where ( , )I x y represents a stabilized input image, ( , )I x y the mean of the stabilized input neighboring frames, and 2 1n + the number of neighboring frames.

If ( , y,k)v x is smaller than a pre-specified threshold L , the corresponding missing pixels are replaced by the median of the neighboring frames.

The second block in Fig. 1 computes the local motion field using the hierarchical Lucas-Kanade optical flow method as [3]

( , ) [ ( , ) ( , )]TF x y u x y v x y= , (3)

where ( , )F x y represents the optical flow field between two adjacent frames. u and v are elements of the flow vector in the x and y directions, respectively.

In order to finally fill in missing pixels, the most similar patch is selected using Patch Likelihood (PL) defined as [2]

IEEE ISCE 2014 1569953799

1

PL ( ) log ( )p ii

I p P I= , (4)

where iP represents a matrix of the i -th extracted patch among all the overlapping patches. log ( )ip P I is the likelihood of the i -th patch given the prior p as

2log ( ) log( ( | , ))i ii

p I N I μ σ= , (5)

where 2( | , )i iN I μ σ represents the density of the Gaussian

distribution with mean iμ and variance 2iσ for the i -th patch.

We minimize the posterior in order to find the reconstructed image using the patch prior p as

2( | ) || || ( )2p pf I J AI J PL Iλ

= − − , (6)

where A represents a diagonal matrix with zeros for all the missing pixels, and J the corrupted image.

We set the regularization parameter as 2/Yλ σ= , where Y is the number of pixels in a patch. The reconstructed image using the patch prior p is given as

1

ˆ T T T T jj j j

j j

I A A P P A J P zλ β λ β

−

= + + . (7)

where z represents the patch iPI and I the resulting image. The regularization parameter β is estimated using the amount

of noise δ , that is 21 /β δ= .

III. EXPERIMENTAL RESULTS

Experimental results of the proposed algorithm which fills in the missing pixels are shown in Fig. 2. Fig. 2 (a) shows a stabilized video frame. The results of the mosaicking and motion inpainting methods are shown in Fig. 2 (b) and Fig. 2 (c), respectively. Finally, Fig. 2 (d) shows the result of the proposed algorithm.

The mosaicking method has some artifacts on dynamic and nonplanar scenes as shown in Fig. 2 (b). The motion inpainting method reduces such artifacts as shown in Fig. 2 (c), but erroneously restored pixel values in the previous frame produce wrong intensity values in the current frame. The proposed algorithm using block matching efficiently reduces error as shown in Fig. 2 (d).

(a) (b)

(c) (d)

Fig. 2 : Comparison of completion results. (a) Stabilized video, (b) The result of a mosaicking method, (c) The result of a motion inpainting method, (d) The proposed algorithm

IV. CONCLUSION

A block matching-based video completion algorithm is presented for restoring missing pixels in a stabilized video. A block matching algorithm selects the most similar patch in the temporally adjacent frames near the missing region, and replaces it with the selected patch.

ACKNOWLEDGMENT

This work was supported by the National Research Foundation of KOREA (NRF) and Center for Women In Science, Engineering and Technology (WISET) Grant funded by the Ministry of Science, ICT & Future Planning of KOREA (MSIP) under the program for the Potential female students with interest in Science connecting with the community of Science and Engineering, by LG Electronics Inc., and by the Ministry of Science, ICT & Future Planning as Software Grand Challenge Project (grant no. 14-824-09-003).

REFERENCES [1] Y. Matsushita, E. Ofek, W. Ge, X. Tang and H. Shum, “Full-Frame

Video Stabilization with Motion Inpainting”, IEEE Transactions on Pattern analysis and Machine Intelligence, vol. 28, no. 7, pp. 1-14, July 2006.

[2] D. Zoran and Y. Weiss, “From learning Models of Natural Image Patches to Whole Image Restoration,” IEEE International Conference on Computer Vision, pp.479-486, November. 2011.

[3] J. Shin, S. Kim, S. Kang, S. Lee, J. Paik, B. Abidi, and M. Abidi, "Optical flow-based real-time object tracking using non-prior training active feature model," Real-Time Imaging, vol. 11, no. 3, pp. 204-218, June 2005.

[4] A. Litvin, J. Konrad, and C. Karl, “Probabilistic Video Stabilization using Kalman Filtering and Mosaicking”, IS&T/SPIE Symposium on Electronic Imaging, Image and Video Communications, pp. 663-674, May 2003.

[5] T. Shiratori, Y. Matsushita, S. Kang, and X. Tang, “Video Completion by Motion Field Transfer”, IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 28, no.7, pp. 1150-1163, March 2006.

[6] S. Kim, E. Chae, E. Lee, W. Kang, H. Cheong, and J. Paik , "Image Enhancement for a Color Shift Model-Based Computational Camera using Patch-Based Inpainting ," International Technical Conference on Circuits/ Systems, Computers and Communications 2013 , pp. 598-599 , June 30, 2013.

2