Enhanced Motion Compensated FrameInterpolation Using Object Layer Inference

T.-S. Wang, K.-S. Choi, H.-S. Jang and S.-J. Ko

Electronics LettersSponsored by Institution of Engineering and Technology

IET Journal

OutlineIntroductionProblem StatementProposed MCI MethodExperimental ResultsConclusions

IntroductionIn various MCFI techniques, even though

reliable motion is estimated in ME, the interpolated frame can be degraded by annoying artifacts(blocky ,flicker).

This is mainly because MCFI can utilize wrong motion information around the occlusion region and the motion ambiguity (MA) region.

Problem StatementIn [4][5](Wang’s & Haan’s methods ),

conventional methods produce the annoying artifacts since they cannot actually discriminate the FG and BG layers.

We utilize the occlusion estimation method [6] to identify the occlusion and to remove the inaccurate MVs caused by the occlusion.

[4] Wang, D.: ‘Comparison of motion-compensated algorithms for frame interpolation’, Opt. Eng., 2003, 42, pp. 586–590[5] Haan, G.D., and Ojo, O.A.: ‘Robust motion-compensated video up-conversion’, IEEE Trans. Consume. Electron., 1997, 43, pp. 1045–1056[6] Ince, S., and Konrad, J.: ‘Geometry-based estimation of occlusion from video frame pairs’. Proc. IEEE Int. Conf. Philadelphia, PA, USA, March 2005, Vol. II, pp. 933–936

In [4], MV having the lowest SAD is selected to solve MA on the assumption that the FG object produces the lowest SAD.

In [5], the median of pixel values associated with the multiple MV candidates is exploited for the interpolated pixel value.

BG and FG objects can pass through a single location in the interpolated frame.

To interpolate the MA region correctly, the motion of the FG object should be correctly chosen.

Problem Statement (cont.)

Inaccurate MVs caused by the occlusion is removed by [6].

Proposed MCI MethodMA occurs whenever the magnitude of

relative motion between the FG and BG layer, ||vFG − vBG||, is larger than the size of the FG object.

The FG object always moves away from the occlusion belonging to the BG layer(U).

Ox : Set of pixel positions of uncovered (U) in fn+1

Sx : Set of the positions which the multiple MVs causing MA pass through in fn+1

Ox Sx

Proposed MCI Method (cont.)First find a position within the occlusion, xoc Ox,

such that ||xoc − xi || is minimal, xi Sx.Then, the element of Sx , farthest from xoc, is

associated with the FG object.For the occlusion regions (no MV passes), the MV

of the BG is easily estimated using the neighboring MV.

Ox Sx

Proposed MCI Method (cont.2)However, such a straightforward approach to the

distance calculation is computationally expensive.Using a simple morphological operator as follows:

1. The Ox area is first do the dilation by repeating with the 3×3 structuring element. At each iteration, the dilated pixels are labeled with the number of the iteration.

2. When the MA occurs, Sx is obtained and the distance label is examined. The pixel with the largest label value is utilized to interpolate the frame(FG).

3. Step 2 is repeated until the overlapped area is completely interpolated.

Proposed MCI Method (cont.2)

Ox Sx

Experimental Results

Experimental Results

ConclusionsWe propose a novel layer inference-based

MCI method using the occlusion information.

Experimental results confirm that the proposed method achieves much higher visual quality of the interpolated frame without annoying artifacts in both the MA and occlusion regions