1 Efficient Reference Frame Selector for H.264 Tien-Ying Kuo, Hsin-Ju Lu IEEE CSVT 2008.

1

Efficient Reference Frame Selector for Efficient Reference Frame Selector for H.264H.264

Tien-Ying Kuo, Hsin-Ju LuTien-Ying Kuo, Hsin-Ju Lu

IEEE CSVT 2008

2

OutlineOutline

IntroductionIntroductionH.264 Inter-CodingH.264 Inter-CodingProposed MethodProposed MethodExperimental ResultsExperimental ResultsConclusionsConclusions

3

IntroductionIntroductionBackground Knowledge (1/4)Background Knowledge (1/4)

H.264 features for inter-frame codingH.264 features for inter-frame codingVariable block size motion compensationVariable block size motion compensationSubpixel motion estimationSubpixel motion estimationMultiple reference frame motion compensationMultiple reference frame motion compensation

N - 5 N - 2 N - 1 Current frame N

4


Why do Multiple reference frame help predictions ?Why do Multiple reference frame help predictions ?Uncovered backgroundUncovered backgroundNon-integer pixel displacementNon-integer pixel displacementLighting changeLighting changeCamera shakingCamera shakingNoises in the source signalNoises in the source signal

Noise effect

5


Sampling/noise effectSampling/noise effect of MobileCalendar of MobileCalendar

Original block (N-1) Ref block (N-2) Ref block

(N-1) Residual block (N-2) Residual block

MSE = 203.8 MSE = 32.4

6


Drawback of MRF-MEDrawback of MRF-MEHigh computational complexityHigh computational complexity

In order to reduce complexityIn order to reduce complexityReduce search pointsReduce search pointsContinuous tracking techniqueContinuous tracking techniqueEarly stop criteriaEarly stop criteria

7

IntroductionIntroductionMain Purpose (1/1)Main Purpose (1/1)

A simple and effective method of selecting proper A simple and effective method of selecting proper reference frames in MRF-ME.reference frames in MRF-ME.

It enables working with any existing ME It enables working with any existing ME algorithms.algorithms.

Experimental results demonstrate the effectiveness Experimental results demonstrate the effectiveness of proposed algorithm.of proposed algorithm.

8

H.264 Inter-CodingH.264 Inter-CodingOverview (1/3)Overview (1/3)

Variable block sizeVariable block sizeMacroblock partition : 16*16, 16*8, 8*16, 8*8Macroblock partition : 16*16, 16*8, 8*16, 8*8Submacroblock partition : 8*8, 8*4, 4*8, 4*4Submacroblock partition : 8*8, 8*4, 4*8, 4*4

16*16 16*8 8*16 8*8

8*8 8*4 4*8 4*4

9


Reference parameter Reference parameter REFREFSignal to which frame referredSignal to which frame referredCoded only once for each submacroblock partitionCoded only once for each submacroblock partition

All subblock smaller than 8*8 in the same subpartition must refer to the same frame

10


Rate-distortion cost of each possible partitionRate-distortion cost of each possible partition

))())()(((

)))(,(,()(),(

REFRREFpREFmR

REFmREFcsDTSAREFJ

REF : Reference parameter

m( REF ) : The motion vector in the reference frame

p ( REF ) : The predicted motion vector from the neighbors

s : Original video signal

c : Coded video signal

R : Rate function of motion vectors

SA(T)D : SAD or SATD

Reference frame selectorReference frame selectorTreating 8*8 block as a minimal unitTreating 8*8 block as a minimal unitMaking the selection using a mode 8*8 motion searchMaking the selection using a mode 8*8 motion search

11

Proposed MethodProposed MethodMultiple Reference Frame Selection Multiple Reference Frame Selection

(1/9)(1/9)

Flow chartFlow chart

12

Proposed MethodProposed MethodMultiple Reference Frame Selection Multiple Reference Frame Selection

(2/9)(2/9)

Input a MB and perform four 8*8 block motion searchon reference frame t-1

Variance of four MVs of 8*8 blocks TH?≦

Perform motion estimation on reference frame t-1

Enter 2nd stage

YesNo

Output results

First Stage

Flow chartFlow chart

13

Proposed MethodProposed Method Multiple Reference Frame Selection Multiple Reference Frame Selection

(3/9)(3/9)Perform four 8*8 block motion search of the MB on

the reference frame t-i

Yes

No

i = 2

Is i N?≧i + 1

Is ref-frame t-k reffered by one of the four 8*8 blocks?

Drop the non-referred ref frame t-k Drop the non-referred ref frame t-k

Is k N?≧

Perform motion estimation on each valid reference frame

Output results

Second Stage

Yes

YesNo

No

First stageFirst stageDisjoint 16*16 macroblock into four 8*8 blocksDisjoint 16*16 macroblock into four 8*8 blocksMotion search on ref-frame t-1 and check thresholdMotion search on ref-frame t-1 and check threshold

14


(4/9)(4/9)

Current Frame tReference frame t - 1

4 MVs and corresponding minimal R-D cost

Motion vectorsMotion vectors

R-D costR-D cost

Check x and y components of Check x and y components of V V t-1t-1

Less than a small threshold ?Less than a small threshold ? Encoder terminate early by designating only the Encoder terminate early by designating only the previous frame as reference frameprevious frame as reference frame

15


(5/9)(5/9)

3,2,1,0|11 iVV ti

t

3,2,1,0|)( 11 iVCC ti

t

Early stop criteria

The design of threshold depends upon the The design of threshold depends upon the computational capacity of the encodercomputational capacity of the encoderLarger threshold, lower complexity, but worse coding Larger threshold, lower complexity, but worse coding

efficiencyefficiency

16


(6/9)(6/9)

Determine upper bound of the coding efficiency and the worst case of complexity

Second stageSecond stageMotion search of mode 8*8 should be tested on all of Motion search of mode 8*8 should be tested on all of

the remaining reference framesthe remaining reference frames

17


(7/9)(7/9)

Nti

ti

ti

ti VVVV ,...,,, 432

NkiVC kti ,...,3,2;3,2,1,0|

N : The maximal number of reference frames

For a given block index i, we let k be set asFor a given block index i, we let k be set as

The frame selector sets the ref-frame n-k as the The frame selector sets the ref-frame n-k as the valid, qualified reference framevalid, qualified reference frame

18


(8/9)(8/9)

)(minarg,...,2,1

kni

NkVCk

Block i has the best motion vector with the lowest cost,

by referring to reference frame n-k

Drop unqualified framesDrop unqualified frames

19


(9/9)(9/9)

Current frame tFrame t-2Frame t-3Frame t-4Frame t-5

N = 5

Verifying the effectiveness of frame selectorVerifying the effectiveness of frame selectorMeasuring the Measuring the hit ratehit rate by comparing retained frames by comparing retained frames

with the actual frames used via exhaustive search with the actual frames used via exhaustive search method (hit rate = 88% ~ 95%)method (hit rate = 88% ~ 95%)

20

Proposed MethodProposed Method Analysis hit rate and Frame Analysis hit rate and Frame

usage(1/6)usage(1/6)

Even with the high hit rate, we expect that frame Even with the high hit rate, we expect that frame selector can drop as many frames as possible.selector can drop as many frames as possible.The The false alarmfalse alarm ranges from 13%~32% ranges from 13%~32%

21



Reference frame usage of the proposed frame Reference frame usage of the proposed frame selector for various video sequenceselector for various video sequence

22



Why using mode 8*8 but not other modes in Why using mode 8*8 but not other modes in frame selection?frame selection?Hit rateHit rateFalse alarmFalse alarmMotion estimation time spendingMotion estimation time spending

23



Analysis of hit rate and false alarm using different Analysis of hit rate and false alarm using different modes on various sequencemodes on various sequence

24



The motion estimation time spending on a MBThe motion estimation time spending on a MB

Judging from the Judging from the hit ratehit rate, , falsefalse alarmalarm and and complexitycomplexity8*8 is the best choice8*8 is the best choice

25



ModeMode 16*1616*16 16*816*8 8*168*16 8*88*8 8*48*4 4*84*8 4*44*4

TimeTime 0.980.98 0.980.98 0.990.99 11 1.031.03 1.031.03 1.081.08

Reference software : JM 9.2Reference software : JM 9.2 Intel Pentium 4 3.0 GHz with 512MB RAMIntel Pentium 4 3.0 GHz with 512MB RAM Encoder parametersEncoder parameters

26

Experimental ResultsExperimental Results Overview (1/2)Overview (1/2)

Using Fast Full Search (FFS) and Fast Motion Using Fast Full Search (FFS) and Fast Motion Estimation (FME) in JM 9.2Estimation (FME) in JM 9.2

Comparing the results with a frame selection Comparing the results with a frame selection method of Li.[1]method of Li.[1]

27

Experimental ResultsExperimental Results Overview (2/2)Overview (2/2)

[1] X. Li, E. Q. Li, and Y. K. Chen, “Fast multi-frame motion estimation algorithm with adaptive search strategies in H.264 ,”in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., May 2004, vol.3, pp.369–372.

R-D curve comparison (Foreman)R-D curve comparison (Foreman)

28

Experimental ResultsExperimental Results

R-D curve comparison (Mobile)R-D curve comparison (Mobile)

29


Using BDPSNR and BDBR to measure the Using BDPSNR and BDBR to measure the performance difference between the methodsperformance difference between the methodsCalculates average PSNR and bitrate distance between Calculates average PSNR and bitrate distance between

two RD curves of two method, respectively.two RD curves of two method, respectively.

30


Discussing the computational complexityDiscussing the computational complexity

31


Speed up of SAD and SATD calculationsSpeed up of SAD and SATD calculations

32


Number of MBs references in each reference Number of MBs references in each reference frame (Mobile)frame (Mobile)

33


Number of MBs references in each reference Number of MBs references in each reference frame (Foreman)frame (Foreman)

34


An efficient reference frame selector is An efficient reference frame selector is proposed for the h.264 encoder to deal with proposed for the h.264 encoder to deal with the complexity issue pertaining to MRFME.the complexity issue pertaining to MRFME.

The experimental results demonstrate that the The experimental results demonstrate that the proposed algorithm can reduce significantly proposed algorithm can reduce significantly the complexity of ME at the encoder end, the complexity of ME at the encoder end, while keeping almost the same R-D while keeping almost the same R-D performance as FFS.performance as FFS.

35

ConclusionsConclusions

1 Efficient Reference Frame Selector for H.264 Tien-Ying Kuo, Hsin-Ju Lu IEEE CSVT 2008.

Documents

Transcript of 1 Efficient Reference Frame Selector for H.264 Tien-Ying Kuo, Hsin-Ju Lu IEEE CSVT 2008.