Modeling and Evaluating Feedback-Based Error Control for Video Transfer

Ph.D. Dissertation Defense

Modeling and Evaluating Feedback-Based Error Control for Video

Transfer PhD Candidate:

Yubing Wang - Computer Science, WPI, EMC Corp.

Committee:

Prof. Mark Claypool - Computer Science, WPI

Prof. Robert Kinicki - Computer Science, WPI

Prof. Dan Dougherty - Computer Science, WPI

Prof. Ketan Mayer-Patel – Computer Science, UNC at Chapel Hill

2Ph.D. Dissertation Defense

Video TransferVideo Transfer

5

Video Frames

InternetInternet

Client

4 3 2 1

Frame Loss

Capacity Constraint

Server

5 3 1

Delay Constraint

Too Late

Error Propagation

4 3 2 15


Error ControlError Control

5

Video Frames

InternetInternet

Client

4 3 2 1

Server

Retransmission

NACK

3

Change Coding Parameter

Local Concealment

3


MotivationMotivation

Frame loss degrades video qualityFeedback-based error control techniques use information from decoder to repair

Feedback indicates damage location.Encoder and decoder cooperate in error control process.Better than error control techniques where no interaction between encoder and decoder Major techniques: RPS, Intra Update, RetransmissionChoice and Effectiveness depends on packet loss, RTT, video content and GOP size

No systematic exploration and comparison of impact of video and network conditions on the performance of feedback-based error control techniques


The DissertationThe Dissertation

Analyze video quality with feedback based error controlDevelop analytical models to predict quality of videos streamed with RPS NACK, RPS ACK, Intra Update or RetransmissionConduct systematic study of effects of reference distance on video qualityValidate analytical models through simulationsAnalysis of loss rate, round-trip time, video content, Group Of Pictures (GOP)Determine choice between RPS NACK, RPS ACK, Intra Update or RetransmissionPublications

“Impact of Reference Distance for Motion Compensation Prediction on Video Quality”, MMCN07“An Analytic Comparison of RPS Video Repair”, MMCN08“Modeling RPS and Evaluating Video Repair with VQM”, IEEE Transactions on Multimedia, 2009, (to appear)


OutlineOutline

Introduction

Background RPS ACKRPS NACKIntra UpdateRetransmission

Impact of Reference Distance on Video QualityAnalytical Models and ResultsModel ValidationsConclusions


Reference Picture Selection (ACK)Reference Picture Selection (ACK)

The decoder acknowledges all correctly received frames Only the acknowledged frames are used as a reference Error propagation is avoided entirely Distance from reference frame is reference distance Reference distance increases with round-trip delay Coding efficiency decreases as reference distance increases Video quality degrades as coding efficiency decreases

1 2 3 4 5 6 7

ACK(1) ACK(2) ACK(3)


Reference Picture Selection (NACK)Reference Picture Selection (NACK)

The previous frame is used as a reference for encoding during the error-free transmission. Reference distance is always 1 regardless of RTT

The decoder sends a NACK for the erroneous frame along with a reference frame number Error propagation Impact of loss increases with RTT

NACK(3)

1 2 3 4 5 6 7 8


Intra UpdateIntra Update

Upon receiving a NACK from the decoder, encodes the current frame with intra mode

Frame is independently encoded without using any information from previous frames

Coding efficiency is reduced because of intra coding

1 2 3 4 5 6 7 8 9

NACK(4)

Intra-coded


RetransmissionRetransmission

Retransmission of lost frames needs extra bandwidth Packets arriving after their display times are not discarded but

instead are used to reduce error propagation

1 2 3 4 5 6 7 8 9

1 2 3 4 5 6 7 8 9

Encoder

DecoderNACK(3)

3


OutlineOutline

IntroductionBackgroundImpact of Reference Distance on Video Quality

HypothesisMethodologyResults and Analysis

Analytical Models and ResultsModel ValidationsConclusions


Impact of Reference Distance on Video QualityImpact of Reference Distance on Video Quality

RPS selects one of several previous frames as a reference frame during compression

Distance from selected frame is reference distance

Higher reference distance, lower quality and vice versa

How reference distance affects video quality has not been quantified

A systematic study of the effects of reference distance on video quality

Data is needed for modeling RPS


HypothesisHypothesis

The y-intersect is determined by motion and scene complexity.

High-motion video sequences starts with low quality, degrade slower.

Low-motion video sequence starts with high quality, degrade faster.

Low Motion:

The similarities among frames are high;

More macro-blocks are inter-coded;

High motion:

The similarities among frames are low;

More macro-blocks are intra-coded;


MethodologyMethodology

Select a set of non-compressed video clips with a variety of motion content.

All in YUV 4:2:2, CIF (352x288)Each video sequence contains 300 video frames with a frame rate of 30 fps.

Change reference distances for each selected video sequenceEncode the video clips using H.264 Measure video quality using

Peak-Signal-to-Noise-Ratio (PSNR) Video Quality Metric (VQM)

Analyze the results.


PSNR vs. Reference DistancePSNR vs. Reference Distance

Video Clips a b R-Squared

Akiyo -2.0116 47.965 0.9953

Container -1.9023 44.838 0.9948

News -1.8556 43.295 0.9984

Silent -1.5283 41.41 0.9929

Mom & Daughter

-1.4581 41.442 0.9904

Foreman -1.1681 38.511 0.9265

Mobile -1.1553 26.663 0.9754

Coastguard -0.8626 35.582 0.9975

The relationship between PSNR and reference distance can be characterized using a logarithmic function: bxay )ln(

y = -0.8626Ln(x) + 35.582

y = -1.4581Ln(x) + 41.442

y = -2.0116Ln(x) + 47.965

30

35

40

45

50

1 2 3 4 5 6 7 8Reference Distance

PS

NR

(db)

Akiyo

mom_daut

coastguard


VQM vs. Reference DistanceVQM vs. Reference Distance

Video Clips a b R-Squared

Akiyo -0.0113 0.9847 0.9869

Container -0.0114 0.9766 0.9848

News -0.0115 0.9732 0.9931

Silent -0.0124 0.9606 0.9937

Mom & Daughter

-0.0085 0.9217 0.9821

Foreman -0.0068 0.9059 0.9779

Mobile -0.0022 0.8055 0.9076

Coastguard -0.0014 0.8423 0.9671

The relationship between VQM and reference distance can be characterized using a linear function: baxy

y = -0.0113x + 0.9847

y = -0.0085x + 0.9217

y = -0.0014x + 0.84230.82

0.84

0.86

0.88

0.9

0.92

0.94

0.96

0.98

1

1 2 3 4 5 6 7 8

Reference Distance

1-VQ

M

akiyo

mom_daut

coastguard


OutlineOutline

IntroductionBackgroundImpact of Ref. Distance on Video QualityAnalytical Models and Results

AssumptionsRPS ACKRPS NACKIntra UpdateRetransmissionResult & Analysis

Model ValidationsConclusions


AssumptionsAssumptionsEach GOB is independent from other GOBs in the same frame.An independent video sub-sequence is referred to as a reference chain.Each GOB is carried in a single network packet.Reliable transmission of feedback messages are assumed. Erroneously-decoded GOBs are repaired by local concealment.Make no assumption on specific local concealment techniques.

1 2 3 4 5 6 7

Assume independent packet loss with a random loss distribution.

In this talk, GOB and Frame is exchangeable.


Model ParametersModel Parameters


Modeling of RPS ACKModeling of RPS ACK

The probability of decoding GOB (n) correctly using GOB (n-δ-i) as a reference:

The probability of GOB (n) being successfully decoded is:

p Packet loss probability

Probability of GOB (n-δ-i) being successfully decoded

Round-trip time

Time-interval between two frames

inq

RTTt

INTt

10,)1( niqpp ini

INT

RTT

t

t

pqn 1

ACK(1)

1 2 3 4 5

ACK(2)


RPS ACK Modeling (cont.)RPS ACK Modeling (cont.)

The expected video quality for n-th GOB:

Average video quality for a GOB encoded using the GOB that is r GOBs backward.

Average video quality for a Intra-Coded GOB

Average PSNR value for a GOB that is repaired using local concealment

nUpUp

nUpqppUQ

n

iin

ii

n

,*)1(

,*)1(

'0

'1

0

rU

0U'U


RPS NACK -- ModelRPS NACK -- Model The probability of GOB (n) being successfully decoded:

np

nqqq

n

n

iinn

n

,)1(

,1

0,1, inq , --- the probability of decoding GOB (n) correctly

using GOB (n- δ -i) as a reference

1- p

1- p

1- p

1- p1- p1- p1- p

1- p

p

p p

p pp

[1]

[1](1)

[2](1)(1)

[1]

(2)

(2)(1)(1) (3) [3]

GOB 1

GOB 2

GOB 3

GOB 4

(2)

p

p

root

[1]

A

B

C

D

NACK(1)

1 2 3 4 5

NACK(2)

GOB Dependency Tree


Intra Update -- ModelIntra Update -- Model The probability of GOB (n) being successfully decoded:

np

nqqq

n

INTRAnn

n,)1(

,,1, INTRAnq , -- the probability of decoding GOB (n) correctly using Intra coding

1 2 3 4 5

NACK

Intra-coded

1- p

1- p

1- p

1- p1- p1- p1- p

1- p

p

p p

p pp

GOB 1

GOB 2

GOB 3

GOB 4

p

p

A E

B

root

C

D

F

GOB Dependency Tree


RetransmissionRetransmission

pNN

pNCC

CNp

pNNC

N

NpppC

RRG

GE

G

RRGE

G

RRE

)1(*

,)1(

)(*)]1(*)(1[* 32

2),1(

11),1(1'1

1

'1

RRNN

RRnn

nNnqUqU

NnqUqUQ

RRRR

Capacity constraint:

The n-th GOB in the reference chain being successfully decoded:

11, RRn

n Nnqq

2,1 RR

Nn Nnqq RR

The expected video quality for GOB (n):


OutlineOutline

IntroductionBackgroundImpact of Ref. Distance on Video QualityAnalytical Models and Results

AssumptionsRPS ACKRPS NACKIntra UpdateRetransmissionResult & Analysis

Model ValidationsConclusions


Analytic ExperimentsAnalytic Experiments

Our analytical models consider a number of factors that may affect feedback-based repair performance:

Reference distance change

Loss probability

Round-trip time

Bitrate constraint

Video content

GOP Size

Select a set of video clips

with a variety of motion content


Quality versus Round-Trip TimeQuality versus Round-Trip Time

30

32

34

36

38

40

42

44

40 80 120 160 200 240 280 320 360 400

Round Trip Time (ms)

PS

NR

(db)

p=0.01

p=0.05

p=0.1

p=0.2

25

27

29

31

33

35

37

39

41

43

45

40 80 120 160 200 240 280 320 360 400


PS

NR

(db

)

P=0.01

p=0.05

p=0.1

p=0.2

RPS ACK RPS NACK

Quality degrades with round-trip time increase

NACK resistant to degradation with round-trip time for low loss

ACK degrades uniformly with round-trip time


Quality versus Loss RateQuality versus Loss Rate

30

32

34

36

38

40

42

44

46

0 0.05 0.1 0.15 0.2

Loss (fraction)

PS

NR

(db

)

RTT=80ms

RTT=160ms

RTT=240ms

RTT=320ms

30

32

34

36

38

40

42

44

0 0.05 0.1 0.15 0.2

Loss (fraction)

PS

NR

(db

)

RTT=80ms

RTT=160ms

RTT=240ms

RTT=320ms

RPS ACK RPS NACK

Quality degrades with loss rate increase

NACK degrades faster with high round trip times

ACK uniform degradation


RPS NACK vs. RPS ACKRPS NACK vs. RPS ACK

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0.1

80 120 160 200 240 280 320 360 400


Lo

ss C

ross-o

ver

Container News Silent Mom-Daughter Foreman Mobile

Above trend line, ACK better. Below trend line, NACK better

Crossover points for low-motion are higher than for high-motion

Error propagation more harmful to quality than reference distance


ComparisonComparison

27

29

31

33

35

37

39

41

43

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2

Loss Rate

PS

NR

(db)

RPS NACK Intra Update

RPS ACK Retransmission

32

34

36

38

40

42

44

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2

Loss Rate

PS

NR

(db

)

RPS NACK Intra Update

RPS ACK Retransmission

RPS NACK performs best in low loss

RPS ACK performs best in high loss

RPS ACK performs worst in low loss

Retransmission performs worst in high loss

Intra Update performs as well as RPS NACK as RTT increases

RTT=80 ms

RTT=240 ms


OutlineOutline

IntroductionBackgroundImpact of Ref. Distance on Video QualityAnalytical Models and ResultsModel Validations

MethodologyResults

Conclusions


Validation -- MethodologyValidation -- Methodology

Randomly drop controllable number of frames in input sequence based on given loss probabilityBased on given round-trip time and randomly selected lost frames, regenerate video sequenceEncode video sequence generated in step 2 using H.264Measure average PSNR and VQM for encoded H.264 video sequenceCalculate average PSNR and VQM based upon video quality measured in step 4

1(I) 2(P) 5(P) 6(P) 7(P)

RPS NACK, round-trip time = 2 frames, frame 3 is lost


Validation – RPS NACKValidation – RPS NACK

25

27

29

31

33

35

37

39

41

43

45

0 0.05 0.1 0.15 0.2 0.25 0.3

Loss (Fraction)

PS

NR

(db

)

RTT=80 ms(simulation) RTT=240 ms(simulation)

RTT=80 ms(model) RTT=240 ms(model)

Error bar represents 95% confidence interval

As loss probability or round-trip time increases, the variance is increased

Simulation results are consistent with values predicted by analytical model for both PSNR and VQM

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

0 0.05 0.1 0.15 0.2

Loss (Fraction)

VQ

M

RTT=80ms (experiment) RTT=240ms (experiment)

RTT=80ms (model) RTT=240ms (model)


OutlineOutline

IntroductionBackgroundImpact of Ref. Distance on Video QualityAnalytical Models and ResultsModel ValidationsConclusions


Major ContributionsMajor Contributions

1. Systematic study of effects of reference distance on video quality for a range of video coding conditions

2. Two utility functions that characterize impact of reference distance on video quality based upon study

3. Modeling prediction dependency among GOBs for RPS NACK and Intra Update using binary tree

4. Analytical models for feedback-based error control techniques including Full Retransmission, Partial Retransmission, RPS ACK, RPS NACK and Intra Update

5. Simulations that verify accuracy of our analytical models

6. Analytic experiments over a range of loss rates, round-trip times and video content using our models


Future WorkFuture Work

Explore and incorporate other existing video quality metrics or develop a new quality metricInvestigate how local concealment may affect the choice of feedback-based repair techniquesInvestigate the impact of the extra bandwidth consumed by feedback messages on performanceBuild a videoconference system that automatically adapts to the best repair techniques


ConclusionsConclusionsDegree of video quality degradation is affected by video content

High-motion video sequences starts with lower quality, degrade slower.Low-motion video sequences starts with higher quality, degrade more rapidly.Mathematical Characterization of the relationship between video quality and reference distance:

PSNR:

VQM:

Analytical models reveal: RPS NACK performs best in low lossRPS ACK performs best in high loss, worst in low lossRPS NACK outperforms RPS ACK over a wider range for low motion videos than for high motion videosRetransmission performs worst in high loss Intra Update performs as well as RPS NACK as RTT increases

bxay )ln(baxy


AcknowledgeAcknowledge

Prof. Claypool and Prof. Kinicki

Prof. Dougherty

Prof. Mayer-Patel from UNC at Chapel Hill

Faculty/Staff of Computer Science Dept., WPI

Huahui Wu, Mingze Li, Feng Li, and everyone from PEDS and CC groups

Attendees today

My Family

Modeling and Evaluating Feedback-Based Error Control for Video Transfer

Documents

Transcript of Modeling and Evaluating Feedback-Based Error Control for Video Transfer