7/25/2019 Video Quality Metrics Discussion

1/21

IETF RMCAT WG: Video Quality Metrics

Discussion for Evaluation Criteria

V. . o!aya"ulu#v.srinivasa.so!aya"ulu$intel.co!%

&assnaa Moustafa#'assnaa.!oustafa$intel.co!%

7/25/2019 Video Quality Metrics Discussion

2/21

(ro)le! tate!ent

Video Quality Perception is critical for end*userQoE

To+ic 'as )een discussed in WG, )ut noconclusion yet on includin- t'is in t'ereuire!ents or t'e evaluation !etrics for t'econ-estion control al-orit'!s

/et0or12trans+ort related !etrics are currently+resent in t'e evaluation criteria for R!catcon-estion control al-orit'!s, )ut do not fully

ca+ture t'e i!+act on video QoE.Purpose of this presentation:1. Provide background on video quality metrics that canhelp quantify impact of congestion control on the video QoE2. Stimulate a discussion on adoption of appropriate metricsin evaluation criteria

7/25/2019 Video Quality Metrics Discussion

3/21

#&i-' 3evel% u!!ary of Evaluationcenario Discussions in RMCAT WG

Current #evolvin-% direction see!s to )e *Evaluate overall +erfor!ance, i.e., Video2Audio4/et0or1 Con-estion control al-orit'!

Cou+le of 0ays to include video c'aracteristics:

5. 6se a tra7c !odel t'at ca+tures statistics of t'evideo source4rate control4s'a+in-

8. (ossi)ly use unco!+ressed video seuences 0it' alive encoder #4 +rescri)ed settin-s% and con-estion

control al-o. Wit' eit'er a++roac', 0e s'ould de9ne a !eans

of ca+turin- i!+act on t'e overall video QoE.

7/25/2019 Video Quality Metrics Discussion

4/21

);ective Metrics < =i- (icture Vie0

=ac1-round );ective !etrics develo+ed to !i!ic 'u!an

+erce+tion of video, e.-., >loo1 at t'e receivedvideo, and ;ud-e 'o0 -ood it loo1s?

Distortions )et0een source and videoconsidered a >)lac1 )o@? and did not aect

!etric desi-n

Good for un)iased evaluation of encoders, etc. /on*reference 2 Reduced reference a di7cult

+ro)le! in t'is -eneric case

Full*reference vs. non*reference VQM Full reference: co!+are t'e !easured

video 0it' t'e ori-inal unco!+ressedvideo (/R, M*IM, (EVQ

/on*reference: analy"e t'e video0it'out a co!+arison E.-. (58B8, etc.

IT6*T G.5BB

Intel Con9dential

ourceVideo

Vie0edVideo

Distortions

&.8Encode

&.8Decode

I( #6D(%(ac1ets

(ac1et3oss2itter

(1t 3ossConceal!ent

=uerMana-e!ent

7/25/2019 Video Quality Metrics Discussion

5/21

Video Conferencin- QoE < IT6*TG.5BB

IT6*T G.5BB

!!endation IT6*T G.5BB +ro+oses an al-orit'! t'at esti!ates video+'one uality for +erience #QoE%2uality of service #Qo% +lanners.

7/25/2019 Video Quality Metrics Discussion

6/21

IT6*T G.5BB Video QualityMetric

Video uality is calculated as:

0'ere Icodingre+resents )asic video uality as a function of video )itrate and

fra!e rate

DPplVre+resents de-ree of video uality ro)ustness due to +ac1et loss and

PplV is t'e +ac1et loss rate in

T'ese uantities are calculated usin- a set of 9@ed +ara!eters de+endenton codec ty+e, video for!at, 1ey fra!e interval, and video dis+lay si"e G.5BB +rovides +rovisional values for &.8, VGA for!at, 5 second 1ey fra!e

interval and H.8 inc' dis+lay, coded )it rates )et0een BB 1)+s < 8 M)+s, +ac1et lossrates J and fra!e rates fro! J*8J f+s.

(ara!eter values for !odelin- any ot'er set of conditions 0ould need to )e derivedfro! video uality evaluation

Furt'er en'ance!ents to t'e )asic !odel for dierent codecs,dis+lay for!ats, content de+endency, etc. 'ave )een e@+lored, e.-.: Kos1o0ic", 8BBHL os1o0ic", . et al >En'ance!ents to t'e +inion Model

for Video*Tele+'ony A++lications?, Proc. 5thLatin American NetworkingConference, ++. *H.

7/25/2019 Video Quality Metrics Discussion

7/21

Considerations on use of IT6*T

G.5BB

6se evaluation scenario#s% to run si!ulation of tar-etcon-estion avoidance al-orit'! 0it' a -iven testvideo seuence #trace 9le% Collect trace of +ac1et arrival ti!es, +ac1et losses, at

sender and receiver e-!ent data into s'ort intervals of ti!e #e.-. J secondsN%

For eac' se-!ent i: Calculate video )itrate sent, assu!e 9@ed fra!e rate

Calculate avera-e +ac1et loss rate at receiver

Calculate +er*se-!ent video uality Vq, i

Calculate over t'e entire video seuence:

Mean and variance of t'e set of OVq, iP

&i-'er !ean and lo0er variance )etter overall video uality

7/25/2019 Video Quality Metrics Discussion

8/21

Discussion (ro+ose to add video uality evaluation !etrics for consideration )y R!cat WG.

Desira)le features of t'e !etrics s'ould include: Good correlation 0it' su);ective video uality +erce+tion

Co!)ine dierent +ara!eters to +rovide an inte-rated loo1 at video QoE i!+act

Relatively si!+le to calculate )ased on data fro! net0or1 si!ulations

Ideally, )ased on +u)lis'ed standards

u);ective uality evaluation is 'ard to or-ani"e and e@ecute, es+ecially in a contri)utionevaluation +'ase

Co!!on o);ective evaluation !etrics of video uality

Easier to use in +ro+osal evaluations,

Full*reference /on*reference: latter !ay )e !ore suited for RMCAT evaluation

IT6*T Rec.G.5BB )ased /R VQ !etrics desi-ned for video conferencin- a++lications

Additionally, t'is doesnt reuire co!+ressed )itstrea! ins+ection, etc.: al-orit'! in+uts are 'i-'level, e.-. t'rou-'+ut, +ac1et loss rate, etc.

Soliciting feedback from group on dening consideration for video qualitymetrics as part of evaluation criteria for congestion control algorithms! "urrent phase

#or further consideration: "ould $Q be e%ploited by congestion controlalgorithms!

Potential for incorporating video quality information into &'"P (&)*+", reports!-*ater phase

7/25/2019 Video Quality Metrics Discussion

9/21

/nne%

7/25/2019 Video Quality Metrics Discussion

10/21

=ac1-round R!cat WG is dealin- 0it' con-estion control for Internet data considerin- interactive

+oint*to*+oint real*ti!e !ulti!edia services over RT(

Reuire!ents for con-estion control al-orit'!s are de9ned considerin- 3o0 delay

e!i*Relia)le data delivery

Fairness to ot'er So0s

Ada+tation to net0or1 conditions

Metrics for con-estion control are de9ned to )e Delay, t'rou-'+ut, !ini!i"in- trans!ission rates oscillations, reactivity to transient events and

+ac1et losses and discards

Evaluation Criteria for con-estion control al-orit'!s 'ave )een de9ned considerin- Avoidin- Con-estion Colla+se

ta)ility

Media Tra7c tartu+ =e'avior

Diverse Environ!ents

Varyin- (at' C'aracteristics

Reactin- to Transient Events or Interru+tions

Fairness 0it' i!ilar Cross Tra7c

I!+act on Cross Tra7c

E@tensions to RT(2RTC(

7/25/2019 Video Quality Metrics Discussion

11/21

Video Quality Variation 02 /et0or1Conditions Variation

Sorce! "#$ Cheng et al., %&ea'rement and Anal('i' of Sk(pe Video

)ra*c,+ APCS -/.

PSN"lo'' for di0erent network 1andwidthlimitation'

PSN"lo'' for di0erent packet lo''rate

7/25/2019 Video Quality Metrics Discussion

12/21

I!+act of T'rou-'+ut Variation onVideo Conferencin- A++lications

Sorce! L. De Cicco et al., %Sk(pe Video "e'pon'i2ene'' to 3andwidth

Variation,+ AC& N4SSDAV -/.

7/25/2019 Video Quality Metrics Discussion

13/21

Video Quality Evaluation:

Introduction

Quality of E@+erienceT'e overall e@+erience t'e consu!er 'as 0'en

accessin- and usin- +rovided video services

Quantifyin- Video Quality Mean +inion core #M%

u);ectively done: recruit a -rou+ of +eo+le to 0atc' a set ofvideo cli+s and -ive a nu!eric score to eac' cli+

Auto!atically done: desi-n al-orit'!s to esti!ate a M)ased on c'aracteristics of !edia strea!, net0or1, device,

etc. Video Quality Issues

Video creation, video encodin-2transcodin-, videotrans!ission, video dis+lay

7/25/2019 Video Quality Metrics Discussion

14/21

Video Quality Issues

Video )loc1iness #encodin-%

Video )lurriness #ca+turin-2encodin-%

Video losses #trans!ission%

Video ;er1iness #trans!ission2encodin-2dis+lay%

Video free"in-2re)uerin- #trans!ission% A2V sync +ro)le! #trans!ission2encodin-%

=loc1iness vs. )lurriness

7/25/2019 Video Quality Metrics Discussion

15/21

(/R #(ea1 i-nal*to*/oiseRatio%

Most co!!only used !etric to!easure t'e uality of reconstructionof lossy co!+ression codecs

Ty+ically values )et0een BUJB d=,

'i-'er is )etter

7/25/2019 Video Quality Metrics Discussion

16/21

M*IM

3u!inance l6, (7 Contrast c6,(7 tructure s#@,y%

Ty+ical values )et0een B.U5, 'i-'er is )etter

7/25/2019 Video Quality Metrics Discussion

17/21

(EVQ

ut+ut (EVQ M ran-in- fro! 5 #)ad%UJ #e@cellent%Distortion Indicators: Delay, =ri-'tness, Contrast, (/R, er1iness, =lur,=loc1iness, Fra!e s1i+s and free"es, Te!+oral Activity and s+atial co!+le@ity

7/25/2019 Video Quality Metrics Discussion

18/21

IT6*T G58 /o*Reference );ective

tandards

Consented orA++roved docu!ents

s'o0n in black Tar-eted A++lications:

I(TV services non*ada+tive strea!in-non*interactive

(.58B5.@ < !ulti!ediaQoE(.58B8.@ < video QoE

Tar-eted +rotocols:non*&TT( #RT(, T*on6D(, etc.%

=ittrea!

(ac1et&eaders

DecodedVideo

(.58B5.5#l)rX%(.58B5.8#')rX%(.58B8.5 #l)rX%(.58B8.8 !ode 5 #')rX%

J.bitvqm,J.mm-noref, J.noref,

VQEG-JEG Hybrid Project

In+uts

Intel Con9dential 5

(.58B8.8 !ode 8 #')rX%

X l)r* lo0 )it rate ')r* 'i-' )it rate

7/25/2019 Video Quality Metrics Discussion

19/21

(.58B@.@ Video I!+air!entModel

(.58B8 A++roac' Desi-n a !etric for evaluatin- uality in s+eci9c instance i.e., I(TV services

T'is 'igni8cantl(constrains t'e +ro)le!, and s'ould !a1e t'e solution !uc'!ore feasi)le

Yno0n video encoder encoder out+ut availa)le

Yno0n c'annel i!+air!ent +attern availa)le

Four !ain video distortions accounted for Co!+ression Artifacts

Due to lossy encodin-

licin- Artifacts Due to (ac1et 3oss Conceal!ent #(3C% of lost +ac1ets

Free"in- Artifacts Due to (3C re+lacin- erroneous fra!es 0it' last -ood fra!e #>free"in- 0it'

s1i++in-?%

Re)uerin- Artifacts Due to (3C re+eatin- a fra!e until fra!e rece+tion reco!!ences #>free"in-

0it'out s1i++in- 2 s+innin- 0'eel?%

7/25/2019 Video Quality Metrics Discussion

20/21

(.58B8 #e@. (./=AM%

T0o a++lication areas :

(.58B8.5,Zlo0er resolution !ode?: a!e as (58B5.5:

i.e., QCIF*QVGA*&VGA, !ostly for !o)ile TVand trea!in-

(.58B8.8, Z'i-'er resolution !ode?:

3inear )roadcast TV Video on*de!and: till under tudy

Intel Con9dential 8B

7/25/2019 Video Quality Metrics Discussion

21/21

(.58B8 #e@. (./=AM%

(ac1et &eaders and)itstrea! in+ut only

/ot intended forcodec evaluation

/ot intended forstrea!s 0it'

si-ni9cant rateada+tation

Video (earsoncorrelation of B.H5for (.58B8.5 #H8sa!+les%

$alidated 'est #actors

Reco!!endation

(.58B8.5 (.58B8.8

Audio =R /A /A

Video =R B.BJ < M)+s 6nder study

(ac1et loss 6nder study

Re*)uerin-

VideoResolution

&VGA, QVGA,QCIF

6nder study

Video

encodin-

&.82AVC

)aseline

6nder study

FRs and 1eyfra!e rates

Fra!e rate J*B&"G( len-t's 8*5Bs

6nder study

(rotocol 6D(*)ased 6nder study

(rotocols not

testedX

TC(*)ased 6nder study

X #can )e used, )ut !ay not )e relia)le%

Intel Con9dential 85