It’s Not the Cost, It’s the Quality!

Ion Stoica

Conviva Networks and UC Berkeley

1

A Brief History

!   Fall, 2006: Started Conviva with Hui Zhang (CMU)

!   Initial goal: use p2p technologies to reduce distribution costs and improve the scale

!   Slowly, realized our customers (content premium producers & aggregators) value more quality than cost

!   Today: maximize distribution quality, distribution management, and provide real-time analytics

2

Where is the Data Coming From?

!   Content Providers and Aggregators

!   CDNs

3

4

Trends

Trends: CDN Pricing

!   CDN pricing has decreased x1.5-2 every year over the last 5 year

5

0

5

10

15

20

25

30

35

40

45

2006 2007 2008 2009 2010

cent

s/G

B

Trends: Streaming Rate for Premium Content !   Average streaming rate has increased 20-40%

every year

6

0

200

400

600

800

1000

1200

1400

1600

2006 2007 2008 2009 2010

Kbp

s

Trends: Per-hour Streaming Cost !   Per-hour streaming cost has decreased 15-35%

every year

7

0

1

2

3

4

5

6

7

8

9

10

2006 2007 2008 2009 2010

cent

s/ho

ur

HTTP Chunking

!   Trend accelerated by switching from proprietary streaming technologies (e.g., Adobe’s FMS) to HTTP Chunking:   Move Networks (2005)

  Apple (2008)

  Microsoft (2008/2009)

  Adobe (2010, 2nd half)

8

How Does HTTP Chunking Work?

9

Viewer

CDN1

Viewer Viewer

CDN2

ISP A ISP B

… origin

http cache

HTTP Chunking Advantages

!   Chunks: immutable, relative large objects (hundreds of KB)   Great for caching

!   Leverage existing HTTP infrastructure   CDNs   ISP deployed caches   Enterprise http proxies

!   Low cost and high scale

10

What Does this Mean?

!   Ad supported premium content   CPM (cost per thousand of ad impressions) for

premium content has reached: $20-$40   One ad covers one hour of streaming!

!   Paid content   $0.99 episode, distribution cost < 3%

!   Subscription based premium content   Distribution, usually a few percents of total cost   It costs $1.6 per month to stream content to an user

watching 2 hours per day

!   Production & rights costs dominate 11

12

Quality Matters

Quality Matters

!   Better quality   Increase viewing time more ad opportunities   Increase retention rate   Protect brand

!   Quality   Join time   Buffering ratio   Rendering quality   Streaming rate

13

Analysis

!   Load:   Four channels of a premier video-on demand (VoD)

content producer   Four days   Number of sessions (views): 1,176,049

  A large live event: ~250,000 concurrent viewers

!   Metrics   Content length distribution   Viewer Hour Loss (VHL): number of viewer hours lost

due to quality issues

VoD Object Length Distribution

15

0

50

100

150

200

250

300

10

40

70

100

130

160

190

220

250

280

310

340

370

400

430

460

490

520

550

580

610

640

680

750

880

1240

13

30

2540

25

70

2600

26

40

4040

54

10

5550

Object Length

Short clips: [2min, 3min] Medium clips:

[9min, 11min] Full Episodes: [42min, 45min]

Num

ber o

f Obj

ects

Quality Metrics

!   Buffering Quality (BQ): PlayingTime/(PlayingTime + BufferingTime)

  Rendering Quality (RQ): RenderingRate/EncoderRate

  Good session   BQ > 95%   RQ > 60%

Analysis Underestimates Quality Impact

!   For most analysis use BQ only   RQ only a small part of quality issues due to low

bit rate (500-700Kbps)

!   Ignore connection failures

Short Clip (2-3min) Analysis

18

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

98<=Q<=100 95<=Q<98 90<=Q<95 75<=Q<90 50<=Q<75 0<=Q<50

Aver

age

Play

ing

Tim

e (M

inut

es)

Quality

Short Clip (2-3min) Analysis

19

0

50000

100000

150000

200000

250000

300000

98<=Q<=100 95<=Q<98 90<=Q<95 75<=Q<90 50<=Q<75 0<=Q<50

Num

ber

of S

essi

ons

Quality

Viewer Hour Gain

!   D : Average duration of sessions with high quality (0.98 <= quality < 1)

!   Dq : Average duration of sessions with quality = q

!   Nq : Number of sessions with quality = q

!   Viewer hour gain for sessions with quality q Nq x (D – Dq)

!   Total viewer hour gain ∑q Nq x (D – Dq)

20

Viewer hour loss for 1-2 minute clips:

Medium Clip (9-11min) Analysis

21

0

1

2

3

4

5

6

7

8

98<=Q<=100 95<=Q<98 90<=Q<95 75<=Q<90 50<=Q<75 0<=Q<50

Aver

age

Play

ing

Tim

e (M

inut

es)

Quality

Viewer hour loss for 9-11min clips:

Full Episodes (42-45min) Analysis

22

0

5

10

15

20

25

98<=Q<=100 95<=Q<98 90<=Q<95 75<=Q<90 50<=Q<75 0<=Q<50

Aver

age

Play

ing

Tim

e (M

inut

es)

Quality

•  Viewer hour loss for episodes: •  Viewer hour loss for all content:

Large Scale Live Event

23

0

5

10

15

20

25

30

35

98<=Q<=100 95<=Q<98 90<=Q<95 75<=Q<90 50<=Q<75 0<=Q<50

Aver

age

Play

ing

Tim

e (M

inut

es)

Quality Viewer hour loss:

Large Scale Live Event: Engagement Funnel

24

1 0.99

0.8624

0.7364 0.6602

0.5509 0.4357

0.3639

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

All 0-0 0-1 1-3 3-5 5-10 10-20 20-30

Frac

tion

of

Sesi

ons

Session Duration (Mins)

0<=Q<=0.5

0.5<=Q<=0.75

0.75<=Q<=0.9

0.9 <= Q <= 0.95

0.95<=Q<=0.98

0.98<=Q<=1.0

Quality = 1.0

Continuing

Half people leave due to quality issues

Another Case Study: Live Event

25

Total sessions 151,980

Unique viewers 73,942

Sessions per viewer 1.9

Total viewer hours 58,436

0

5000

10000

15000

20000

7:00

PM

7:

15 P

M

7:30

PM

7:

45 P

M

8:00

PM

8:

15 P

M

8:30

PM

8:

45 P

M

9:00

PM

9:

15 P

M

9:30

PM

9:

45 P

M

10:0

0 PM

10

:15

PM

10:3

0 PM

10

:45

PM

11:0

0 PM

11

:15

PM

11:3

0 PM

11

:45

PM

12:0

0 AM

12

:15

AM

12:3

0 AM

12

:45

AM

1:00

AM

Peak Concurrent Views Quality Engagement

Total views 151,980 25 minutes

Failed views 13,815 (9%) 0 minutes

Quality impacted views

21,584 (14%) 16 minutes

Good views 116,581 (77%) 27 minutes

Unique viewers 75,328 48 minutes

Failed viewers 1,386 (2%) 0 minutes

Quality impacted viewers

14,309 (19%) 30 minutes

Good viewers 59,633 (79%) 51 minutes

Total viewer hours 58,436 hours

Lost viewer hours 5,134 hours (9%)

Viewer with poor quality watch 41% less minutes!

Does High Bit Rate Video Help?

!   Comparing Engagement of low and high bitrates   Viewers watch longer on average on 1500Kbps

0

5

10

15

20

25

30

35

min

utes

Average Play Time (minutes)

500 kbps

1500 kbps

10-15% increase

Summary

!   Quality impact:   BQ can impact viewer engagement by up to 40%   Higher bit-rates can increase viewer engagement by

up to 15%

!   Engagement loss due to quality issues: between 4 and 30%   Even a 4% improvement, may offset distribution

costs   Ignore other quality issues, like connectivity and

media failures

27

28

Root Cause Analysis

Viewers vs. Buffering Quality

29

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

0

50000

100000

150000

200000

250000

0:00

:00

0:15

:00

0:30

:00

0:45

:00

1:00

:00

1:15

:00

1:30

:00

1:45

:00

2:00

:00

2:15

:00

2:30

:00

2:45

:00

3:00

:00

3:15

:00

3:30

:00

3:45

:00

4:00

:00

4:15

:00

4:30

:00

4:45

:00

5:00

:00

5:15

:00

5:30

:00

5:45

:00

6:00

:00

6:15

:00

Viewers Average Buffering Quality %

Light Period Heavy Period

What causes quality issues?

Root Cause Analysis

!   Root cause a quality issue to:   Viewer machine (CPU)   Last mile + ISP (Autonomous System Number)   CDN

!   Note:   Cannot differentiate between edge and core ISPs   Use only passive measurements, no IP traceroute

Viewer

Last Mile +

ISP

CDN Peering ISP

Metrics and Definitions

!   Quality metrics   Buffering quality (BQ)

  playing time/(playing time + buffering time)   Rendering quality (RQ)

  rendering frame rate/encoded frame rate

!   Session classification:   Good: (BQ >= 95%) AND (RQ >= 60%)   Low BQ: (BQ < 95%)   Low RQ: (BQ >= 95%) AND (RQ < 60%)

Methodology: Root Causing Viewer Machine

!   CPU likely to be the issue when:   Rendering quality low   Buffering quality high

!   Conclude CPU is the issue when session’s   RQ < 60%   BQ > 95%

Good Low BQ Low RQ

Quality Issues: Light Period

74%

21.5%

4% (CPU issue) Network/CDN issues

Good Low BQ Low RQ

Quality Issues: High Period

62%

6.2% (CPU issue)

31.5%

Network/CDN issues

Explaining Buffering Issues

!   Assume buffering quality issues are either due to:   CDN, or   ISP

!   Recall: a session has buffering quality issues if   BQ < 95%

Methodology: Root Causing CDN (1/2)

!   Viewers connected to same ASN but using two CDNs

!   Intuition: if quality experienced by CDN 1 viewers is significantly lower than of CDN 2 viewers for same ASN, CDN 1 has quality issues

Last Mile +

ISP

CDN 1 Peering ISP

CDN 2 Peering

ISP

Methodology: Root Causing CDN (2/2)

!   Select all ASNs who have more than 50 sessions for each CDN

•  If difference between quality of viewers in CDN1 and CDN2 for same ASN is > 10%   Lower quality CDN is root cause at current time

Methodology: Root Causing ASN/ISP

!   Two CDNs:   Conclude ASN A has quality issues if ASN A’s viewers

connected to either CDN1 or CDN2 experience “bad quality”

  Average quality of viewers connected to other ASNs higher

!   One CDN   ASN A’s viewers connected to CDN have much lower

quality than the average quality of viewers connected to CDN

Buffering Quality: Light Period

Unqualified CDN ISP Uknown

46% 33%

16%

5%

Buffering Quality: Heavy Period

40

Unqualified CDN ISP Uknown

36.7%

39.7%

15.4%

8.2%

Some Findings

!   Most of ASNs who had quality issues were enterprise ASNs   Expected given that the large scale event was during

the workday   One ASN had 44% buffering quality!

!   No CDN was uniformly bad   (see next)

CDN Comparison

!   Quantify quality difference between CDNs

!   Methodology: 1.  Select all ASNs which have more than 50 sessions

on both CDNs 2.  Compute average quality for CDN1 and CDN2

viewers per ASN 3.  Order ASNs by difference in quality between CDN1

and CDN2

Internet delivery is more variable than realized…

!   Content Delivery Networks all have problems sometime

!   Even in the same viewer session the best quality changed many times during the event

CDN 1 was best

CDNs were even

CDN 2 was best

Summary 24

-38%

of

Tota

l Ses

sion

s ha

ve Q

ualit

y Is

sues

Quality Issues Classification

Solution

CDN (7-12% of total sessions)

Resource switching

End-Host CPU (4-6% of total sessions)

Bit-rate switching

ISP (2-3% of total sessions)

Localize traffic, bit-rate switching

Unqualified (9-11% of total sessions)

Mitigated by above

Unknown (1-4% of total sessions)

N/A

Conclusions

!   At least for premium content   Reducing cost is important, but…   … improving quality is even more important

!   P2P can play an important role   Localize traffic   Highly robust to source failures

!   Great opportunity   Adobe has announced full p2p support for Flash

Player 10.1   No need for client download!

45