15-744 Computer Networking Multicast (some slides borrowed from Srini Seshan)
1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas...
Transcript of 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas...
![Page 1: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/1.jpg)
1
Developing a Predictive Model for Internet Video Quality-of-Experience
Athula Balachandran, Vyas Sekar,Aditya Akella, Srinivasan Seshan,
Ion Stoica, Hui Zhang
![Page 2: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/2.jpg)
2
QoE $$$
CDN
UsersContentProviders
$$$Better QualityVideo
HigherEngagement
The QoE model
Diagram courtesy: Prof. Ramesh Sitaraman, IMC 2012
![Page 3: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/3.jpg)
3
Adapting video bitrates quicker
Picking the best server
Why do we need a QoE model?
Comparing CDNs
![Page 4: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/4.jpg)
4
Traditional Video Quality Metrics
Objective Scores (e.g., Peak Signal to Noise Ratio)
Subjective Scores(e.g., Mean Opinion Score)
Does not capture new effects (e.g., buffering, switching
bitrates)
User studies not representative of “in-the-wild” experience
![Page 5: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/5.jpg)
5
Internet Video is a new ball game
Objective Scores (e.g., Peak Signal to Noise Ratio)
Subjective Scores(e.g., Mean Opinion
Score)
Engagement(e.g., fraction of video viewed)
Quality metrics
![Page 6: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/6.jpg)
6
Commonly used Quality MetricsJoin Time
Average Bitrate
Buffering ratio
Rate of buffering
Rate of switching
![Page 7: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/7.jpg)
7
Which metric should we use?
Objective Scores (e.g., Peak Signal to Noise Ratio)
Subjective Scores(e.g., Mean Opinion
Score)
Engagement(e.g., fraction of video viewed)
Quality metrics Buffering Ratio, Average bitrate?
Today:QualitativeSingle-metric
![Page 8: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/8.jpg)
8
Unified and Quantitative QoE Model
Objective Scores (e.g., Peak Signal to Noise Ratio)
Subjective Scores(e.g., Mean Opinion
Score)
Engagement(e.g., fraction of video viewed)
Quality metrics Buffering Ratio, Average bitrate?
ƒ (Buffering Ratio, Average bitrate,…)
![Page 9: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/9.jpg)
9
Outline
• What makes this hard?
• Our approach
• Conclusion
![Page 10: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/10.jpg)
Complex Engagement-to-metric RelationshipsEn
gage
men
t
Quality Metric
10
Non-monotonic
E
ngag
emen
t
Average bitrate
En
gage
men
t
Rate of switching
Threshold
[Dobrian et al. Sigcomm 2011]
Ideal Scenario
![Page 11: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/11.jpg)
11
Complex Metric Interdependencies
Join
Tim
e
Average bitrate
Average bitrate
Rate
of b
uffer
ing
Join Time Bitrate
Buffering ratio
Rate of buffering
Rate of switching
![Page 12: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/12.jpg)
Confounding Factors
12
Confounding Factors can affect:1) Engagement
Live and Video on Demand (VOD) sessions have different viewing patterns.
CD
F (
% o
f use
rs)
Engagement
Type of VideoLive
VOD
![Page 13: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/13.jpg)
Confounding Factors
13
Confounding Factors can affect:1) Engagement2) Quality Metrics
Type of VideoLive
VOD
CD
F (
% o
f use
rs)
Join Time
Live and Video on Demand (VOD) sessions had different join time distribution.
![Page 14: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/14.jpg)
Confounding Factors
14
Confounding Factors can affect:1) Engagement2) Quality Metrics3) Quality Metric
Engagement
Type of Video
ConnectivityDSL/Cable
Wireless (3G/4G)
Eng
agem
ent
Rate of buffering
Users onwireless connectivity weremore tolerant to rate of buffering.
![Page 15: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/15.jpg)
Confounding Factors
15
Type of Video
Connectivity
Device
Location
Popularity
Time of day
Day of week
Need systematic approach to identify and incorporate confounding factors
![Page 16: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/16.jpg)
16
Summary of Challenges
1. Capture complex engagement-to-metric relationships and metric-to-metric dependencies.
2. Identify confounding factors3. Incorporate confounding factors
![Page 17: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/17.jpg)
17
Outline
• What makes this hard?
• Our approach
• Conclusion
![Page 18: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/18.jpg)
18
Challenge 1: Capture complex relationships
![Page 19: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/19.jpg)
19
Cast as a Learning Problem
MACHINE LEARNING
Engagement Quality Metrics
QoE Model
Decision Trees performed the best.Accuracy of 40% for predicting within a 10% bucket.
![Page 20: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/20.jpg)
20
Challenge 2: Identify the confounding factors
![Page 21: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/21.jpg)
21
Test Potential Factors
Engagement
Confounding Factors
Quality Metrics
![Page 22: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/22.jpg)
22
Test Potential Factors
Test 1: Relative Information Gain
Engagement
Confounding Factors
Quality Metrics
![Page 23: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/23.jpg)
23
Test Potential Factors
Test 1: Relative Information GainTest 2: Decision Tree StructureTest 3: Tolerance Level
Engagement
Confounding Factors
Quality Metrics
![Page 24: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/24.jpg)
24
Identifying Key Confounding Factors
Factor Relative Information Gain
Decision TreeStructure
Tolerance Level
Type of video ✓ ✓ ✓Popularity ✗ ✗ ✗Location ✗ ✗ ✗Device ✗ ✓ ✓Connectivity ✗ ✗ ✓Time of day ✗ ✗ ✓Day of week ✗ ✗ ✗
![Page 25: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/25.jpg)
25
Identifying Key Confounding Factors
Factor Relative Information Gain
Decision TreeStructure
Tolerance Level
Type of video ✓ ✓ ✓Popularity ✗ ✗ ✗Location ✗ ✗ ✗Device ✗ ✓ ✓Connectivity ✗ ✗ ✓Time of day ✗ ✗ ✓Day of week ✗ ✗ ✗
![Page 26: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/26.jpg)
26
Challenge 3: Incorporate the confounding factors
![Page 27: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/27.jpg)
Refine the Model
27
MACHINE LEARNING
EngagementQuality Metrics
QoE Model
Confounding Factors
Adding as a feature Splitting the dataConfounding Factors 1e.g., Live, Mobile
ML
EngmntQualityMetrics
Model 2
ML
EngmntQualityMetrics
Model 1
ML
EngmntQualityMetrics
Model 3
Confounding Factors 2e.g., VOD, Mobile
Confounding Factors 3e.g., VOD, TV
QoE Model
![Page 28: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/28.jpg)
28
Comparing Candidate Solutions
Final Model: Collection of decision treesFinal Accuracy- 70% (c.f. 40%) for 10% buckets
![Page 29: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/29.jpg)
29
Summary of Our Approach
1. Capture complex engagement-to-metric relationships and metric-to-metric dependencies
Use Machine Learning
2. Identify confounding factors Tests 3. Incorporate confounding factors
Split
![Page 30: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/30.jpg)
30
Evaluation: Benefit of the QoE Model
Preliminary results show that using QoE model to select bitrate leads to 20% improvement in engagement
![Page 31: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/31.jpg)
31
Conclusions• Internet Video needs a unified and quantitative QoE model
• What makes this hard?– Complex relationships– Confounding factors (e.g., type of video, device)
• Developing a model– ML + refinements => Collection of decision trees
• Preliminary evaluation shows that using the QoE model can lead to 20% improvement in engagement
• What’s missing?– Coverage over confounding factors– Evolution of the metric with time
![Page 32: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/32.jpg)
32
EXTRA SLIDES
![Page 33: 1 Developing a Predictive Model for Internet Video Quality-of-Experience Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica,](https://reader036.fdocuments.in/reader036/viewer/2022070307/551b2052550346f70d8b66ea/html5/thumbnails/33.jpg)
34
Choice of ML Algorithm Matters
ML algorithm must be expressive enough to handle the complex relationships and interdependencies
• Classify engagement into uniform classes• Accuracy = # of accurate predictions/ # of cases