Quality Prediction of Mobile Video Service Based on Radio Access Network Log Data

Seungbo Yoo Advanced Technology Group

Samsung Electronics Suwon, Korea

sb1.yoo@samsung.com

Dongmin Kim Advanced Technology Group

Samsung Electronics Suwon, Korea

Imdm.kim@samsung.com

Dongsoo S. Kim School of Eng. & Tech.

Indiana University-Purdue University Indianapolis

Indianapolis, United States dskim@iupui.edu

Hyunseung Choo School of Info & Comm. Eng. Sungkyunkwan University

Suwon, Korea choo@skku.edu

Abstract— Quality of Experience (QoE) of mobile video is being raised as the most important parameter for network operators due to the rapid increase in mobile video traffic. But in the mobile network, the network-level Quality of Service (QoS) is provided and this network-level QoS does not exactly represent the user-level QoE. In this paper, we propose a system with two modules for forecasting mobile video quality based on network Radio Frequency (RF)metrics that are related to QoS. We develop a Network Quality Prediction (NQP) module which can analyze patterns of RF metrics and predict the next RF metrics patterns. In addition, we design a Video Quality Scoring (VQS) module which can predict Video Mean Opinion Score (VMOS) by mapping predicted RF metrics to VMOS. The proposed system, NQP and VQS, shows a meaningful prediction performance of VMOS.

Keywords- Mobile video service; Radio Access Network; QoE; QoS; Quality prediction; Quality analysis

I. INTRODUCTION

Fourth generation (4G) mobile communication technologies, such as Long Term Evolution (LTE), provide high throughput, low latency, and wide connectivity. Because of these technologies, mobile users can enjoy not only voice and text services but also various data services such as web-browsing, game and video. In case of mobile video, it accounted for 55% of mobile traffic by the end of 2014 and will grow 66% annually and account for 72% of total mobile data traffic by 2019 [1]. According to increasing mobile video traffic, analyzing Quality of Experience (QoE) of mobile video will be one of the critical issues for mobile operators.

However, The QoE analysis is not easy in the existing mobile network because network-level Quality of Service(QoS) does not directly correlate to the user-level QoE. In other words, QoE monitoring is difficult even when QoE degradation happens in the network. This motivates us to develop a Mobile video Quality Analysis and Prediction System (MQAPS) based on the Radio Frequency (RF) data related network QoS.

In this paper, we propose the MQAPS which is composed of two modules. We develop a Network Quality Prediction (NQP) module which predicts RF metrics patterns based on current and historical patterns of RF metrics. In addition, we design a Video Quality Scoring (VQS) module which can predict future Video Mean Opinion Score (VMOS) by mapping predicted RF metrics to VMOS.

The proposed system shows a meaningful performance of VMOS prediction based on network-level RF metrics. The system performance is analyzed using two modeling schemes such as NQP and VQS. The simulation results from the R programming language show that our proposed system achieves up to 94.1% VMOS forecast rate for the next 2 days. The rest of the dissertation is organized as following: Section 2 reviews previous research on QoE analysis and time series forecast methods. Section 3 introduces the proposed MQAPS including NQP and VQS modules. And the results of evaluation are shown in section 4. Finally, section 5 concludes the dissertation and introduces future works.

II. RELATED WORK

For quality analysis of mobile video service, basically we need the instrumentation, data collection, and modeling process. On the network side, log data can be collected directly by network equipment or by probe equipment installed in the network as instrumentation process. In parallel, log data can also be generated by exclusive applications in the User Equipment (UE). For the analyzing the service quality, the log data should be collected when customer uses the service with various network environments (e.g. bad RF quality, hot spot area).

Log type can be divided into two categories: QoS and QoE metrics. QoS is the overall performance of the network, and QoE is a measure of a customer's experiences with a video service. In the existing approach, LASSO (Least Absolute Shrinkage and Selection Operator) regression as machine learning method is used for detecting QoE based on the QoS. And regression accuracy rate is up to 80% in terms of binary decision (e.g. good or bad) [7]. We used the Support Vector Regression (SVR) for mapping from QoS to QoE and obtained results including three decision levels (i.e. good, normal, and bad).

QoE and QoS are connected through a non-linear (i.e. exponential) relationship [8]. We employ a non-linear SVR as supervised learning technique to map RF metrics to VMOS.

Time series patterns can show four kinds of features; trend, seasonality, cycle, and irregular patterns. Analysis and prediction of time series patterns is already used in many areas such as stock market [9], weather forecast [10], and so on. RF patterns also have these characteristics in live network. We used exponential smoothing method including error, trend, and seasonal parameters to forecast RF pattern [2].

2015 11th International Conference on Signal-Image Technology & Internet-Based Systems

978-1-4673-9721-6/15 $31.00 © 2015 IEEE

DOI 10.1109/SITIS.2015.112

599

2015 11th International Conference on Signal-Image Technology & Internet-Based Systems

978-1-4673-9721-6/15 $31.00 © 2015 IEEE

DOI 10.1109/SITIS.2015.112

599

III. PROPOSED SYSTEM

A. System Architecture Overview The proposed MQAPS consists of two modules;

Network Quality Prediction (NQP) and Video Quality Scoring (VQS). Both NQP and VQS in MQMPS have the common procedures that are data collection via network element, data pre-processing, machine learning and output modeling as shown in Figure 1.

Figure 1. MQAPS architecture overview.

When MQAPS receives the log data from the LTE radio base station called evolved NodeB (eNB) and UE application, MQAPS removes the invalid log data and extracts necessary metrics during the data collection procedure. The refined metrics are grouped or calculated as input data for machine learning in the data pre-processing procedure. And NQP performs the exponential smoothing method as a time-series forecasting and VQS conducts the Support Vector Regression as a cross-sectional forecasting. Finally NQP and VQS construct RF metrics forecasting and RF-VMOS modeling as output results. Before going into the discussion of the proposed system, this paper has assumptions that other nodes (i.e. video app server, backhaul network impairment and UE HW performance) have no problems

B. Network Quality Prediction The NQP is composed of sub-modules such as Data

Collection, Time-bin Level Grouping and Exponential Smoothing. In addition, NQP constructs modeling of RF metrics forecast and finally outputs a predicted RF metrics as shown in Figure 2.

Figure 2. Network Quality Prediction (NQP) module

For getting RF metrics, NQP uses the Call Summary Log (CSL). CSL is an event-based call log which is generated by the eNB. When the call ends, the eNB records a CSL with call-related information such as call

time, data volume and RF info. When NQP receives the CSL, NQP removes the abnormal log data like error, duplication or null data. Then NQP extracts the Reference Signal Received Power (RSRP), Reference Signal Received Quality (RSRQ), and call time factors as RF metrics from the CSL. RSRP is the average received power of a single reference signal resource element and their reporting range is from -60 to -140 dBm. RSRQ indicates the quality of the received reference signal which range from -4 to -20 dB [6] [11].

After extracting RSRP, RSRQ, and call time, first, RSRP-RSRQ log is grouped by specific time range. Then the time grouping of RSRP-RSRQ log is divided by specific RSRP and RSRQ range on two dimensional bins once more. Finally, the number of RSRP-RSRQ count per time and bin is logged. We constructed and evaluated various levels of groupings with time (2, 4, and 6 hours) and bin levels (4x4 and 8x8 RSRP-RSRQ grids). An example of 6 hours and 8x8 bins is shown in Figure 3.

Figure 3. Example of 6 hours and 8x8 bin level grouping

In addition, example of grouping the 6 hours and 8x8 bins can be calculated by following equation.

One_bint,i,j = {(t,i,j)| Timet ∩ RSRPi ∩ RSRQj}, (1)

where

Timet = Logging_time[6t, 6(t+1)), (0≤t)

RSRPi=RSRP[-140+10i, -140+10(i+1)), (0≤i≤7)

RSRQj=RSRQ[-20+2i, -20+2(i+1)), (0≤i≤7)

We use the exponential smoothing method including the error, trend and seasonal smoothing components (i.e. l[t], b[t], and s[t] as shown in equation (2)) called additive Holt-Winters seasonal method with additive errors [2].Because RF metrics have trend and seasonal (e.g. daily) patterns. The exponential smoothing method has three smoothing parameter; α, β, and γ. Each of parametersspecifies how to smooth the error, trend, and seasonal components. The equation of exponential smoothing is shown in equation (2) [19].

��[t+h] = l[t] + hb[t] + s[t - p + 1 + (h - 1) mod p], (2)

where

h = forecast time

p= period of seasonality

l[t] = α (Y[t] - s[t-p]) + (1-α) (l[t-1] + b[t-1])

b[t] = β (l[t] - l[t-1]) + (1-β) b[t-1]

s[t] = γ (Y[t] - l[t]) + (1-γ) s[t-p]

600600

The time series of each 4x4 or 8x8 bin is used in exponential smoothing as the input data for time series forecasting. When each time group (e.g. 2, 4 or 6 hours) gets into the NQP, The exponential smoothing parameters (i.e. α, β, and γ) are optimized by minimizing the squared prediction error. And finally, NQP predicts all RF metrics after combining each of the predicted bins.

Figure 4. One bin level prediction and combination.

C. Video Quality Scoring The VQS is composed of Data Collection, QoE-VMOS

Calculation and Support Vector Regression sub modules.The VQS initially needs RF metrics and QoE metrics as input data via UE application to construct RF-VMOS mapping using SVR. After that the VQS only need to input the RF metrics which is from eNB and then estimate VMOS by RF-VMOS mapping. The application logs are collected in various RF environments to widely map between RF and VMOS. The detailed architecture of VQS is shown in Figure 5

Figure 5. Video Quality Scoring (VQS) module

For collecting QoS and QoE metrics in the UE, we developed a mobile video application which is a customized VLC media player as open-source multimedia solutions. This video application can record the RF metrics (i.e. RSRP and RSRQ) and QoE metrics; Initial buffering time, re-buffering count, and re-buffering duration per video play. If RSRP and RSRQ as radio power and quality respectively are bad, it will cause initial playout delay and re-buffering because of insufficient video throughput and finally result in bad VMOS.

We calculate the VMOS through QoE metrics and their weight factors. To estimate VMOS, we refer to the VMOS

formula which is defined by Measuring the QoE of HTTP video streaming [3], given by

VMOS=4.23–0.0672Tinit–0.742frebuf–0.106 Trebuf, (3)

Where Tinit is the initial buffering time, frebuf is the re-buffering count, and Trebuf is the re-buffering duration. In our scope, VMOS score is defined by three levels like Good, Normal, and Bad according to the following equation;

���� = ⎩⎪⎨⎪⎧

���, �� ���� ≥ 3.5_������, �� 1.5 ≤ ���� < 3.5_

���, �� ���� < 1.5 (4)

After obtaining RF metrics and VMOS for each video played, we applied the support vector regression; RF metrics are input data and VMOS are response value. In the SVR, we used the radial basis function kernel to consider non-linear characteristics because there is an exponential relationship between RF metrics and VMOS as shown in Figure 6. Non-linear Support Vector Regression outperforms other simpler linear techniques [4].After training SVR, The VQS finalize the modeling of RF-VMOS mapping and then outputs VMOS

Figure 6. Relationship between RF matrics and VMOS

D. NQP and VQS Combination NQP collects and analyzes the CSL log and outputs the

predicted future RF metrics to VQS. Then VQS converts RSRP-RSRQ metrics to VMOS as shown in Figure 7.NQP need to collect the CSL data periodically from eNB for forecasting RF metric. On the contrary, VQS only needs the predicted RSRP-RSRQ log from NQP and then outputs predicted VMOS based on the modeling of RFmetrics-VMOS mapping.

Figure 7. NQP and VQS combination

601601

IV. PERPORMANCE EVALUTAION

A. NQP Evaluation We collected the CSL data in live network for 16 days.

Among the collected CSL logs, we selected five eNBs (named eNB A, B,…, E) which have more than 1000 CSLs per day. And we used 14 days’ worth of CSL data for training and 2 days of CSL data for evaluation.

We evaluated the forecast RSRP-RSRQ count value compared to the actual RSRP-RSRQ count value. For the evaluation, we refer to Weighted Absolute Percent Error (WAPE) [12] for calculating RF forecast accuracy rate like following equation.

�� �������� ���!���" ����(%) = ( 1 − ∑ (|$&'|)*+-,/+02∑ $*+-,/+02 ) × 100 (5)

Where, A is actual RSRP-RSRQ count value and F is forecast RSRP-RSRQ count value per bin and time. The evaluation results of five eNBs are that we achieved up to 90.0% forecast accuracy rate between actual count value and forecast count value in terms of 6 hour and 4x4 bin grouping. The detailed result table of forecast accuracy rate per bin and time resolution is shown in Table I. The results of forecast accuracy indicate that the larger time (e.g 6 hour) and bin resolution (e.g. 4x4 bin), the more forecast accuracy because strong periodical RF pattern is shown in large time and bin resolution.

TABLE I. RESULTS OF RF FORECAST ACCURACY RATE PER BIN AND TIME RESOLUTION

eNB A case; 6hour and 8x8 bin, for example, detailed actual count and forecast results of all bins during every 6 hours within evaluation days (i.e. 2 days) are shown in Figure 9 and 10.

The actual count graphs (i.e. Figure 9 (a) ~ (h)) show that many of count is in specific bins; RSRP[-120, -100], RSRQ[-16,-6]. It means that many calls are practically generated at these bins in live network. And the forecast graphs (i.e. Figure 10 (a) ~ (h)) show forecast results predicted by RF forecast modeling in NQP. The forecast count also indicates clumped distribution same as actual count and forecast accuracy range is from 66.8 to 84.0 %. The Figure 10 (a), 0~6h time group shows the lowest accuracy rate (i.e. 66.8%) because sometimes actual count in the early hours of the morning reveals more irregular RF pattern than other time groups caused by less actual count. In the eNB A case; 6 hours and 8x8 bins at evaluation days, total number of actual count is 4267 and forecast count is 3401. Finally, RF forecast accuracy rate is 79.7% after

calculating each time group and bin count respectively by using equation (5).

B. VQS Evaluation We collected and analyzed the 970 video play logs

with 6 UEs in test-bed network. The detailed video properties for test are shown in Table II. Each video play log including RSRP, RSRQ, and QoE information (i.e. Initial playout delay, re-buffering count, and duration) was generated by UE application at the end of the playing time.For calculating performance of VQS, we compared between the estimated VMOS from VQS and actual VMOS calculated by using equations (3) and (4).

TABLE II. VIDEO PROPERTIES

Figure 8. Actual and estimated VMOS

We defined the VMOS precision rate for evaluating VQS performance as shown below equation.

���� 6�������7 ����(%) = 89::;>? @ABC∗EFF89::;>? @ABCGHI>9::;>? @ABC (6)

We obtained the evaluation result that is 90.0% of VMOS precision rate among actual VMOS and estimated VMOS. The more detailed results of precision rate are shown in Table III. and equation (7). The correct count of Video MOS is 873 and in-correct count is 97 in total video plays.

TABLE III. VQS EVALUATION RESULTS

VMOS precision rate (%)= 873/(873+97)×100=90.0 % (7)

602602

Figure 9. Detailed actual RSRP-RSRQ count per time group

Figure 10. Detailed forecast RSRP-RSRQ count per time group

603603

C. Evaluation of NQP and VQS Combination According to interworking between NQP and VQS

modules, we achieved analytically up to 94.1% as VMOS forecast accuracy rate with 90% precision rate. The formula of VMOS forecast accuracy is given by

���� �������� ���!���" ���� (%) = ( 1 − ∑ (|$&'|)JKLM_N2O2N∑ $JKLM_N2O2N ) × 100 (8)

Where A is estimated VMOS level (e.g. good, normal, and bad) value via actual RF metrics and F is forecast VMOS value by mapping predicted RF metrics respectively.

Final results of VMOS forecast accuracy for all eNBs are shown in Table IV. The performance results show that the larger time resolution (e.g. 6 hour), the more high performance because the RF pattern is more clear at large time resolution whereas there is no big difference results between 4x4 and 8x8 bin resolution. All results of VMOS forecast accuracy are higher than each of RF forecast accuracy (i.e. Table I.) because 4x4 or 8x8 RSRP-RSRQ resolution is shrunk to 3 level VMOS such as good, normal and bad.

In the eNB A case (6 hours and 8x8 bin grouping), the detailed result between actual and forecast VMOS is shown on Figure 11. The results indicate VMOS distribution within the next 2 days (evaluation days) with

94.1% VMOS forecast accuracy and 90% VMOS precision rate.

TABLE IV. VMOS FORECAST ACCURACY RATE

In Figure 11 (a) and (c), estimated VMOS shows several features: (i) the results show the daily based seasonal pattern that the most count is in +12~18 and +36~42 hour (i.e. 12~18 PM) while the smallest count is in 0~6 and 24~30 hour (i.e. 0~6 AM). (ii) all trends of VMOS levels are almost flat in the whole evaluation day. (iii) proportion of good level VMOS is low compared to other VMOS levels at all evaluation. (iv) at 18~24 time, all VMOS level are concentrated in 30~35% range. The forecast VMOS also show all these features (i.e. (i) ~ (iv)) as shown in Figure 11 (b) and (d).

Figure 11. Estimated and forecast VMOS distribution (eNB A case)

604604

V. CONCLUSION

The growing popularity of mobile video service has made it necessary to look into the user QoE analysis in cellular networks. In this paper, we demonstrate the analysis and prediction system for mobile video quality called MQAPS. By using 2-step machine learning (Exponential smoothing and Support Vector Regression) method, we can forecast the RF quality and analyze the video quality in the future. Performance results point out that proposed system is capable of forecasting video quality with up to 94.1% accuracy and 90% precision rate. In the commercial network, we would need expensive probe equipment at each node in order to analyze mobile video quality. However, the differentiator of our proposed system is that we can analyze and forecast the mobile video quality without other external equipment. Our future work lies in investigating into optimized quality analysis system with various log data as well as call log and analysis and prediction system for other services (e.g. web-browsing, VoLTE) using their proper QoS and QoE metrics.

REFERENCES

[1] Cisco Visual Networking Index, "Global Mobile Data Traffic Forecast Update, 2014–2019 White Paper," Cisco,2015.

[2] R. Hyndman, A. B. Koehler, J. K. Ord, and R. D. Snyder, Forecasting with exponential smoothing: the state space approach: Springer Science & Business Media, 2008.

[3] R. K. Mok, E. W. Chan, and R. K. Chang, "Measuring the quality of experience of HTTP video streaming," in Integrated Network Management (IM), 2011 IFIP/IEEE International Symposium on, 2011, pp. 485-492.

[4] D. Joumblatt, J. Chandrashekar, B. Kveton, N. Taft, and R. Teixeira, "Predicting user dissatisfaction with internet application performance at end-hosts," in INFOCOM, 2013 Proceedings IEEE, 2013, pp. 235-239.

[5] 3GPP TR 26.944 V9.0.0, “End-to-end Multimedia Services Performance Metrics,” Dec.2009.

[6] 3GPP TS 36.214 V9.2.0, “Physical layer; Measurements,”Jun. 2010.

[7] V. Aggarwal, E. Halepovic, J. Pang, S. Venkataraman, and H. Yan, "Prometheus: Toward quality-of-experience estimation for mobile apps from passive network measurements," in Proceedings of the 15th Workshop on Mobile Computing Systems and Applications, 2014, p. 18.

[8] M. Fiedler, T. Hossfeld, and P. Tran-Gia, "A generic quantitative relationship between quality of experience and quality of service," Network, IEEE, vol. 24, pp. 36-41, 2010.

[9] R. Lawrence, "Using neural networks to forecast stock market prices," University of Manitoba, 1997.

[10] D. B. Botkin, H. Saxe, M. B. Araujo, R. Betts, R. H. Bradshaw, T. Cedhagen, et al., "Forecasting the effects of global warming on biodiversity," Bioscience, vol. 57, pp. 227-236, 2007.

[11] 3GPP TS 36133 V9.22.0, “Requirements for support of radio resource management,” Jan. 2015.

[12] Calculating demand forecast accuracy, https://en.wikipedia.org/wiki/Calculating_demand_forecast_accuracy

[13] A. J. Smola and B. Schölkopf, "A tutorial on support vector regression," Statistics and computing, vol. 14, pp. 199-222, 2004.

[14] S. R. Gunn, "Support vector machines for classification and regression," ISIS technical report, vol. 14, 1998..

[15] R. Adhikari and R. Agrawal, "An Introductory Study on Time Series Modeling and Forecasting," arXiv preprint arXiv:1302.6613, 2013.

[16] R. Schatz, T. Hoßfeld, and P. Casas, "Passive youtube QoE monitoring for ISPs," in Innovative Mobile and Internet Services in Ubiquitous Computing (IMIS), 2012 Sixth International Conference on, 2012, pp. 358-364.

[17] T. Hastie, R. Tibshirani, J. Friedman, and J. Franklin, "The elements of statistical learning: data mining, inference and prediction," The Mathematical Intelligencer, vol. 27, pp. 83-85, 2005.

[18] G. James, D. Witten, T. Hastie, and R. Tibshirani, An introduction to statistical learning: Springer, 2013

[19] The additive Holt-Winters prediction equation,https://stat.ethz.ch/R-manual/R-devel/library/stats/html/ HoltWinters . html

605605