Quantifying Skype User Satisfaction
description
Transcript of Quantifying Skype User Satisfaction
![Page 1: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/1.jpg)
Quantifying Skype User Satisfaction
Carol K. L. Wong
19 March, 2007
CSC7221
![Page 2: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/2.jpg)
Skype a P2P Internet teleph
ony network > 2 million Skype dow
nloads ~ 85 millions users wo
rldwide
From Wikipedia.org
![Page 3: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/3.jpg)
Skype’s Performance
Q: Is Skype providing a good enough voice phone service to the users?
![Page 4: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/4.jpg)
Comparison of Proposed & Existing Methods
Speech quality measures
Proposed User Satisfaction Index
To quantify Speech quality User satisfaction
Built upon Subjective mean opinion score (MOS)
Call duration
Predictors Distortion of signals QoS factors: • the bit rate, • network latency, • network delay var
iations, • packet loss
![Page 5: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/5.jpg)
Methodology
Collect Skype VoIP sessions and their network parameters
Analysis of Call Duration and propose an objective index, the User Satisfaction Index (USI), to quantify the level of user satisfaction
Validate USI by an independent set of metrics that quantify the interactivity and smoothness of a conversation.
![Page 6: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/6.jpg)
Trace Collection
Collect Skype VoIP sessions and their network parameters. Present the network setup and filtering method
used in the traffic capture stage. Introduce the algorithm for extracting VoIP sess
ions from packet traces Strategy to sample path characteristics. Summarize the collected VoIP sessions.
![Page 7: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/7.jpg)
Network Setup
![Page 8: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/8.jpg)
Capturing Skype Traffic
Use 2-phase filtering to identify Skype VoIP sessions:
filter and store possible Skype traffic on the disk.
apply an off-line identification algorithm on the capture packet traces to extract actual Skype sessions.
![Page 9: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/9.jpg)
Detect Possible Skype Traffic
Known properties of Skype clients:dynamic port number chosen randomly wh
en the application is installed and can be configured by users – “Skype port”
In the login process, submits HTTP requests to a well-known server, ui.skype.com
![Page 10: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/10.jpg)
Heuristic to Detect Skype Hosts and their Skype Ports
treat sender for each HTTP request sent to ui.skype.com as a Skype host
choose the port number used most frequently for outgoing UDP packets sent from that host within the next 10s as the Skype port.
classify all peers that have bi-directional communication with the Skype port as Skype hosts.
maintained a table of identified Skype hosts and their respective Skype ports, and
recorded all traffic sent from or to these (host, port) pairs.
![Page 11: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/11.jpg)
Identification of VoIP Sessions
regard An active flow as a valid VoIP session if The flow’s duration > 10s. The average packet rate is within a reasonable
range, (10, 100) pkt/s. The average packet size is within (30,300) byte
s. The EWMA of the packet size process must be
within (35, 500) bytes all the time.
![Page 12: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/12.jpg)
Relayed Session
Merge a pair of flow into a relayed session if The flows’ start and finish time are close to
each other with errors < 30s; The ratio of their average packet rates < 1.5;
and Their packet arrival processes are positively
correlated with a coefficient > 0.5.
![Page 13: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/13.jpg)
Path Characteristics Measurement
RTT and their jitters send out ICMP and traceroute-like probe packet
s to measure paths’ RTT while capturing Skype traffic
used
![Page 14: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/14.jpg)
Collected VoIP Sessions
Category Calls Hosts Cens. TCP Duration Bit Rate (mean/std)
Avg. RTT (mean/std)
Direct 253 240 1 7.10% (6.43, 10.42) min
32.21 Kbps / 15.67 Kpbs
157.3 ms/ 269.0 ms
Relayed 209 369 5 9.10% (3.12,5.58) min
29.22 Kbps / 10.28 Kpbs
376.7 ms/ 292.1 ms
Total 462 570 6 8.00% (5.17,7.70) min
30.86 Kbps / 13.57 Kpbs
256.5 ms/ 300.0 ms
![Page 15: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/15.jpg)
Methodology
Collect Skype VoIP sessions and their network parameters
Analysis of Call Duration and propose an objective index, USI, to quantify the level of user satisfaction
Validate USI by an independent set of metrics that quantify the interactivity and smoothness of a conversation.
![Page 16: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/16.jpg)
Analysis of Call Duration
Develop a model to describe the relationship between call duration and QoS factors.
propose an objective index, the User Satisfaction Index (USI) to quantify the level of user satisfaction.
validate USI by voice interactivity measures.
![Page 17: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/17.jpg)
Survival Analysis
With proper transformation, the relationships of session time and predictors can be described well by the Cox Proportional Hazards model (Cox Model) in survival analysis.
![Page 18: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/18.jpg)
Survival Curves for Sessions with Different Bit Rate Levels
![Page 19: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/19.jpg)
Survival Curves for Sessions with Different Bit Rate Levels
Group Median (min)
1 2
3 20
Bit rates Last > 40 min
< 25 Kbps 3%
> 35 Kbps 30%
The log-rank test strongly suggests that call duration varies with different levels of bit rates.
![Page 20: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/20.jpg)
Relation of the bit rate with call duration
The trend of median duration shows a strong, consistent, positive, correlation with the bit rate.
![Page 21: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/21.jpg)
Effect of Network Conditions
Network conditions are also considered to be one of the primary factors that affect voice quality.
the fluctuations in the data rate observed at the receiver should reflect network delay variations to some extent.
used – jitter to denote the standard deviation of the bit rate, and packet rate jitter, or pr.jitter, to denote the standard devi
ation of the packet rate.
![Page 22: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/22.jpg)
Effect of Round-Trip Times
divided sessions into 3 equal-sized groups based on their RTTs, and compare their lifetime patterns with the estimated survival functions.
the 3 group differ significantly
Group Median duration of sessions (min)
RTTs > 270 ms 4
RTTs: 80 - 270 ms 5.2
RTTs < 80 ms 11
![Page 23: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/23.jpg)
Effect of Jitter Jitter has a much higher correlation with call
duration than RTT.
![Page 24: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/24.jpg)
Group Median session time (min)
Jitter > 2 Kbps 3
1 < Jitter < 2 Kbps 11
Jitter < 1 Kbps 21
These groups differs statistically
![Page 25: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/25.jpg)
QoS related to Call Duration
most of the QoS factors they defined, including: the source rate, RTT, and jitter
are related to call duration.
![Page 26: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/26.jpg)
Collinearity
Given that the bit rate & jitter are significantly correlated, true source of user dissatisfaction is unclear.
Use the Cox model and treat QoS factors, e.g. the bit rate, as risk factors or covariates; i.e. as variables that can cause failures. The hazard function of each session is decided
completely by a baseline hazard function and the risk factors related to that session.
![Page 27: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/27.jpg)
Collinearity 7 factors - bit rate (br),packet rate (pr),jitter, pr.jitt
er, packet size (pktsize), and round trip time (rtt)
br pr jitter pr.jitter pktsize rtt
br * +++ ++ +++ -
pr +++ * - --- + --
jitter ++ - * +++ +++
pr.jitter --- +++ *
pktsize +++ + +++ *
rtt - -- *+/- : positive or negative correlation collinearity is computed by Kendall’s statistic (Pearson’s product moment statistic yields similar results)
![Page 28: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/28.jpg)
Collinearity
the bit rate, packet rate, and packet size are strongly interrelated;
jitter and packet rate jitter are strongly interrelated.
the bit rate, jitter, and RTT are retained in the model
![Page 29: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/29.jpg)
Cox Model
define the risk factors of a session as a risk vector Z
h(t|Z) = h0(t) exp(tZ) = h0(t)exp(pk=1kZk)
h(t|Z) - the hazard rate at time t for a session with risk vector Z;
h0(t) - the baseline hazard function computed during the regression process;
= (1,…, p)t - the coefficient vector that corresponds to the impact of risk factors.
Zp is the pth factor of the session
![Page 30: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/30.jpg)
The Cox model 2 sessions with risk vectors Z and Z’, the
hazard ratio:
h(t|Z)/ h(t|Z’) = exp(pk=1kZk – kZ’k)
is time-independent constantHence, the validity of the model relies on t
he assumption – the hazard rates for any 2 sessions must be in proportion all the time.
![Page 31: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/31.jpg)
Sampling of QoS Factors
In the regression modeling, we use a scalar value for each risk factor to capture user perceived quality.
Divide the original series s into sub-series of length w, from which network conditions are sampled.
Choose one of the min, average and max measures taken from sampled QoS factors having length [|s|/w] depending on their ability to describe the user perceived experience during a call.
![Page 32: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/32.jpg)
Evaluation
evaluate all kinds of measures and window sizes by: fitting the extracted QoS factors into the Cox m
odel and comparing the model’s log-likelihood, i.e. an indi
cator of goodness-of-fit.
Finally, the max bit rate and min jitter are chosen, both sampled with a window of 30s.
![Page 33: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/33.jpg)
Model Fitting
the Cox model assumes a linear relationship between the covariates and the hazard function
the impact of the covariates on the hazard functions with the following equation:
This corresponds to a Poisson regression model if h0(s) is known.
0
0 )()())(exp(][ dsshstIZfsE it
i
ti - the censoring status of session i, f(Z) – the estimated functional form of the covariate Z.
![Page 34: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/34.jpg)
The influence of the bit rate is not proportional to its magnitude - scale transformation.
![Page 35: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/35.jpg)
The RTT factor has an approximate linear impact.
![Page 36: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/36.jpg)
Jitter factor
![Page 37: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/37.jpg)
Verification Employ a more generalized Cox model that allo
ws time-dependent coefficients to check the proportional hazard assumption by hypothesis tests. After adjustment, none of covariates reject the linearity hypothesis at = 0.1, the transformed variables have an approximate linear impact on the hazard functions.
Use the Cox and Snell residuals ri (for session i) to assess the overall goodness-of-fit of the model. Except for a few sessions that have unusual call duration, most sessions fit the model very well.
![Page 38: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/38.jpg)
Model Interpretation
Variable Coef, eCoef Std. Err. z P > |z|
br.log -2.15 0.12 0.13 -16.31 0.00e+00
jitter.log 1.55 4.7 0.09 16.43 0.00e+00
rtt 0.36 1.4 0.18 2.02 4.29e-02
define the factors’ relative weights as their contribution to the risk score, i.e., tZ.
- coeff
![Page 39: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/39.jpg)
Relative Influence of Difference QoS for each session
The degrees of user dissatisfaction caused by the bit rate, jitter and round-trip time are 46%:53%:1%.
![Page 40: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/40.jpg)
Conclusion
Possible to improve user satisfaction by fine tuning the bit rate used.
As the use of relaying does not seriously degrade user experience, higher round-trip times do not impact on users very much.
Jitters have much more impact on user perception.
The choice of relay node should focus more on network conditions, i.e., the level of congestion, rather than rely on network latency.
![Page 41: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/41.jpg)
User Satisfaction Index (USI)
As the risk score tZ represents the levels of instantaneous hang up probability, it can be seen as a measure of user intolerance. Accordingly, define the USI of a session as its minus risk score:
USI = - tZ = 2.15xlog(bit rate) – 1.55xlog(jitter) – 0.36xRTT
where the bit rate, jitter, and RTTs are sampled using a 2-level sampling approach
![Page 42: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/42.jpg)
The prediction is based on the median USI for each group. y-axis is logarithmic to make the short duration groups clearer.
![Page 43: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/43.jpg)
Advantages of USI over Other Objective Sound Quality Measures
USI’s parameters are readily accessible: the 1st and 2nd moment of the packet counting process
can be obtained by simply counting the number and bytes of arrival packets
the round-trip times.Usually available in peer-to-peer applications for overlay network construction and path selection.
developed the USI based on passive measurement rather than subjective surveys, it can also capture sub-conscious reactions of participants, which may not be accessible through surveys.
![Page 44: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/44.jpg)
Methodology
Collect Skype VoIP sessions and their network parameters
Analysis of Call Duration and propose an objective index, the User Satisfaction Index (USI), to quantify the level of user satisfaction
Validate USI by an independent set of metrics that quantify the interactivity and smoothness of a conversation.
![Page 45: Quantifying Skype User Satisfaction](https://reader036.fdocuments.in/reader036/viewer/2022070408/56814404550346895db096f4/html5/thumbnails/45.jpg)
Validation
Results of the validation tests using a set of independent measures derived from user interactivities show a strong correlation between the call durations and user interactivities. This suggests that the USI based on call duration is significantly representative of Skype user satisfaction.