Post on 14-Aug-2019
Quality of Service and Quality of Experience@ T-Labs Berlin
Sebastian Möller, Alexander Raake and Marcel Wältermann
Quality and Usability LabDeutsche Telekom LaboratoriesTU Berlin
{sebastian.moeller, alexander raake, marcel.waeltermann}@telekom.de
1
Outline
Quality of Service (QoS) and Quality of Experience (QoE)
– Definitions
– Quality taxonomy
Subjective evaluation methods
Quality prediction
2
QoS and QoEDefinitions.
System developer‘s point-of-view:
Performance: “The ability of a unit to provide the function it has been designed for.”(Möller, 2005)
Quality of Service (QoS): “The collective effect of service performance which determines the degree of satisfaction of the user of the service.” (ITU-T Rec. E.800, 1994)
Includes service support, service operability, serveability, and service security
User‘s point-of-view:
Quality: “Result of appraisal of the perceived composition of the service with respect to its desired composition.”(ITU-T Rec. P.851, 2003, following Jekosch, 2000, 2005)
Quality of Experience (QoE): “The overall acceptability of an application or service,as perceived subjectively by the end user.” (ITU-T Rec. P.10, 2007)
Includes the complete end-to-end system effects
May be influenced by user expectations and context
3
QoS and QoEDefinitions.
Definitions discussed by participants of the Dagstuhl Seminar 09192"From Quality of Service to Quality of Experience“ (May 2009)
Quality of Experience (QoE): “Degree of delight of the user of a service.In the context of communication services, it is influenced by content,network, device, application, user expectations and goals, and context of use.”
Service: “An event in which an entity takes the responsibility that something desirable happens on the behalf of another entity.”
Acceptability: “Characteristic of a service describing how readily a person willuse the service. Acceptability is the outcome of a decision which is partially based on the Quality of Experience.”
4(Möller et al., 2009)
QoS and QoETaxonomy.
Hedonic Pragmatic
User Context
Cognitiveworkload
Perceptualeffort
Physical responseeffort
Formappropriatness
Output modalityappropriatness Contextual
appropriateness DialogmanagementperformanceInput
performance Input modalityappropriatness
Interpretationperformance
Environmentalfactors
Servicefactors
Inputquality
Output quality
Cooperativity
Efficiency
Interaction quality
Effectiveness
System Personality Utility
Usefulness
Acceptability
Joy of use
Learnability
IntuitivityEase of use
Usability
Quality of Experience (Q
oE)
Staticfactors
Dynamicfactors
User System
System
Agentfactors
Functionalfactors
Appeal
Aesthetics
Quality of Service (Q
oS)
Interactionperform
anceInfluencing
factors
5
Outline
Quality of Service (QoS) and Quality of Experience (QoE)
Subjective evaluation methods
– Dimension-based approach
– Evaluation of conversational quality
Quality prediction
6
Dimension-based Assessment of Speech QualityAuditory analysis.
(Wältermann et al., 2006; 2008)
Auditory tests
– Multidimensional Scaling
– Semantic Differential
Quality Dimensions:
– “Discontinuity”
– “Noisiness”
– “Coloration”
7
Dimension-based Assessment of Speech QualityModelling principle.
(Heute et al., 2005; Wältermann et al. 2009)
ModelEstimatedQuality Index
TransmissionSystem
Estimated Quality Dimensions
Knowledge
SubjectiveQuality and FeatureJudgements
dis noi col
tot
8
Subjective Evaluation MethodsEvaluation of conversational quality – from 2 to 3 subjects.
21 3
welcome
persons
summary
discussion of open question
goodbye
interactivetask
open question
request/proposal
objection/proposal
necessaryinformation
9
3CTs conversation scenarios. Conversation tests.
Two conversation test series conducted, yielding a total of 2*8*10 = 160 (recorded) 3-person conversations.
Conversation analysis ongoing.
– Basis for further research.
Speech quality judgments
– 3D Audio versus diotic.
– In conversation, reduced spatial quality advantage when compared with listening only (e.g. Baldis, 2001; Raake et al., 2007)
Listening test in preparation including memory & performance.
(Raake and Schlegel, 2008)
10
Outline
Quality of Service (QoS) and Quality of Experience (QoE)
Subjective evaluation methods
Quality prediction
– Signal-based: P.OLQA
– Parametric: E-model
– IPTV
11
Quality Prediction ModelsSignal-based speech quality prediction: P.OLQA.
(Wältermann et al., 2008)
TransmissionSystem
Model
Listener
Measurement Tool
MOS
Estimated MOS
Linear Dist.
Signal Processing
Packet loss
RoomAcoustics Noise
Codec...
12
P.OLQA Candidate Model.Signal-based Core & Single Dimension Estimators.
TransmissionSystem
InternalRepresent.
InternalRepresent.
Pre-Processing
Pre-Processing
Comparison Integration Transform.
dMOS
NoisinessIndicator
ColorationIndicator
LoudnessIndicator
DiscontinuityIndicator
(Côté 2008; Wältermann et al., 2008b,c)
bIdisbInoibIcol ≈ IbwbIlou
13
Quality Prediction ModelsParametric speech quality prediction: E-model.
Backgr.noise,acousticcoupling
Linear distortion, delay
Codec Packetloss
Jitterbuffer,VAD
Talker echo,listener echo
Circuitnoise
Backgr.noise,acousticcoupling
IP WANIP WAN
4
4
14
IP WANIP WAN
4
4
Overall quality R = Ro - Is - Id - Ie,eff
Estimated user judgment MOS = f (R )
Impairments SNR simultaneous delayed nonlin./timevar.
Ps, Ds,STMR
SLR, RLR, Ta
Ie, qdu PplBpl TELR, T,WEPL, Tr
Nc, Nfor Pr, Dr,LSTR
Quality Prediction ModelsParametric speech quality prediction: E-model.
15
Quality Prediction ModelsParametric speech quality prediction: E-model.
Extension to wideband and beyond
(Raake, 2006; Möller et al, 2006; Appendix II, ITU-T Rec. G.107, 2006)
0 20 40 60 80 1000
20
40
60
80
100
120
140Rmax=129→
RNB/WBR
NB
Ro,max = 129AMR-WB (23.05)
AMR-WB (6.6)
16
Quality of IPTVOverview – perceptual bitstream model
(Raake & Garcia, 2008; Garcia & Raake, 2009)
17
Quality of IPTVVideo quality model – first algorithm
VVVVV FdisFtraFcodFresQoQv ++++=
bitrateccbbPpl
PplFcodbFtra
abitrateaaFcodQvFdisFresQo
VV
V
subjectiveVVV
/
)(
)exp(
)max(
211
12
321
+=+
⋅−=
+⋅⋅=
=++
(Raake et al., 2008)
BplPplPplIeIeeffIe+
⋅−+= )95(,
effIeIdIsRoR ,−−−=
Video VoIP (E-model)
Comparison with test results: r > 0.95,RMSE < 7
18
G.729, 2 frames/packet, native PLC
Qua
lity
(test
)
Quality of IPTVVideo quality under packet loss
2-state Markov lossParemeter: Average number of packets lost in a row μ10 = 1/q
H.264, SD, 4 MBit/s, 1 fpsPLCs: 1 slice/2 Macroblock lines vs. skipping
Video VoIP
Thank you for your attention.
Visit www.qu.tlabs.tu-berlin.de for more information.