Maritime Deception and Concealment: Concepts for Defeating ...
Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the...
Transcript of Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the...
TechnischeUniversität Berlin URL: www.tkn.tu-berlin.de
Predicting the Perceptual Service QualityUsing a Trace of VoIP Packets
C. Hoene, S. Wiethölter, A. WoliszSeptember 29th, 2004.
Qofis’04, Barcelona, Spain
Content
Introduction/BackgroundOur ApproachPlayout SchedulingListening-only Tests
Delay SpikesNon-Random Packet Losses
Conclusions
VoIP transmission of a telephone call
How to Judge the Quality of a Call?
Measure the quality of a VoIP system:Gather packet loss rates, mean delay, etc.
Easy but inaccurate
In the end the service quality is importantHuman based listening tests are extensive
ITU P.862 (PESQ algorithm) measures speech qualityCompares original sample with the transmitted versioncalculates Mean Option Score (MOS) (1=bad, 5=excellent)
ITU G.107 (E-Model) predicts quality of tel.-systemConsiders echo, loudness, coding, packet loss rate, delay, …Result: R Factor (0=bad, 70=toll quality, 100=excellent)
Limits of PESQ and E-Model
PESQ and E-Model cannot judge the impact of:Impact of playout scheduling cannot be predicted.Variable packet delay
adaptive playout timeNon-random packet losses Not applicable to assess VoIP packet traces resulting from simulations or experiments
Predicting Playout Scheduling
A VoIP phone can implement any playout scheduler. How to know which is going to be used? Actually, you cannot know it.But you know the most common, published playout scheduler schemes, which are
Fixed-deadline and adaptive from Van Jacobsen, Mills, Schulzrinne, Ramjee, and Moon
Calculate packets' playout times and the mean transmission delay.
Consider only speech frames during voice activity because during silence a human cannot identify the delay.
We can predict the behavior of most phones !
Assessing a Trace of VoIP Packets
Our solution:Using the most common different playout schedulers and encoding schemescombine of E-Model and PESQ (MOS to R formula approved by the ITU)
PESQ calculate speech qualityE-Model combinesMOS rating and transmission delays
Speech Quality and Transmission Delay
Speech Recordings
SpeechCoding
(coding distortion)
Experimentor Simulation
(loss or delay)
Decoder PLC
PESQReferenceDegraded
Audio
AudioMOS
Playout Buffer
(loss or scheduling)
E-ModelIe Result:Id R Factor
Degraded Audio
Mean Delay
Equations – referring to the SPECTS’04 paper…
Content
Introduction/BackgroundOur ApproachPlayout SchedulingListening-only Tests
Delay SpikesNon-Random Packet Losses
Conclusions
PESQ not verified conduct listening tests
Can PESQ measure playout rescheduling?or non-random packet losses?
PESQ can measure variable playout but its performance has never been verified.It has been tested only for random/burstybut not for non-random losses.
Conduct formal listening-only tests!
Test Sample Design
Construct packet traceswith delay spikes!Delay spikes with a height of 50 to 300ms and a width of height+10%.1 Spike in one sample
What to do?1. Drop late packets2. Or delay playout?
Playout Decisions
Original
a) Packet loss b) Positive adaptation c) Negative adaptation
1 2 2+ 2++ 5 6 1 2 2+ 2++ 3 4 5 6 1 2 5 6
Legend:1...6 Frame seq. no. 2+;2++ Concealed frame
1 2 3 4 5 6
1. DROP: Drop late packets (a)2. ADAPT: Delay playout to late packets (b)3. ADAPT&FALLBACK: Delay playout (b) and return to
normal during the next silence period (c)
Listening-Only Tests
Follow ITU P.830 butMultiple person at the same timeUse native and foreign speakers Use linear MOS scale (no discrete scale!)Sample Sequence: For each sentence
1. Original at the beginning, 2. MNRU with 5dB (worst),15,25,35 (best)3. Randomly distributed: packet loss samples
Produced on CD containing all the samples
Studio...
Listening-Test and Sample DesignDifferent encoding schemes:
G.711, G729
Produced delay spikes (only during voice activity)with length 50,100,150,200,250,300ms with width=length*1.1
Different samples with different length 5-10s), German (using Kiel Corpus Vol. I)220 samples (including MNRU samples as a reference)
Coding Loss Generator
DecodingPLC
Listening-only Tests
Frame Analysis PESQ
reference
speechproperties
importancePESQ MOS
X
seed
Speech Recordings R
(Correlation)
pack
etiz
atio
n
loss
rate
rate
/mod
e
algo
rithm
sam
ple
spea
ker
lang
uage
MOS
Rating Performance
Three parts: intro, first and second half. First is best.Native speaker and foreign speaker rate as good.Some persons did not kept track with the sample no.
Human LQS-MOS vs. PESQ LQO-MOS
y = 0,9403x - 0,2139R= 0,8316
1
2
3
4
1 2 3 4
MOS (Humans)
MO
S (P
ESQ
)
MNRU vs. MOS: Correlation: R=0.977
1
2
3
4
5 15 25 35 45
MNRU
MO
S
MOS PESQ MOS MOS cited scaled MOS
Results for G.711 coding
MOS Variance vs. Prediction Performance
trend line: R=0.768
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1,0
0 0,2 0,4 0,6 0,8 1PESQ sample variance for different impairment patterns
Cor
rela
tion
betw
een
PESQ
and
hum
ans
Content
Introduction/BackgroundOur ApproachPlayout SchedulingListening-only Tests
Delay SpikesNon-Random Packet Losses
Conclusions
Introduction
Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range,depending on
the frame speech propertiesthe encoder/decoder/concealment algorithmsdecoders resynchronization time after loss (especially low-rate decoders might maintain a wrong state after loss for the following frames.)the surrounding speech.
Example: Discontinuous Transmission (DTX)Speech frames during silence are less importantLower frame rate during silence
Packet Importance
The packet’s importance is the quality degradation that its loss would cause.
Definition: The importance of frame losses is the difference between the speech quality due to coding loss and the quality due to coding loss and frame losses, times the length of the analyzed sample.
Conducted some million PESQ ratings Dropped only one VoIP packet in a sample
Just try it: www.tkn.tu-berlin.de/research/mongoliaSelect predefined parameter sets
chose sample
Compression?
Listen to it!
amount of loss
Choose an another loss pattern
Drop only frames with an importance between
min. and max.
Talking or silence? Voiced or unvoiced?
Judge the speech quality by your self!
Sample statistics
Conclusion
Presented an approach to assess VoIP packet tracesCombined PESQ, E-Model and playout scheduling because
Speech frames differ in their importancePlayout scheduler are not standardizedPlayout rescheduling harms speech quality.
Verified PESQ for non-random packet losses (R=0.94) and delay spikes (R=0.87).
Precision performances depend on the variance of the samples’ speech quality.
Assessing VoIP: SummaryOpen-source software including
Implementation of all common playout schedulersG.711 and G.729 codingSample database including human rating results
Verified software in various research projects, e.g. voice over WLAN, impact of handover, ad-hoc.http://www.tkn.tu-berlin.de/reseach/qofis
C. Hoene, S. Wiethölter, and A. Wolisz, "Predicting the Perceptual Service Quality Using a Trace of VoIP Packets", In Proceedings of QofIS’04, Barcelona, Spain, September 2004.C. Hoene, H. Karl, and A. Wolisz, "A Perceptual Quality Model for Adaptive VoIP Applications", In Proceedings of International Symposium on Performance Evaluation of Computer and Telecommunication Systems (SPECTS'04), San Jose, California, USA, July 2004, Paper won the Best Paper Award of the conference. C. Hoene and E. Dulamsuren-Lalla, "Predicting Performance of PESQ in Case of Single Frame Losses", In Proc. MESAQIN 2004, Prague, CZ, June 2004.S. Möller and C. Hoene, "Information About a New Method For Deriving the Transmission Rating Factor R From MOS in Closed Form", ITU, May 2002, Temporary Document for the study group 12.