Telchemy – IPtel 2001

Modeling the effects of Burst Packet Loss and Recency on

Subjective Voice Quality

Alan Clark

Telchemy

alan@telchemy.com

Telchemy – IPtel 2001

Embedded Monitoring• Need to monitor QoS to provide feedback

on network performance / impact on subjective quality

• Desirable to provide monitoring in the form of a lightweight software agent

• Focus on time varying impairments – burst packet loss and “recency”

Telchemy – IPtel 2001

Embedded Monitoring

IPNetwork

GatewayGateway

VQmon Agent embedded into VoIP Gateway

QoSmetrics

SMS/NMS

Telchemy – IPtel 2001

The E Model• “Mouth to ear” transmission quality

measurement

• Produces an “R” factor typically in the range 50 (bad) -95 (good)

• R factor can be related to MOS score, Terminate Early (TME) etc.

• ITU G.107/ G.108 and ETSI ETR250

Telchemy – IPtel 2001

R Factor vs MOS

50

60

70

80

90

100

0.1 1 10

Percentage of users that terminate calls early

4.5

4.0

3.0

R Factor MOS

Telchemy – IPtel 2001

E Model

R = Ro - Is - Id - Ie + A

Base R value- Noise level

Impairments thatoccur simultaneouslywith speech- received speech level- sidetone level- quantization noise

Impairments thatare delayed withrespect to speech- talker echo- listener echo- round trip delay

Equipment ImpairmentFactor- CODEC- multiplexing effects

Advantage factor

Telchemy – IPtel 2001

Extended E Model

Delay, measuredusing RTCP

NetworkR FactorIe

PacketLoss

Jitter

Codectype

LossModel

JitterModel

CodecModel

Burstmodel

Recencymodel

UserR Factor

Delaymodel

Telchemy – IPtel 2001

Burst vs average loss

Burst of packet lossZero packet loss

Non-bursty packet loss

Telchemy – IPtel 2001

Instantaneous Quality

Source France Telecom

Telchemy – IPtel 2001

“Recency” effect

“Good” 4.3MOS“Bad” 1.8 MOS(3dB SNR)

MOS 3.82

MOS 3.28

MOS 3.18

Source AT&TT1A1.7/98-031

60 second call

Telchemy – IPtel 2001

Jitter and Packet Loss

ArrivingRTPpackets

Jitterbuffer

Late packetsdiscarded

Monitor jitter andpacket loss afterjitter buffer

CODEC

Telchemy – IPtel 2001

Loss Model - Markov model

3Lost

1Rcvd

P13

P22

P11

P312

Rcvd

P23P32

4Lost

P14

P41

Burst state

Gap state

Model parametersreconstructed at endof call

Telchemy – IPtel 2001

Loss Model - mapping loss to Ie

0

10

20

30

40

50

0 5 10 15

Packet Loss Rate

Ie (p

acke

t los

s)Curve is CODECdependant

Telchemy – IPtel 2001

Determining QoS metrics

1. Determine “good”and “bad” state Ie Factor

Telchemy – IPtel 2001

Determining QoS metrics

t = 5

t = 15

1. Determine “good”and “bad” state Ie Factor

2. EstimateInstantaneousR Factor foreach state

Telchemy – IPtel 2001

Determining QoS metrics

t = 5

t = 15

1. Determine “good”and “bad” state Ie Factor

2. EstimateInstantaneousR Factor foreach state

3. Determine average Ie

Telchemy – IPtel 2001

Delay effects

Source NTT

Conversation

Verify sequenceof random numbers

4

3

2

0 1 2 3 4

MeanOpinionScore

Delay (seconds)

Telchemy – IPtel 2001

Measuring Delay

CODECCODEC

Accumulateframe

Encode

Transmission Jitterbuffer

Decode

RTCP exchange

Telchemy – IPtel 2001

Delay Model

0

10

20

30

0 100 200 300 400

175 mS “knee”

End to end delay (mS)

R FactorReduction

Telchemy – IPtel 2001

Estimation of recency effect

Delay since last significant burst

Effects decay over 30-60 secondsAverage for call

Telchemy – IPtel 2001

Execution model

Call 1 Call 2

InitializePorts

Startcall

Endcall

Startcall

EndcallLoss events Loss events

Typically 1.4 assemblylanguage instructionsper packet

Telchemy – IPtel 2001

Integration with VoIP SMS

VQmonSNMPSet, Get, Trap

End of callmsg (DRQ)

CDRServiceMgtSystem

NetworkMgtSystem

Telchemy – IPtel 2001

Ranking accuracy – Set 1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

1 2 3 4 5 6 7

Predictor (1=VQmon, 2-7=listeners)

Acc

ura

cy

Telchemy – IPtel 2001

Ranking Accuracy – Set 2

00.10.20.30.40.50.60.70.80.91

1 2 3 4 5 6 7

Predictor (1=VQmon, 2-7=listeners)

Acc

ura

cy

Telchemy – IPtel 2001

Ranking Accuracy – Set 4

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

1 2 3 4 5 6 7

Predictor (1=VQmon, 2-7=listeners)

Acc

ura

cy

Telchemy – IPtel 2001

Conclusions• Computational model meets design goals

• Ranking accuracy is comparable to human listener

• But need - – Systematic comparison with PSQM/ PESQ– Increased level of subjective testing– Add support for VAD, non-PLC– Improved accuracy requires some information

on voice frame content

Telchemy – IPtel 2001

Further work areas• Use CODEC generated frame loss event –

indicate presence of speech energy

• Additional subjective testing– Wider variety of audio sources– Force listeners to focus on call content– Design impairments to isolate recency effect,

burst characteristics, masking effects