3/4/98 1

Internet Telephony & Internet Performance Issues

Les Cottrell SLAC<cottrell@slac.stanford.edu>

Presented at the XIWT/IPWT meeting San Jose February 4th, 1998

Partially funded by DOE/MICS Field Work Proposal on Internet End-to-end Performance Monitoring (IEPM)

3/4/98 2

Outline of talk• Credits• VoIP

– Why, how, setup, how to use, protocols, traces, results so far, next

• Relating to PingER performance measurements– delay, loss, jitter, availability

3/4/98 3

Credits• VoIP

– SLAC - Charley Granieri, Dave Millsom– CERN - Olivier Martin, Denise Heagerty– DESY - Michael Ernst, Andrey Bobyshev– FNAL - Phil DeMar, Vyto Grigaliunas– LBNL - Becca Nitzan, Mike Collins

• Monitoring– Warren Matthews, Les Cottrell - SLAC

3/4/98 4

Internet Telephony Pilot• Goal is to understand issues and see if, when and

where VoIP is applicable for HEP collaborations• Five National Lab sites (CERN (ch), DESY (de),

FNAL(Chicago), LBNL (Berkeley) & SLAC (Silicon Valley))– Testing/evaluating VoIP between them– Various PBXs including Alcatel, Siemens?, Lucent

5ESS, Nortel SL/1

3/4/98 5

Typical Set Up

Production Internet with no special purpose

router code

NortelPBX

AlcatelPBX

CiscoGateway

CiscoGateway

SLAC CERN

PSTN

G.729

3/4/98 6

Mechanism• User calls access code to get thru PBX to gateway• Gateway gives 2nd dial tone, enter site code, then

get dial tone from site PBX, enter extension at site• Encoding/compression done by gateway

(64kbps=>24kbps (8kbps w/o headers))• ~ 60 byte IP packets using UDP & RTP

3/4/98 7

HeadersOverhead in IP packet

20 ms @ 8kbps yields 20 byte payloadIP headers 20; UDP header 8; RTP header 12 bytes

2 x payload!Plus media headers

3/4/98 8

VoIP Packet traces

3/4/98 9

VoIP Inter Packet Delay distributions

Both UDP streams viewed on wire at SLAC end

3/4/98 10

Results• Tried two vendors’ gateways

– voice quality on first poor– voice quality on second very usable

• Occasional drop out (every few minutes)• Delay OK• No echo

• Lots of teething problems:– difficulty in disconnecting when hang-up– hard to get gateway vendors and PBX vendors “all in the same

room” & to be able to talk in each others terms• Cisco sent 3 people (escalation team=1 PBX expert, 1 VoIP expert, 1

cable expert).

3/4/98 11

Scale of Measurements• 17 Monitoring sites - 7 in US (5 ESnet, 2

vBNS), 2 in Canada, 6 in Europe (ch, de, dk, hu, it, uk), 2 in Asia (jp, tw)

• 1129 monitoring-remote-site pairs• 379 unique hosts• 272 sites• 27 countries• Data goes back 4 years• Metrics include response, jitter, loss,

reachability

PingER pair distribution by global areaRussian

Fed4%

Gov7%

South America

1%

Org1%Australasia

1%

Canada5%

China2%

Europe38%

Mil0%

Edu33%

Com2%

Japan3%

Asia2%

3/4/98 12

CERN Link to US very busy

3/4/98 13

QoS• All 4 sites set precedence bits for VoIP traffic• Precedence only takes effect on CERN-US link• What does ping performance look like for major

metrics (delay, “jitter”, loss, reachability ...)?

3/4/98 14

Nov-98 100 byte pings SLAC - CERNPercent

loss

Response in

msec

G.114 300msec RTT limit

Packet loss = 0.1%Median response = 182 msec

3/4/98 15

It’s getting better all the time 1/2Response time to selected groups of

sites from SLAC

0

50

100

150

200

250

300

350

400

450Ja

n-95

May

-95

Sep

-95

Jan-

96

May

-96

Sep

-96

Jan-

97

May

-97

Sep

-97

Jan-

98

May

-98

Sep

-98

Jan-

99

Ping

roun

d tri

p de

lay

in m

sec.

Canada (6 pairs)Edu/US (50 pairs)ESnet (14 pairs)Japan (6 pairs)Europe (27 pairs)cern.ch (1 pair)Expon. (Canada (6 pairs))Expon. (Edu/US (50 pairs))Expon. (ESnet (14 pairs))Expon. (Japan (6 pairs))Expon. (Europe (27 pairs))

1.2%/mo1.7%/mo1.4%/mo0.22%/mo0.72%/mo1.3%/mo

STAR-TAP

3/4/98 16

What about loss?• BCR Feb ‘98 & Jan ‘99 shows even with 10%

random loss can get almost toll quality • Our experience in other areas is to say problems start

between 2.5 and 5% packet loss• ITU/TIPHON defines a loss of < 3% as being “good”

for Internet telephony Consecutive frames lost: 1 2 3 4 5 Quality (MOS scale 1-5) 4.2 3.2 2.4 2.1 1.7 Perception good fair poor v.poor unsat

• if 20 msec between VoIP packets, then lose adjacent packets:– every 2 secs for 10% loss, every ~ 30 sec for 2.5% loss

3/4/98 17

It’s getting better all the time 2/3Round trip packet loss between

SLAC & VoIP sites

0.001

0.01

0.1

1

10

100

Jan-

95

May

-95

Sep

-95

Jan-

96

May

-96

Sep

-96

Jan-

97

May

-97

Sep

-97

Jan-

98

May

-98

Sep

-98

Jan-

99

May

-99

Sep

-99Pe

rcen

t pac

ket l

oss

DESY, 3.5%/mo, 33%

CERN, 8.5%/mo, 16%

FNAL, 9.0%/mo, 5%

3/4/98 18

It’s getting better all the time 3/3Packet loss between SLAC & selected sites

0.01

0.1

110

100

Jan-

95

Apr-

95

Jul-9

5

Oct

-95

Jan-

96

Apr-

96

Jul-9

6

Oct

-96

Jan-

97

Apr-

97

Jul-9

7

Oct

-97

Jan-

98

Apr-

98

Jul-9

8

Oct

-98

Jan-

99

Apr-

99

Jul-9

9

Oct

-99

Jan-

00

Perc

ent r

ound

trip

pac

ket l

oss

Europe 3.0%/moCanada 6.8%/moUS/Edu 7.6%/moJapan 0.4%/mo

ESnet 8.7%/moSLAc<=>vBNS ~ 2 * SLAC <=> ESnet

3/4/98 19

Countries Expected to be Good

3/4/98 20

What about “jitter”?• Jitter ~ instantaneous variability of delay• ITU/TIPHON defines a one-way jitter of < 75 msec.

as providing “good” Internet telephony– < 125msec is “medium” quality and < 225msec is “poor”

• How may it be measured?– ITU specifies inject packets at regular intervals and

measure the variability as received– IETF has the one-way Instantaneous Packet Delay

Variability (IPDV) draft– we are experimenting with 3 ways

3/4/98 21

RTT frequency histogram snapshotSLAC <=> CERN round trip

delay

y = 985.78x-1.3652

0

20

40

60

80

100

120

140

160

180

200

150 170 190 210 230 250 270 290 MorePing response time in msec.

Freq

uenc

y

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Cum

ulat

ive

FrequencyCumulative %Power (Frequency)

Average = 188msec.Std dev = 34 msec.Median = 173 msec.IQR = 42 msec.Loss = 0.3%1000 samples

3/4/98 22

Instantaneous inter-packet RTT SLAC<=>CERN two-way inter

packet delay difference

0102030405060708090

Ping inter packet delay difference in msec.

Freq

uenc

y

0102030405060708090

FrequencyGaussian

Average = -0.03 msec.Std dev = 35 msec.Median = 0 msec.IQR = 29 msecLoss = 0.3%1000 samples

Gaussian-prob=79*exp(-x**2/(2*(IQR/2)**2))

IPDD=dD(i)=D(i+1)-D(i)D(i) = RTT ping delay

3/4/98 23

Instantaneous inter-packet RTT magnitude

IQR(|IPDD|) between SLAC & CERN

y = 58.628e-0.064x

R2 = 0.8816

y = 351.11x-1.1879

R2 = 0.811

0

20

40

60

80

100

120

140

0 8 16 24 32 40 48 56 64 72 80 88 96 104

112

120

IQR(|IPDD|) msec.

Freq

uenc

y

0%10%20%30%40%50%60%70%80%90%100%

Cum

ulat

ive

FrequencyCumulative %Expon. (Frequency)Power (Frequency)

Average = 23.4 msec.Std dev = 25.8 msec.Median = 15 msec.IQR = 29 msec.Pings = 990

|IPDD| = abs(D(i)-D(i-1))

3/4/98 24

Packet Jitter

IQR(ipdv) & IQR(mag) between CERN & SLAC from Surveyor measurements (12/15/98 & medians for Dec-98)

0.1

1

10

100

0 5 10 15 20 25Time since midnight (GMT)

IQR

(IPD

V) in

mse

c.

IQR(ipdv) CERN>SLAC IQR(ipdv) SLAC>CERNMonthly IQR(ipdv) CERN>SLAC Monthly IQR(ipdv) SLAC>CERNIQR(mag) CERN>SLAC IQR(Mag) SLAC>CERNMonthly IQR(mag) CERN>SLAC Monthly IQR(mag) SLAC>CERN

CERN link utilization

TIPHON delayjitter threshold

3/4/98 25

Jitter Correlations

3/4/98 26

Effect of setting precedence on jitterIQR(Delay) and IQR(IPDD) with & without

precedence bits set SLAC=>CERN

0

2

4

6

8

10

12

14

16

22:1

0

23:1

0

0:10

1:10

2:10

3:10

4:10

5:10

6:10

7:10

8:10

9:10

10:1

0

11:1

0

12:1

0

13:1

0

14:1

0

15:1

0

16:1

0

17:1

0

18:1

0

Time GMT

IQR

in m

sec.

delay-iqr ipdv-iqrdelay-iqr ipdv-iqr

Precedence bits setNo precedence bits set

January 9, 1999

3/4/98 27

Effect of setting precedence bits

January 8-9, 1999

3/4/98 28

Reachability• Unreachable = all pings are lost• Varies a lot from site to site, it’s not phone quality

Unreachability of selected sites from SLAC

0.01

0.10

1.00

10.00

100.00FN

AL

CE

RN

DE

SY

ES

net

US

Can

ada

W.

Eur

ope

Japa

n

Site or group of sites

Perc

ent u

nava

iabl

e

25%Median75%95%

3/4/98 29

Next• How does ping jitter relate to VoIP jitter• How does one way jitter relate to two-way

– compare Surveyor & pingER in more detail• Working with CERN & Esnet/LBNL to understand

effect of preferential service– Set up test path, look at behavior of important metrics,

VoIP and video as vary load and use DiffServ

3/4/98 30

More Information & extra info follows• WAN Monitoring at SLAC has lots of links

– http://www.slac.stanford.edu/comp/net/wan-mon.html

• Tutorial on WAN Monitoring (including jitter & quality of service thresholds etc.)

– http://www.slac.stanford.edu/comp/net/wan-mon/tutorial.html• PingER History tables

– http://www.slac.stanford.edu//xorg/iepm/pinger/table.html