EU DataGrid - Network Monitoring Richard Hughes-Jones, University of Manchester

DataTAG/WP7 Amsterdam June 2002R. Hughes-Jones Manchester

1

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

CdL was here

EU DataGrid - Network Monitoring

Richard Hughes-Jones, University of ManchesterThanks to a wireless connected Titanium in the

garden@Houten


2


CdL was here

European Topology: NRNs & Geant

SuperJANET4

CERN

UvA

Manc

SURFnet


3


CdL was here

Gigabit Throughput on the Production WAN

Manc - RAL 570 Mbit/s 91% of the 622 Mbit access link

between SuperJANET4 and RAL 1472 bytes propagation ~21s

Manc-UvA (SARA) 750 Mbit/s SJANET4 + Geant + SURFnet

Manc – CERN 460 Mbit/s CERN PC had a 32 bit PCI bus


4


CdL was here

Gigabit TCP on the Production WAN (01)

Throughput vs TCP buffer size

TCP window sizes in Mbytes calculated from RTT*bandwidth

Link RTT ms BW Measured by UDP

Expected BW for 1 Gbit/s

750 Mbit/s 460 Mbit/s

Man – Ams

14.5 1.81 1.36

Man – CERN

21.4 2.68 1.23

Window forBW 1 Gbit/s

Window Mbytes forUDP BW 750 Mbit/s 460 Mbit/s


5


CdL was here

Gigabit TCP on the Production WAN Man-CERN

Throughput vs n-streams Default buffer size slope = ~25 Mbit/s/stream up to 9 streams then 15 Mbit/s/stream With larger buffers rate of increase per stream is larger Plateaus at about 7 streams giving a total throughput of ~400 Mbit/s


6


CdL was here

UDP Throughput: SLAC - Man

SLAC – Manc 470 Mbit/s 75% of the 622 Mbit access link SuperJANET4 peers with ESnet

at 622Mbit in NY


7


CdL was here

Gigabit TCP on throughput Man-SLAC

Throughput vs n-streams Much less than for European links Buffer required: rtt*BW (622Mbit) = ~14 Mbytes With larger buffers > default, rate of increase per stream is ~ 5.4 Mbit/s/stream No Plateau

TCP Man-SLAC

0

20

40

60

80

100

120

140

160

0 2 4 6 8 10 12

Number of TCP streams

Received throughput Mbit/s

1048576

1600512

2097152

3145728

4194304


8


CdL was here

RiPE 1 wayUvA RAL ms

RiPE 1 wayRAL UvA ms

UDPmon Loss

ThroughputMbit/sMAN – UvA

~750Mbit/s

UDPmon & Ripe MAN-SARA from 25 May 02


9


CdL was here

UDPmon Loss

Throughput~140 Mbit/sMan Lyon

Ping

Man Lyon

40 ms rtt

Iperf TCP Throughput~ 47 Mbit/sMan Lyon1048576 byte

buffer

iperf Pinger & UDPmon UK-Lyon From 25 May 02 – 04 Jun 02


10


CdL was here

Work on End Systems: PCI: SysKonnect SK-9843

Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz

RedHat 7.1 Kernel 2.4.14

SK301 1400 bytes sent Wait 20 us

Sk303 1400 bytes sent Wait 10 us Frames are back-to-back Can drive at line speed Cannot go any faster !

Gig Eth frames back to back


11


CdL was here

PCI: Intel Pro/1000 Motherboard: SuperMicro 370DLE Chipset:: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz

RedHat 7.1 Kernel 2.4.14

IT66M212 1400 bytes sent Wait 11 us ~4.7us on send PCI bus PCI bus ~45% occupancy ~ 3.25 us on PCI for data recv

IT66M212 1400 bytes sent Wait 11 us Packets lost Action of pause packet?


12


CdL was here

UDP Throughput: Intel Pro/1000 on B2B P4DP6

Max throughput 950Mbit/s Some throughput drop for packets

>1000 bytes

Loss NIC dependent Loss not due to user Kernel

moves

Traced to discards in the receiving IP layer ???

Motherboard: SuperMicro P4DP6 Chipset: Intel E7500 (Plumas) CPU: Dual Xeon Prestonia (2cpu/die) 2.2 GHz Slot 4: PCI, 64 bit, 66 MHz RedHat 7.2 Kernel 2.4.14


13


CdL was here

Packet Loss: Where? Intel Pro 1000 on 370DLE 1472 byte packets Expected loss in transmitter !

UDPmon

UDP

IP

Eth drv

UDPmon

UDP

IP

Eth drv

HW HW

N Gen

N Transmit

N Lost

IpDiscards

N Received

Gig Switch

No loss at switch But Pause packet seen to sender


14


CdL was here

Interrupt Coalescence: Latency Intel Pro 1000 on 370DLE 800 MHz CPU

0

20

40

60

80

100

120

140

160

180

0 200 400 600 800 1000 1200 1400

Packet size bytes

Latency uscoal5_0 coal10_0coal20_0 coal40_0coal64_0 coal100_0coal0_0


15


CdL was here

Interrupt Coalescence: Latency Intel Pro 1000 on 370DLE 800 MHz CPU

coal50_tx0

0500

100015002000250030003500400045005000

50 59 68 77 86 95104113122131140149158167176185194

coal50

0

500

1000

1500

2000

2500

3000

3500

4000

50 60 70 80 90 100110120130140150160170180190

coal10

0

1000

2000

3000

4000

5000

6000

7000

8000

50 58 66 74 82 90 98106114122130138146154162170178186194

coal10_tx0

0

1000

2000

3000

4000

5000

6000

7000

8000

50 58 66 74 82 90 98106114122130138146154162170178186194


16


CdL was here

Interrupt Coalescence: Throughput

Intel Pro 1000 on 370DLE

Throughput 1472 byte packets

0

100

200

300

400

500

600

700

800

900

0 10 20 30 40

Delay between transmit packets us

Received Wire rate Mbit/s

coa5

coa10

coa20

coa40

coa64

coa100

Throughput 1000 byte packets

0

100

200

300

400

500

600

700

800

0 10 20 30 40

Delay between transmit packets us

Received Wire rate Mbit/s

coa5

coa10

coa20

coa40

coa64

coa100

EU DataGrid - Network Monitoring Richard Hughes-Jones, University of Manchester

Documents

Transcript of EU DataGrid - Network Monitoring Richard Hughes-Jones, University of Manchester