Post on 17-Jan-2016
description
DataTAG/WP7 Amsterdam June 2002R. Hughes-Jones Manchester
1
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
CdL was here
EU DataGrid - Network Monitoring
Richard Hughes-Jones, University of ManchesterThanks to a wireless connected Titanium in the
garden@Houten
DataTAG/WP7 Amsterdam June 2002R. Hughes-Jones Manchester
2
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
CdL was here
European Topology: NRNs & Geant
SuperJANET4
CERN
UvA
Manc
SURFnet
DataTAG/WP7 Amsterdam June 2002R. Hughes-Jones Manchester
3
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
CdL was here
Gigabit Throughput on the Production WAN
Manc - RAL 570 Mbit/s 91% of the 622 Mbit access link
between SuperJANET4 and RAL 1472 bytes propagation ~21s
Manc-UvA (SARA) 750 Mbit/s SJANET4 + Geant + SURFnet
Manc – CERN 460 Mbit/s CERN PC had a 32 bit PCI bus
DataTAG/WP7 Amsterdam June 2002R. Hughes-Jones Manchester
4
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
CdL was here
Gigabit TCP on the Production WAN (01)
Throughput vs TCP buffer size
TCP window sizes in Mbytes calculated from RTT*bandwidth
Link RTT ms BW Measured by UDP
Expected BW for 1 Gbit/s
750 Mbit/s 460 Mbit/s
Man – Ams
14.5 1.81 1.36
Man – CERN
21.4 2.68 1.23
Window forBW 1 Gbit/s
Window Mbytes forUDP BW 750 Mbit/s 460 Mbit/s
DataTAG/WP7 Amsterdam June 2002R. Hughes-Jones Manchester
5
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
CdL was here
Gigabit TCP on the Production WAN Man-CERN
Throughput vs n-streams Default buffer size slope = ~25 Mbit/s/stream up to 9 streams then 15 Mbit/s/stream With larger buffers rate of increase per stream is larger Plateaus at about 7 streams giving a total throughput of ~400 Mbit/s
DataTAG/WP7 Amsterdam June 2002R. Hughes-Jones Manchester
6
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
CdL was here
UDP Throughput: SLAC - Man
SLAC – Manc 470 Mbit/s 75% of the 622 Mbit access link SuperJANET4 peers with ESnet
at 622Mbit in NY
DataTAG/WP7 Amsterdam June 2002R. Hughes-Jones Manchester
7
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
CdL was here
Gigabit TCP on throughput Man-SLAC
Throughput vs n-streams Much less than for European links Buffer required: rtt*BW (622Mbit) = ~14 Mbytes With larger buffers > default, rate of increase per stream is ~ 5.4 Mbit/s/stream No Plateau
TCP Man-SLAC
0
20
40
60
80
100
120
140
160
0 2 4 6 8 10 12
Number of TCP streams
Received throughput Mbit/s
1048576
1600512
2097152
3145728
4194304
DataTAG/WP7 Amsterdam June 2002R. Hughes-Jones Manchester
8
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
CdL was here
RiPE 1 wayUvA RAL ms
RiPE 1 wayRAL UvA ms
UDPmon Loss
ThroughputMbit/sMAN – UvA
~750Mbit/s
UDPmon & Ripe MAN-SARA from 25 May 02
DataTAG/WP7 Amsterdam June 2002R. Hughes-Jones Manchester
9
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
CdL was here
UDPmon Loss
Throughput~140 Mbit/sMan Lyon
Ping
Man Lyon
40 ms rtt
Iperf TCP Throughput~ 47 Mbit/sMan Lyon1048576 byte
buffer
iperf Pinger & UDPmon UK-Lyon From 25 May 02 – 04 Jun 02
DataTAG/WP7 Amsterdam June 2002R. Hughes-Jones Manchester
10
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
CdL was here
Work on End Systems: PCI: SysKonnect SK-9843
Motherboard: SuperMicro 370DLE Chipset: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz
RedHat 7.1 Kernel 2.4.14
SK301 1400 bytes sent Wait 20 us
Sk303 1400 bytes sent Wait 10 us Frames are back-to-back Can drive at line speed Cannot go any faster !
Gig Eth frames back to back
DataTAG/WP7 Amsterdam June 2002R. Hughes-Jones Manchester
11
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
CdL was here
PCI: Intel Pro/1000 Motherboard: SuperMicro 370DLE Chipset:: ServerWorks III LE Chipset CPU: PIII 800 MHz PCI:64 bit 66 MHz
RedHat 7.1 Kernel 2.4.14
IT66M212 1400 bytes sent Wait 11 us ~4.7us on send PCI bus PCI bus ~45% occupancy ~ 3.25 us on PCI for data recv
IT66M212 1400 bytes sent Wait 11 us Packets lost Action of pause packet?
DataTAG/WP7 Amsterdam June 2002R. Hughes-Jones Manchester
12
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
CdL was here
UDP Throughput: Intel Pro/1000 on B2B P4DP6
Max throughput 950Mbit/s Some throughput drop for packets
>1000 bytes
Loss NIC dependent Loss not due to user Kernel
moves
Traced to discards in the receiving IP layer ???
Motherboard: SuperMicro P4DP6 Chipset: Intel E7500 (Plumas) CPU: Dual Xeon Prestonia (2cpu/die) 2.2 GHz Slot 4: PCI, 64 bit, 66 MHz RedHat 7.2 Kernel 2.4.14
DataTAG/WP7 Amsterdam June 2002R. Hughes-Jones Manchester
13
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
CdL was here
Packet Loss: Where? Intel Pro 1000 on 370DLE 1472 byte packets Expected loss in transmitter !
UDPmon
UDP
IP
Eth drv
UDPmon
UDP
IP
Eth drv
HW HW
N Gen
N Transmit
N Lost
IpDiscards
N Received
Gig Switch
No loss at switch But Pause packet seen to sender
DataTAG/WP7 Amsterdam June 2002R. Hughes-Jones Manchester
14
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
CdL was here
Interrupt Coalescence: Latency Intel Pro 1000 on 370DLE 800 MHz CPU
0
20
40
60
80
100
120
140
160
180
0 200 400 600 800 1000 1200 1400
Packet size bytes
Latency uscoal5_0 coal10_0coal20_0 coal40_0coal64_0 coal100_0coal0_0
DataTAG/WP7 Amsterdam June 2002R. Hughes-Jones Manchester
15
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
CdL was here
Interrupt Coalescence: Latency Intel Pro 1000 on 370DLE 800 MHz CPU
coal50_tx0
0500
100015002000250030003500400045005000
50 59 68 77 86 95104113122131140149158167176185194
coal50
0
500
1000
1500
2000
2500
3000
3500
4000
50 60 70 80 90 100110120130140150160170180190
coal10
0
1000
2000
3000
4000
5000
6000
7000
8000
50 58 66 74 82 90 98106114122130138146154162170178186194
coal10_tx0
0
1000
2000
3000
4000
5000
6000
7000
8000
50 58 66 74 82 90 98106114122130138146154162170178186194
DataTAG/WP7 Amsterdam June 2002R. Hughes-Jones Manchester
16
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
CdL was here
Interrupt Coalescence: Throughput
Intel Pro 1000 on 370DLE
Throughput 1472 byte packets
0
100
200
300
400
500
600
700
800
900
0 10 20 30 40
Delay between transmit packets us
Received Wire rate Mbit/s
coa5
coa10
coa20
coa40
coa64
coa100
Throughput 1000 byte packets
0
100
200
300
400
500
600
700
800
0 10 20 30 40
Delay between transmit packets us
Received Wire rate Mbit/s
coa5
coa10
coa20
coa40
coa64
coa100