1

Experiences from SLAC SC2004 Bandwidth Challenge

Les Cottrell, SLACwww.slac.stanford.edu/grp/scs/net/talk03/bwc-04.ppt

2

Sponsors/partners

3

SLAC/SC04 Bandwidth Challenge (plan C)

ESnet/QWest OC192/SONET

Sunnyvale/Qwest/ESnet

Juniper T320

Sunnyvale/Level(3)/NLR

SLAC/FNAL booth (2418)

1400 Kifer

1380 Kifer

10Gbps from NLR (via SEA, DEN, CHI)

PSC SciNet

SiSi

LoanedCisco Rtr

NLR-PITT-SUNN-10GE-17 NLR-PITT-SUNN-10GE-17

1 Sun file server 1 GE

2 Sun Opteron/Chelsio-10GE

SiSi

1554015808

NLR demarc 15808

15454

NLR demarc

SLACCisco Rtr

6 Sun Opteron/Chelsio-10GE

6 Boston file servers 1 GE 2 Sun file server 1 GE2 Sun Opteron/S2io-10GE

1 Sun Opteron/Chelsio-10GE

4

SC2004: Tenth of a Terabit/s Challenge

Joint Caltech, SLAC, Joint Caltech, SLAC, FNAL, CERN, UF, FNAL, CERN, UF, SDSC, BR, KR, ….SDSC, BR, KR, ….

10 10 Gbps waves to 10 10 Gbps waves to

HEP on show floorHEP on show floor Bandwidth Bandwidth

challenge: aggregate challenge: aggregate throughput throughput of 101.13 Gbpsof 101.13 Gbps

FAST TCPFAST TCP

5

Components10 Gbps NICs

S2io

Chelsio

1982 10Mbps3COM

v20z

v40z

3510 disk array

SR XENPAK

SVL/NLR

6

Challenge aggregates from SciNet

• Aggregate Caltech & SLAC booth, in & out

• 7 lambdas to Caltech, 3 ro SLAC

7

Challenge aggregates from MonALISA

• Sustained ~ 10Gbps for extended periods

8

Weathermap showing 8.7Gbps on ESnet

9

To/From SLAC booth• NLR: 9.43Gbps (9.07 goodput) + 5.65Gbps

(5.44Gbps goodput) in reverse– Two hosts to two hosts

• ESnet: 7.72Gbps (7.43Gbps goodput)– Only one 10Gbps host at SVL

• Single V40Z host with 2*10GE NICs to 2*V20Z across country got 11.4Gbps

• S2io and Chelsio (& Cisco & Juniper) all interwork

• Chelsio worked stably on uncongested paths

10

TOE• Chelsio had TCP Offload Engine

– Utilization factor of throughput & parallel streams

– Reduced cpu c.f. S2io non0TOE by factor ~ 3

11

Challenges• Could not get 10Gbps waves to SLAC only SVL

– Equipment in 3 locations• Keeping configs in lock-step (no NFS, no name

service)• Security concerns, used iptables• Machines only available 2 weeks before, some not

until we got to SC04• Jumbo frames not configured correctly at SLAC booth,

used 1500B frames mainly• Mix of hdw/swr: Opterons with various GHz & disks,

Xeons; Solaris 10, Linux 2.4, 2.6• Coordination between booths (sep by 100 yds)• Everything state of art (Linux 2.6.6, SR XENPAKs,

NICs

12

Award