Olivier Martin, CERN NEC’2005 Conference, VARNA (Bulgaria)

The ongoing evolution from Packet based networks to Hybrid Networks in Research & Education NetworksOlivier Martin, CERNNEC2005 Conference, VARNA (Bulgaria)

Presentation Outline

The demise of conventional packet based networks in the R&E communityThe advent of community managed dark fiber networksThe Grid & its associated Wide Area Networking challengeson-demand Lambda GridsEthernet over SONET & new standardsWAN-PHY, GFP, VCAT/LCAS, G.709, OTN

OC-768c40-GE

Internet Backbone SpeedsT1 LinesT3 linesOC3cOC12cIP/ATM-VCsMBPS

Chart1

0.056

0.256

4.5

4.5

10

20

40

180

360

900

2000

4000

8000

100000

5000000

Backbone Speed

Internet Backbone Speed (in Mbps)

Sheet1

YearBackbone Speed

19860.05656,000

19870.256256,000

19984.51,560,000

19894.51,560,000

1990103,000,000

1991204,500,000

19924010,000,000

199318045,000,000

1994360130,000,000

1995900300,000,000

19962000900,000,000

199740002,000,000,000

199880004,000,000,000

1999100000100,000,000,000

200050000005,000,000,000,000

Sheet1

Backbone Speed

Internet Backbone Speed (in Mbps)

Sheet2

Sheet3

High Speed IP Network Transport TrendsHigher Speed, Lower cost, complexity and overheadB-ISDNATMSONET/SDHIPOpticalMultiplexing, protection and management at every layerSignalling

Network ExponentialsNetwork vs. computer performanceComputer speed doubles every 18 monthsNetwork speed doubles every 9 monthsDifference = order of magnitude per 5 years1986 to 2000Computers: x 500Networks: x 340,0002001 to 2010Computers: x 60Networks: x 4000Moores Law vs. storage improvements vs. optical improvements. Graph from Scientific American (Jan-2001) by Cleo Vilett, source Vined Khoslan, Kleiner, Caufield and Perkins.

l

Know the userBW requirements# of usersCABA -> Lightweight users, browsing, mailing, home useB -> Business applications, multicast, streaming, VPNs, mostly LANC -> Special scientific applications, computing, data grids, virtual-presenceADSLGigE LAN(3 of 12)F(t)

l

What the userBW requirementsTotal BWCABA -> Need full Internet routing, one to manyB -> Need VPN services on/and full Internet routing, several to several C -> Need very fat pipes, limited multiple Virtual Organizations, few to fewADSLGigE LAN(4 of 12)

l

So what are the factsCosts of fat pipes (fibers) are one/third of equipment to light them upIs what Lambda salesmen told Cees de Laat (University of Amsterdam & Surfnet) Costs of optical equipment 10% of switching 10 % of full routing equipment for same throughput100 Byte packet @ 10 Gb/s -> 80 ns to look up in 100 Mbyte routing table (light speed from me to you on the back row!)Big sciences need fat pipesBottom line: create a hybrid architecture which serves all users in one coherent and cost effective way(5 of 12)

Utilization trends

GbpsNetwork Capacity LimitJan 2005

Todays hierarchical IP networkUniversityNational or Pan-National IP NetworkOther national networksNREN ANREN BNREN CNREN D

Tomorrows peer to peer IP networkWorldUniversityServerWorldWorldNational DWDM NetworkNREN ANREN BNREN CNREN DChildLightpathsChild Lightpaths

Creation of application VPNsCommodityInternetBio-informaticsNetworkUniversityUniversityUniversityCERNUniversityUniversityHigh Energy Physics NetworkeVLBI NetworkDeptResearch NetworkDirect connect bypasses campus firewall

Production vs Research Campus NetworksIncreasingly campuses are deploying parallel networks for high end usersReduces costs by providing high end network capability to only those who need itLimitations of campus firewall and border router are eliminatedMany issues in regards to security, back door routing, etcCampus networks may follow same evolution as campus computingDiscipline specific networks being extended into the campus

UCLP intended for projects like National LambdaRail

GEANT2 POP Design

EMBED Word.Picture.8

_1147001627.doc

Nx10Gbps to other GANT2 PoP

2x10Gbps to local NREN

Dark fibre to other GANT2 PoP

GANT2 PoP

Juniper M-160

DWDM

UltraLight Optical Exchange PointL1, L2 and L3 servicesInterfaces1GE and 10GE10GE WAN-PHY (SONET friendly)Hybrid packet- and circuit-switched PoPInterface between packet- & circuit-switched networksControl plane is L3Photonic switchCalient or Glimmerglass Photonic Cross Connect Switch

LHC Data Grid HierarchyTier 1Online System CERN 700k SI95 ~1 PB Disk; Tape RobotFNAL: 200k SI95; 600 TBIN2P3 Center INFN Center RAL Center InstituteInstituteInstituteInstitute ~0.25TIPSWorkstations~100-400 MBytes/sec2.5/10 Gbps0.11 GbpsPhysicists work on analysis channelsEach institute has ~10 physicists working on one or more channelsPhysics data cache~PByte/sec10 Gbps~2.5 GbpsTier 0 +1Tier 3Tier 4Tier 2ExperimentCERN/Outside Resource Ratio ~1:2 Tier0/( Tier1)/( Tier2) ~1:1:1

Deploying the LHC [email protected] LHC Computing Centre

What you [email protected]

Main Networking Challenges

Fulfill the, yet unproven, assertion that the network can be nearly transparent to the GridDeploy suitable Wide Area Network infrastructure (50-100 Gb/s)Deploy suitable Local Area Network infrastructure (matching or exceeding that of the WAN)Seamless interconnection of LAN & WAN infrastructuresfirewall?End to End issues (transport protocols, PCs (Itanium, Xeon), 10GigE NICs (Intel, S2io), where are we today:memory to memory: 7.5Gb/s (PCI bus limit)memory to disk: 1.2MB (Windows 2003 server/NewiSys)disk to disk: 400MB (Linux), 600MB (Windows)

Main TCP issuesDoes not scale to some environmentsHigh speed, high latencyNoisyUnfair behaviour with respect to:Round Trip Time (RTTFrame size (MSS)Access BandwidthWidespread use of multiple streams in order to compensate for inherent TCP/IP limitations (e.g. Gridftp, BBftp):Bandage rather than a cureNew TCP/IP proposals in order to restore performance in single stream environmentsNot clear if/when it will have a real impactIn the mean time there is an absolute requirement for backbones with:Zero packet losses,And no packet re-orderingWhich re-inforces the case for lambda Grids

TCP dynamics(10Gbps, 100ms RTT, 1500Bytes packets)Window size (W) = Bandwidth*Round Trip TimeWbits = 10Gbps*100ms = 1GbWpackets = 1Gb/(8*1500) = 83333 packetsStandard Additive Increase Multiplicative Decrease (AIMD) mechanisms:W=W/2 (halving the congestion window on loss event)W=W + 1 (increasing congestion window by one packet every RTT)Time to recover from W/2 to W (congestion avoidance) at 1 packet per RTT:RTT*Wp/2 = 1.157 hourIn practice, 1 packet per 2 RTT because of delayed acks, i.e. 2.31 hourPackets per second:RTT*Wpackets = 833333 packets

Single TCP stream performance under periodic lossesTCP throughput much more sensitive to packet loss in WANs than LANsTCPs congestion control algorithm (AIMD) is not well-suited to gigabit networksThe effect of packets loss can be disastrousTCP is inefficient in high bandwidth*delay networksThe future performance-outlook for computational grids looks bad if we continue to rely solely on the widely-deployed TCP RENOLoss rate =0.01%:LAN BW utilization= 99%WAN BW utilization=1.2%Bandwidth available = 1 Gbps

Chart7

1.265957446822.8723404255

4.031914893635.4255319149

12.765957446864.6808510638

19.893617021377.7659574468

27.872340425597.4468085106

34.468085106497.9787234043

39.468085106498.4042553191

53.617021276698.5106382979

0.00000498.9361702128

0.000003333399.5744680851

0.00000299.6808510638

0.000001666799.7872340426

0.0000013333100

WAN (RTT=120ms)

LAN (RTT=0.04 ms)

Packet Loss frequency (%)

Bandwidth Utilization (%)

Effect of packet loss

CHI

1/PBW (MbpsBW (Mbpsbw utilsation50900150200250300350400450500550600650

0

10100.26781249780.37555315790.0023360.0001297778215

10010.8468974791.18760336130.023360.0012977778333

10000.12.67812497833.75553157870.23360.0129777778608

100000.018.468974789911.876033613411.91.26595744682.3360.1297777778731

1000000.00126.781249782737.555315787237.94.031914893623.361.2977777778

10000000.000184.6897478992118.760336134512012.7659574468233.612.9777777778570776.255707763428082.191780822916

25000000.00004133.9062489134187.77657893618719.893617021358432.4444444444921

50000000.00002189.3720332998265.556184627426227.8723404255116864.8888888889925

75000000.0000133333231.932426569325.238575188732434.4680851064175297.3333333333926

100000000.00001267.8124978268375.553157872137139.46808510642336129.7777777778930

175000000.0000057143354.2826336224496.810129907250453.61702127664088227.1111111111936

250000000.000004423.4487394958593.80168067235840324.4444444444937

300000000.0000033333463.864853138650.47715037747008389.3333333333938

500000000.000002598.8469503648839.762390166711680648.8888888889940

600000000.0000016667656.0039664159919.913608077514016778.6666666667940

750000000.0000013333733.43473120771028.494680544217520973.3333333333940

1000000000.000001846.89747899161187.6033613445233601297.7777777778

10000000000.00000012678.1249782683755.531578720723360012977.7777777778

CHI

00215

00333

00608

00731

00916

00921

00925

00926

00930

00936

00937

00938

00940

00940

00940

00940

00940

00940

MSS*C/ (RTT*sqrt(p))

WAN (RTT=120ms)

LAN (RTT=0.04 ms)

Packet Loss frequency (one packet every x packets transmitted)

Throuhput (Mbps)


Gva

022.8723404255

035.4255319149

064.6808510638

077.7659574468

097.4468085106

097.9787234043

098.4042553191

098.5106382979

098.9361702128

099.5744680851

099.6808510638

099.7872340426

0100

0100

0100

WAN (RTT=120ms)

LAN (RTT=0.04 ms)

Packet Loss frequency (one packet every x packets transmitted)

Bandwidth Utilisation %


Sheet3

00215

00333

00608

00731

00916

00921

00925

00926

00930

00936

00937

00938

00940

00940

00940

940

940

940

MSS*C/ (RTT*sqrt(p))

WAN (RTT=120ms)

LAN (RTT=0.04 ms)


Throuhput (Mbps)


022.8723404255

035.4255319149

064.6808510638

077.7659574468

097.4468085106

097.9787234043

098.4042553191

098.5106382979

098.9361702128

099.5744680851

099.6808510638

099.7872340426

0100

WAN (RTT=120ms)

LAN (RTT=0.04 ms)


Bandwidth Utilisation (%)


w05gva -> w06gva

w=1M

RTT=

txqueuelen = 100

Loss rate (%)Rate (Mbit/s)Link Utilisation (%)

10101021522.87234042550.0005432558

205533335.42553191490.0007015015

502260864.68085106380.0009605263

1001173177.76595744680.0015978112

5000.20.291697.44680851060.0063755459

7500.13333333330.133333333392197.97872340430.0095114007

10000.10.192598.40425531910.012627027

50000.020.0292698.51063829790.0630669546

100000.010.0193098.93617021280.1255913978

1000000.0010.00193699.57446808511.2478632479939

10000000.00010.000193799.680851063812.4653148346

25000000.000040.0000493899.787234042631.1300639659

50000000.000020.0000294010062.1276595745

75000000.00001333330.000013333394010093.1914893617

100000000.000010.00001940100124.2553191489

175000000.00000571430.0000057143940100217.4468085106

250000000.0000040.000004940100310.6382978723

300000000.00000333330.0000033333940100372.7659574468

09

ResponsivenessLarge MTU accelerates the growth of the windowTime to recover from a packet loss decreases with large MTULarger MTU reduces overhead per frames (saves CPU cycles, reduces the number of packets) r =C . RTT2 . MSS2C : Capacity of the linkTime to recover from a single packet loss:

Internet2 land speed record history (IPv4 & IPv6) period 2000-2004

Chart1

MonthMonthMonthMonth

0.760.956Mar-00Mar-00

0.4020.402Apr-02Apr-02

Sep-02Sep-020.4830.483

Oct-02Oct-020.3480.348

0.9230.923Nov-02Nov-02

2.382.38Feb-03Feb-03

May-03May-030.9830.983

5.445.44Oct-03Oct-03

5.645.64Nov-03Nov-03

Nov-03Nov-0344

Feb-046.25Feb-04Feb-04

4.2226Apr-04Apr-04Apr-04

May-047.09May-04May-04

IPv4 (Gb/s) single stream

IPv4 (Gb/s) multiple streams



DataTAG-Web-Hits

4241

4910

3889

6949

8788

17356

18616

20420

18851

37266

36076

Hits/month

Month

Hits

DataTAG Web site

DataTAG-Web-Hits-v2

Month

4241

4910

3889

6949

8788

17356

18616

20420

18851

37266

36076

31379

25998

23407

26423

28340

22866

27250

40518

29530

28480

30335

58530

79073

DataTAG

Month

Hits

IPv4-IPv6-Gbs



Month

Internet2 landspeed record history(in Gigabit/second)

IPv4-IPv6-Tbms

MonthMonth

4278Mar-00

4933Apr-02

Sep-021215

Oct-025154

10136Nov-02

23886Feb-03

May-036947

38420Oct-03

61752Nov-03

Nov-0346156

IPv4 terabit-meters/second)

IPv6 (terabit-meters/second)

Month

Internet2 landspeed record history(in terabit-meters/second)

IP-v4-IPv6-Gbs-v1



Month

Evolution of the I2LSR in Gigabit/second

IPv4-IPv6-Tbms-v1

MonthMonth

4278Mar-00

4933Apr-02

Sep-021215

Oct-025154

10136Nov-02

23886Feb-03

May-036947

38420Oct-03

61752Nov-03

Nov-0346156



Month

Evolution of the I2LSR using terabit-meters/second metrics

IPv4-v6-Tbms

MonthMonthMonthMonthMonthMonth

Mar-00Mar-00Mar-00Mar-004278Mar-00

Apr-02Apr-02Apr-02Apr-024933Apr-02

Sep-02Sep-02Sep-02Sep-02Sep-021215

Oct-02Oct-02Oct-02Oct-02Oct-025154

Nov-02Nov-02Nov-02Nov-0210136Nov-02

Feb-03Feb-03Feb-03Feb-0323886Feb-03

May-03May-03May-03May-03May-036947













MONTH

TERABIT-METERS/SECOND

EVOLUTION OF THE I2LSR IN TERABITS-METERS/SECOND

IPv4-v6-Gbs


0.760.956Mar-00Mar-00

0.4020.402Apr-02Apr-02

Sep-02Sep-020.4830.483

Oct-02Oct-020.3480.348

0.9230.923Nov-02Nov-02

2.382.38Feb-03Feb-03

May-03May-030.9830.983

5.445.44Oct-03Oct-03

5.645.64Nov-03Nov-03

Nov-03Nov-0344


4.2226Apr-04Apr-04Apr-04






MONTH

Gigabits/second

Category

EVOLUTION OF THE I2LSR IN GIGABITS/SECOND

IPv4-IPv6

IPv4 (Gb/s) single streamIPv4 (Gb/s) multiple streamsIPv6 (Gb/s) single streamIPv6 (Gb/s) multiple streamsIPv4 terabit-meters/second)IPv6 (terabit-meters/second)Owner(s)

Month

Mar-000.7600.9564,278Microsoft, Qwest, University of Washington, USC ISI

Apr-020.4020.4024,933University of Alberta, University of Amsterdam (UvA), Surfnet

Sep-020.4830.4831,215ARNES, DANTE, RedIris

Oct-020.3480.3485,154ARNES, DANTE, RedIris

Nov-020.9230.92310,136Caltech, DataTAG, Nikhef, SLAC, UvA

Feb-032.382.3823,886Caltech, CERN, DataTAG, Los Alamos National Laboratory (LANL), SLAC

May-030.9830.9836,947Caltech, CERN, DataTAG

Oct-035.445.4438,420Caltech, CERN, DataTAG

Nov-035.645.6461,752Caltech, CERN, DataTAG

Nov-034446,156Caltech, CERN, DataTAG

Feb-046.2568,431Caltech, CERN, DataTAG (multi-stream)

Apr-044.222669,073SUNET

May-047.0977,699Caltech, CERN, DataTAG (multi-stream)

IPv4-IPv6





Datag related Web sites

MonthMonthMonth

4241Apr-02Apr-02

4910May-02May-02

3889Jun-02Jun-02

6949Jul-02Jul-02

8788Aug-02Aug-02

17356Sep-02Sep-02

18616Oct-02Oct-02

20420Nov-02Nov-02

18851Dec-02Dec-02

37266Jan-03Jan-03

36076Feb-03Feb-03

31379Mar-03Mar-03

25998Apr-03Apr-03

23407May-03May-03

26423Jun-03Jun-03

28340Jul-03Jul-03

22866Aug-03Aug-03

27250Sep-03Sep-03

405181058Oct-03

29530297Nov-03

2848037916569

303353351968

58530306915

DataTAG

Telecom

ICT4D

Month

Hits/month

DataTAG related Web sites

DataTAG Web site activity

Month

4241

4910

3889

6949

8788

17356

18616

20420

18851

37266

36076

31379

25998

23407

26423

28340

22866

27250

40518

29530

28480

30335

58530

79073

DataTAG

Month

Hits/month

DataTAG Web site

DataTAG Tables

DataTAGGridcafeTelecomICT4D

Month

Apr-024241

May-024910

Jun-023889

Jul-026949

Aug-028788

Sep-0217356

Oct-0218616

Nov-0220420

Dec-0218851

Jan-0337266

Feb-0336076

Mar-0331379

Apr-0325998

May-0323407

Jun-0326423

Jul-0328340

Aug-0322866

Sep-0327250

Oct-03405181716761058

Nov-0329530379547297

Dec-032848031876437916569

Jan-04303352700003351968

Feb-0458530291255306915

Mar-04790732699846001339

Apr-0446165497284332818

May-04409906141719-May

Jun-04

Sheet1

year16002000250030000.800.710.630.5

19984.05.06.08.0400.00400.00400.00400.00

199920.025.031.038.080.0080.0080.0080.00

200044.455.669.483.336.0036.0036.0036.00

2001160.0200.0250.0300.010.0010.0010.0010.00Based on actual 155Mbps pri


20031265.81582.31977.82373.41.261.121.000.79

20041582.31977.82472.32966.81.010.800.630.40

20051977.82472.33090.43708.50.810.570.400.20

20062472.33090.43863.04635.60.650.400.250.10

20073090.43863.04828.75794.50.520.290.160.05

20083863.04828.76035.97243.10.410.200.100.02

year1600.02000.02500.03000.0

19984.05.06.08.0

199920.025.031.038.0

200044.455.669.483.3

2001160.0200.0250.0300.0

20021012.71265.81582.31898.7

20031426.31782.82228.62674.3

20042008.82511.13138.83766.6

20052829.43536.74420.95305.0

20063985.04981.36226.67471.9

20075612.77015.98769.810523.8

20087905.29881.512351.914822.2

year1600.02000.02500.03000.0

19984.05.06.08.0

199920.025.031.038.0

200044.455.669.483.3

2001160.0200.0250.0300.0

20021012.71265.81582.31898.7

20031607.42009.22511.63013.9

20042551.43189.33986.64783.9

20054049.95062.36327.97593.5

20066428.48035.510044.312053.2

200710203.812754.715943.419132.0

200816196.420245.625306.930368.3

year1600200025003000

19984.05.06.08.0

199920.025.031.038.0

200044.455.669.483.3

2001160.0200.0250.0300.0

20021012.71265.81582.31898.7

20032025.32531.63164.63797.5

20044050.65063.36329.17594.9

20058101.310126.612658.215189.9

200616202.520253.225316.530379.7

200732405.140506.350632.960759.5

200864810.181012.7101265.8121519.0

622 Kb[s-0.20-0.29-0.37-0.50Hypothesis 622Mbps=2*155Mbps

20013100.003100.003100.003100.00

2002982.76982.76982.76982.76

2003786.21697.76619.14491.38

2004628.97495.41390.06245.69

2005503.17351.74245.74122.85

2006402.54249.74154.8161.42

2007322.03177.3197.5330.71

2.5 Gb[s-0.20-0.29-0.33-0.50Hypothesis 2.5Gbps=2*622Mbps

20016200.006200.006200.006200.00

20021965.521965.521965.521965.52

20031572.421395.521238.28982.76

20041257.93990.82780.11491.38

20051006.35703.48491.47245.69

2006805.08499.47309.63122.85

2007644.06354.62195.0761.42

1 Gb[s-0.20-0.29-0.33-0.50EuropeUSA

200110000.0010000.0010000.0010000.00

20021580.001580.001580.001580.00

20031264.001121.80995.40790.00

20041011.20796.48627.10395.00376.26156.78206.9414.4966992465

2005808.96565.50395.07197.50237.0498.77130.3753.6914786909

2006647.17401.50248.9098.75149.3462.2282.1466.9621616327

2007517.73285.07156.8049.3894.0839.2051.7585.0313163589

2008414.19202.4098.7924.6959.2724.7032.60104.2952653753

Sheet2

0.800.710.630.5

400.00400.00400.00400.001998

80.0080.0080.0080.001999

36.0036.0036.0036.002000

10.0010.0010.0010.002001

8.007.106.305.002002

6.405.043.972.502003

5.123.582.501.252004

4.102.541.580.632005

3.281.800.990.312006

2.621.280.630.162007

22522522522520012250

36340846058020022900

578734932148020033700

87912571800360020044500

DataTAG-Web-Hits

4241

4910

3889

6949

8788

17356

18616

20420

18851

37266

36076

Hits/month

Month

Hits

DataTAG Web site

DataTAG-Web-Hits-v2

Month

4241

4910

3889

6949

8788

17356

18616

20420

18851

37266

36076

31379

25998

23407

26423

28340

22866

27250

40518

29530

28480

30335

58530

79073

DataTAG

Month

Hits

IPv4-IPv6-Gbs



Month

Internet2 landspeed record history(in Gigabit/second)

IPv4-IPv6-Tbms

MonthMonth

4278Mar-00

4933Apr-02

Sep-021215

Oct-025154

10136Nov-02

23886Feb-03

May-036947

38420Oct-03

61752Nov-03

Nov-0346156



Month

Internet2 landspeed record history(in terabit-meters/second)

IP-v4-IPv6-Gbs-v1



Month

Evolution of the I2LSR in Gigabit/second

IPv4-IPv6-Tbms-v1

MonthMonth

4278Mar-00

4933Apr-02

Sep-021215

Oct-025154

10136Nov-02

23886Feb-03

May-036947

38420Oct-03

61752Nov-03

Nov-0346156



Month

Evolution of the I2LSR using terabit-meters/second metrics

IPv4-v6-Tbms

MonthMonthMonthMonthMonthMonth

Mar-00Mar-00Mar-00Mar-004278Mar-00


Sep-02Sep-02Sep-02Sep-02Sep-021215

















MONTH

TERABIT-METERS/SECOND

EVOLUTION OF THE I2LSR IN TERABITS-METERS/SECOND

IPv4-v6-Gbs


0.760.956Mar-00Mar-00

0.4020.402Apr-02Apr-02

Sep-02Sep-020.4830.483

Oct-02Oct-020.3480.348

0.9230.923Nov-02Nov-02

2.382.38Feb-03Feb-03

May-03May-030.9830.983

5.445.44Oct-03Oct-03

5.645.64Nov-03Nov-03

Nov-03Nov-0344


4.2226Apr-04Apr-04Apr-04






MONTH

Gigabits/second

Category

EVOLUTION OF THE I2LSR IN GIGABITS/SECOND

Chart3

IPv4 (Gb/s) single stream0.760.402IPv4 (Gb/s) single streamIPv4 (Gb/s) single stream0.9232.38IPv4 (Gb/s) single stream5.445.64IPv4 (Gb/s) single streamIPv4 (Gb/s) single stream4.2226IPv4 (Gb/s) single stream

IPv4 (Gb/s) multiple streams0.9560.402IPv4 (Gb/s) multiple streamsIPv4 (Gb/s) multiple streams0.9232.38IPv4 (Gb/s) multiple streams5.445.64IPv4 (Gb/s) multiple streams6.25IPv4 (Gb/s) multiple streams7.09

IPv6 (Gb/s) single streamIPv6 (Gb/s) single streamIPv6 (Gb/s) single stream0.4830.348IPv6 (Gb/s) single streamIPv6 (Gb/s) single stream0.983IPv6 (Gb/s) single streamIPv6 (Gb/s) single stream4IPv6 (Gb/s) single streamIPv6 (Gb/s) single streamIPv6 (Gb/s) single stream

IPv6 (Gb/s) multiple streamsIPv6 (Gb/s) multiple streamsIPv6 (Gb/s) multiple streams0.4830.348IPv6 (Gb/s) multiple streamsIPv6 (Gb/s) multiple streams0.983IPv6 (Gb/s) multiple streamsIPv6 (Gb/s) multiple streams4IPv6 (Gb/s) multiple streamsIPv6 (Gb/s) multiple streamsIPv6 (Gb/s) multiple streams

Month

Mar-00

Apr-02

Sep-02

Oct-02

Nov-02

Feb-03

May-03

Oct-03

Nov-03

Nov-03

Feb-04

Apr-04

May-04

Type

Gigabit/second

Month

Evolution of Internet2 Landspeed record

IPv4-IPv6

IPv4 (Gb/s) single streamIPv4 (Gb/s) multiple streamsIPv6 (Gb/s) single streamIPv6 (Gb/s) multiple streamsIPv4 terabit-meters/second)IPv6 (terabit-meters/second)Owner(s)

Month

Mar-000.7600.9564,278Microsoft, Qwest, University of Washington, USC ISI

Apr-020.4020.4024,933University of Alberta, University of Amsterdam (UvA), Surfnet

Sep-020.4830.4831,215ARNES, DANTE, RedIris

Oct-020.3480.3485,154ARNES, DANTE, RedIris

Nov-020.9230.92310,136Caltech, DataTAG, Nikhef, SLAC, UvA

Feb-032.382.3823,886Caltech, CERN, DataTAG, Los Alamos National Laboratory (LANL), SLAC

May-030.9830.9836,947Caltech, CERN, DataTAG

Oct-035.445.4438,420Caltech, CERN, DataTAG

Nov-035.645.6461,752Caltech, CERN, DataTAG

Nov-034446,156Caltech, CERN, DataTAG

Feb-046.2568,431Caltech, CERN, DataTAG (multi-stream)

Apr-044.222669,073SUNET

May-047.0977,699Caltech, CERN, DataTAG (multi-stream)

Datag related Web sites

MonthMonthMonth

4241Apr-02Apr-02

4910May-02May-02

3889Jun-02Jun-02

6949Jul-02Jul-02

8788Aug-02Aug-02

17356Sep-02Sep-02

18616Oct-02Oct-02

20420Nov-02Nov-02

18851Dec-02Dec-02

37266Jan-03Jan-03

36076Feb-03Feb-03

31379Mar-03Mar-03

25998Apr-03Apr-03

23407May-03May-03

26423Jun-03Jun-03

28340Jul-03Jul-03

22866Aug-03Aug-03

27250Sep-03Sep-03

405181058Oct-03

29530297Nov-03

2848037916569

303353351968

58530306915

DataTAG

Telecom

ICT4D

Month

Hits/month

DataTAG related Web sites

DataTAG Web site activity

Month

4241

4910

3889

6949

8788

17356

18616

20420

18851

37266

36076

31379

25998

23407

26423

28340

22866

27250

40518

29530

28480

30335

58530

79073

DataTAG

Month

Hits/month

DataTAG Web site

DataTAG Tables

DataTAGGridcafeTelecomICT4D

Month

Apr-024241

May-024910

Jun-023889

Jul-026949

Aug-028788

Sep-0217356

Oct-0218616

Nov-0220420

Dec-0218851

Jan-0337266

Feb-0336076

Mar-0331379

Apr-0325998

May-0323407

Jun-0326423

Jul-0328340

Aug-0322866

Sep-0327250

Oct-03405181716761058

Nov-0329530379547297

Dec-032848031876437916569

Jan-04303352700003351968

Feb-0458530291255306915

Mar-04790732699846001339

Apr-0446165497284332818

May-04409906141719-May

Jun-04

Sheet1

year16002000250030000.800.710.630.5

19984.05.06.08.0400.00400.00400.00400.00

199920.025.031.038.080.0080.0080.0080.00

200044.455.669.483.336.0036.0036.0036.00



20031265.81582.31977.82373.41.261.121.000.79

20041582.31977.82472.32966.81.010.800.630.40

20051977.82472.33090.43708.50.810.570.400.20

20062472.33090.43863.04635.60.650.400.250.10

20073090.43863.04828.75794.50.520.290.160.05

20083863.04828.76035.97243.10.410.200.100.02

year1600.02000.02500.03000.0

19984.05.06.08.0

199920.025.031.038.0

200044.455.669.483.3

2001160.0200.0250.0300.0

20021012.71265.81582.31898.7

20031426.31782.82228.62674.3

20042008.82511.13138.83766.6

20052829.43536.74420.95305.0

20063985.04981.36226.67471.9

20075612.77015.98769.810523.8

20087905.29881.512351.914822.2

year1600.02000.02500.03000.0

19984.05.06.08.0

199920.025.031.038.0

200044.455.669.483.3

2001160.0200.0250.0300.0

20021012.71265.81582.31898.7

20031607.42009.22511.63013.9

20042551.43189.33986.64783.9

20054049.95062.36327.97593.5

20066428.48035.510044.312053.2

200710203.812754.715943.419132.0

200816196.420245.625306.930368.3

year1600200025003000

19984.05.06.08.0

199920.025.031.038.0

200044.455.669.483.3

2001160.0200.0250.0300.0

20021012.71265.81582.31898.7

20032025.32531.63164.63797.5

20044050.65063.36329.17594.9

20058101.310126.612658.215189.9

200616202.520253.225316.530379.7

200732405.140506.350632.960759.5

200864810.181012.7101265.8121519.0

622 Kb[s-0.20-0.29-0.37-0.50Hypothesis 622Mbps=2*155Mbps

20013100.003100.003100.003100.00

2002982.76982.76982.76982.76

2003786.21697.76619.14491.38

2004628.97495.41390.06245.69

2005503.17351.74245.74122.85

2006402.54249.74154.8161.42

2007322.03177.3197.5330.71

2.5 Gb[s-0.20-0.29-0.33-0.50Hypothesis 2.5Gbps=2*622Mbps

20016200.006200.006200.006200.00

20021965.521965.521965.521965.52

20031572.421395.521238.28982.76

20041257.93990.82780.11491.38

20051006.35703.48491.47245.69

2006805.08499.47309.63122.85

2007644.06354.62195.0761.42

1 Gb[s-0.20-0.29-0.33-0.50EuropeUSA

200110000.0010000.0010000.0010000.00

20021580.001580.001580.001580.00

20031264.001121.80995.40790.00

20041011.20796.48627.10395.00376.26156.78206.9414.4966992465

2005808.96565.50395.07197.50237.0498.77130.3753.6914786909

2006647.17401.50248.9098.75149.3462.2282.1466.9621616327

2007517.73285.07156.8049.3894.0839.2051.7585.0313163589

2008414.19202.4098.7924.6959.2724.7032.60104.2952653753

Sheet2

0.800.710.630.5

400.00400.00400.00400.001998

80.0080.0080.0080.001999

36.0036.0036.0036.002000

10.0010.0010.0010.002001

8.007.106.305.002002

6.405.043.972.502003

5.123.582.501.252004

4.102.541.580.632005

3.281.800.990.312006

2.621.280.630.162007

22522522522520012250

36340846058020022900

578734932148020033700

87912571800360020044500

Layer1/2/3 networking (1) Conventional layer 3 technology is no longer fashionable because of:High associated costs, e.g. 200/300 KUSD for a 10G router interfaces Implied use of shared backbonesThe use of layer 1 or layer 2 technology is very attractive because it helps to solve a number of problems, e.g. 1500 bytes Ethernet frame size (layer1)Protocol transparency (layer1 & layer2)Minimum functionality hence, in theory, much lower costs (layer1&2)

Layer1/2/3 networking (2) 0n-demand Lambda Grids are becoming very popular: Pros: circuit oriented model like the telephone network, hence no need for complex transport protocols Lower equipment costs (i.e. in theory a factor 2 or 3 per layer)the concept of a dedicated end to end light path is very elegant Cons: End to end still very loosely defined, i.e. site to site, cluster to cluster or really host to hostHigher circuit costs, Scalability, Additional middleware to deal with circuit set up/tear down, etcExtending dynamic VLAN functionality is a potential nightmare!

Lambda Grids What does it mean?

Clearly different things to different people, hence the apparently easy consensus!Conservatively, on demand site to site connectivityWhere is the innovation?What does it solve in terms of transport protocols?Where are the savings?Less interfaces needed (customer) but more standby/idle circuits needed (provider)Economics from the service provider vs the customer perspective?Traditionally, switched services have been very expensive, Usage vs flat chargeBreak even, switches vs leased, few hours/dayWhy would this change? In case there are no savings, why bother?More advanced, cluster to clusterImplies even more active circuits in paralleIs it realistic?Even more advanced, Host to Host or even per flowAll opticalIs it really realisitic?

Some ChallengesReal bandwidth estimates given the chaotic nature of the requirements.End-end performance given the whole chain involved (disk-bus-memory-bus-network-bus-memory-bus-disk)Provisioning over complex network infrastructures (GEANT, NRENs etc)Cost model for options (packet+SLAs, circuit switched etc)Consistent Performance (dealing with firewalls)Merging leading edge research with production networking

Tentative conclusions

There is a very clear trend towards community managed dark fiber networksAs a consequence National Research & Education Networks are evolving into Telecom Operators, is it right?In the short term, almost certainly YESIn the longer term, probably NOIn many countries, there is NO other way to have affordable access to multi-Gbit/s networks, therefore this is clearly the right moveThe Grid & its associated Wide Area Networking challengeson-demand Lambda Grids are, according to me, extremely doubtful!Ethernet over SONET & new standards will revolutionize the InternetWAN-PHY (IEEE) has, according to me NO future!However, GFP, VCAT/LCAS, G.709, OTN are very likely to have a very bright future.

Single TCP stream between Caltech and CERNAvailable (PCI-X) Bandwidth=8.5 Gbps RTT=250ms (16000 km) 9000 Byte MTU15 min to increase throughput from 3 to 6 Gbps

Sending station: Tyan S2882 motherboard, 2x Opteron 2.4 GHz , 2 GB DDR.Receiving station: CERN OpenLab:HP rx4640, 4x 1.5GHz Itanium-2, zx1 chipset, 8GB memoryNetwork adapter: S2IO 10 GbE

Burst of packet lossesSingle packet lossCPU load = 100%

High Throughput Disk to Disk Transfers: From 0.1 to 1GByte/sec Server Hardware (Rather than Network) Bottlenecks:Write/read and transmit tasks share the same limited resources: CPU, PCI-X bus, memory, IO chipset PCI-X bus bandwidth: 8.5 Gbps [133MHz x 64 bit]Link aggregation (802.3ad): Logical interface with two physical interfaces on two independent PCI-X buses.LAN test: 11.1 Gbps (memory to memory)

Performance in this range (from 100 MByte/sec up to 1 GByte/sec) is required to build a responsive Grid-based Processing and Analysis System for LHC

Transferring a TB from Caltech to CERN in 64-bit MS Windows Latest disk to disk over 10Gbps WAN: 4.3 Gbits/sec (536 MB/sec) - 8 TCP streams from CERN to Caltech; 1TB file 3 Supermicro Marvell SATA disk controllers + 24 SATA 7200rpm SATA disksLocal Disk IO 9.6 Gbits/sec (1.2 GBytes/sec read/write, with

UltraLight: Developing Advanced Network Services for Data Intensive HEP ApplicationsUltraLight: a next-generation hybrid packet- and circuit-switched network infrastructurePacket switched: cost effective solution; requires ultrascale protocols to share 10G efficiently and fairlyCircuit-switched: Scheduled or sudden overflow demands handled by provisioning additional wavelengths; Use path diversity, e.g. across the US, Atlantic, Canada,Extend and augment existing grid computing infrastructures (currently focused on CPU/storage) to include the network as an integral componentUsing MonALISA to monitor and manage global systemsPartners: Caltech, UF, FIU, UMich, SLAC, FNAL, MIT/Haystack; CERN, NLR, CENIC, Internet2; Translight, UKLight, Netherlight; UvA, UCL, KEK, TaiwanStrong support from Cisco

UltraLight MPLS NetworkCompute path from one given node to another such that the path does not violate any constraints (bandwidth/administrative requirements)Ability to set the path the traffic will take through the network (with simple configuration, management, and provisioning mechanisms)Take advantage of the multiplicity of waves/L2 channels across the US (NLR, HOPI, Ultranet and Abilene/ESnet MPLS services) EoMPLS will be used to build layer2 pathsnatural step toward the deployment of GMPLS

SummaryFor many years the Wide Area Network has been the bottleneck; this is no longer the case in many countries thus making deployment of a data intensive Grid infrastructure possible!Recent I2LSR records show for the first time ever that the network can be truly transparent and that throughputs are limited by the end hostsChallenge shifted from getting adequate bandwidth to deploying adequate infrastructure to make effective use of it!Some transport protocol issues still need to be resolved; however there are many encouraging signs that practical solutions may now be in sight.1GByte/sec disk to disk challenge. Today: 1 TB at 536 MB/sec from CERN to Caltech Still in Early Stages; Expect Substantial ImprovementsNext generation network and Grid system: UltraLightDeliver the critical missing component for future eScience: the integrated, managed network Extend and augment existing grid computing infrastructures (currently focused on CPU/storage) to include the network as an integral component.

10G DataTAG testbed extension to Telecom World 2003 and Abilene/CenicOn September 15, 2003, the DataTAG project was the first transatlantic testbed offering direct 10GigE access using Junipers VPN layer2/10GigE emulation.

And some statistics about the capacity of networks with Ethernet and the Internet backbone compared.The green line shows specially the increase in available bandwidth with DWDM.It shows at the same time that xxGE slower in time will follow backbone bandwidth.

Both statistics show clearly that with the old Ethernet technology building frames in the OS is not anymore possible at this level and something has to be done to offload processor or hub internals with clever hardware oriented algorithms. Stabilizing and improving HW/SW drivers for production useE.g. Reducing extra memory copies; interrupt timing moderation

Olivier Martin, CERN NEC’2005 Conference, VARNA (Bulgaria)

Documents

Transcript of Olivier Martin, CERN NEC’2005 Conference, VARNA (Bulgaria)