ICFA Standing Committee on ICFA Standing Committee on Interregional Connectivity (SCIC) Interregional Connectivity (SCIC)

Harvey B. NewmanHarvey B. Newman California Institute of Technology California Institute of Technology

ICFA Seminar, CERNICFA Seminar, CERNOctober 9, 2002October 9, 2002

IIIICCCCFFFFAAAA aaaannnndddd GGGGlllloooobbbbaaaallll NNNNeeeettttwwwwoooorrrrkkkkssss ffffoooorrrrHHHHEEEENNNNPPPP

IIIICCCCFFFFAAAA aaaannnndddd GGGGlllloooobbbbaaaallll NNNNeeeettttwwwwoooorrrrkkkkssss ffffoooorrrrIIIICCCCFFFFAAAA aaaannnndddd GGGGlllloooobbbbaaaallll NNNNeeeettttwwwwoooorrrrkkkkssss ffffoooorrrrHHHHEEEENNNNPPPPHHHHEEEENNNNPPPP

��National and International Networks, with sufficientNational and International Networks, with sufficientcapacity and capability, are essential forcapacity and capability, are essential for➨➨The daily conduct of collaborative work in bothThe daily conduct of collaborative work in both

experiment and theory experiment and theory➨➨Detector development, construction on a global scale;Detector development, construction on a global scale;

Data analysis involving physicists from all world regionsData analysis involving physicists from all world regions➨➨The conception, design and implementation ofThe conception, design and implementation of

next generation facilities as next generation facilities as ““global networksglobal networks””➨➨The formation of worldwide collaborationsThe formation of worldwide collaborations

�� ““Collaborations on this scale would never haveCollaborations on this scale would never havebeen attempted, if they could not rely on excellentbeen attempted, if they could not rely on excellentnetworksnetworks””

ICFA and International NetworkingICFA and International Networking

�� ICFA Statement on Communications in IntICFA Statement on Communications in Int’’l HEPl HEPCollaborations of October 17, 1996Collaborations of October 17, 1996 See See http://http://www.fnal.gov/directorate/icfa/icfa_communicaes.htmlwww.fnal.gov/directorate/icfa/icfa_communicaes.html

““ICFA urges that all countries and institutions wishingICFA urges that all countries and institutions wishingto participate even more effectively and fully into participate even more effectively and fully ininternational HEP Collaborations should:international HEP Collaborations should:�� Review their operating methods to ensure theyReview their operating methods to ensure they

are fully adapted to remote participation are fully adapted to remote participation

�� Strive to provide the necessary communications Strive to provide the necessary communications facilities and adequate international bandwidth facilities and adequate international bandwidth””

ICFA Network Task Force: 1998Bandwidth Requirements Projection

(Mbps)

ICFA Network Task Force: 1998Bandwidth Requirements Projection

(Mbps)1998 2000 2005

BW Utilized Per Physicist(and Peak BW Used)

0.05 - 0.25(0.5 - 2)

0.2 – 2(2-10)

0.8 – 10(10 – 100)

BW Utilized by a UniversityGroup

0.25 - 10 1.5 - 45 34 - 622

BW to a Home Laboratory OrRegional Center

1.5 - 45 34 - 155 622 - 5000

BW to a Central LaboratoryHousing One or More MajorExperiments

34 - 155 155 - 622 2500 - 10000

BW on a Transoceanic Link 1.5 - 20 34 - 155 622 - 5000

100100––1000 X Bandwidth Increase Foreseen for 1998-20051000 X Bandwidth Increase Foreseen for 1998-2005See the ICFA-NTF Requirements Report:See the ICFA-NTF Requirements Report:

http://l3www.cern.ch/~newman/icfareq98.htmlhttp://l3www.cern.ch/~newman/icfareq98.html

NTF

ICFA Standing Committee onInterregional Connectivity (SCIC)

ICFA Standing Committee onInterregional Connectivity (SCIC)

�� Created by ICFA in July 1998 in Vancouver ; Following ICFA-NTFCreated by ICFA in July 1998 in Vancouver ; Following ICFA-NTF�� CHARGE:CHARGE:

Make recommendations to ICFA concerning theMake recommendations to ICFA concerning theconnectivity between connectivity between the Americasthe Americas, Asia and Europe, Asia and Europe

➨➨ As part of the process of developing theseAs part of the process of developing theserecommendations, the committee shouldrecommendations, the committee should�� Monitor trafficMonitor traffic�� Keep track of technology developmentsKeep track of technology developments�� Periodically review forecasts of futurePeriodically review forecasts of future

bandwidth needs, andbandwidth needs, and�� Provide early warning of potential problemsProvide early warning of potential problems

�� Create subcommittees when necessary to meet the chargeCreate subcommittees when necessary to meet the charge�� The chair of the committee should report to ICFA once perThe chair of the committee should report to ICFA once per

year, at its joint meeting with laboratory directors (Feb. 2003)year, at its joint meeting with laboratory directors (Feb. 2003)�� Representatives: Major labs, ECFA, ACFA, NA Users, S. AmericaRepresentatives: Major labs, ECFA, ACFA, NA Users, S. America

ICFA-SCIC Core MembershipICFA-SCIC Core Membership

�� Representatives from major HEPRepresentatives from major HEPlaboratories:laboratories:

Manuel Manuel DelfinoDelfino (CERN)(CERN)

to W. Von to W. Von RuedenRueden Michael Ernst Michael Ernst (DESY)(DESY) Matthias Matthias KasemannKasemann (FNAL) (FNAL) Yukio Karita Yukio Karita (KEK)(KEK) Richard Mount Richard Mount (SLAC)(SLAC)

�� User RepresentativesUser Representatives Richard Hughes-Jones (UK)Richard Hughes-Jones (UK) Harvey Newman Harvey Newman (USA)(USA)

Dean Dean KarlenKarlen (Canada)(Canada)

�� For Russia:For Russia: SlavaSlava IlyinIlyin (MSU)(MSU)

�� ECFA representatives: ECFA representatives:

Frederico Ruggieri (INFNFrederico Ruggieri (INFNFrascati),Frascati),Denis Linglin (IN2P3, Lyon)Denis Linglin (IN2P3, Lyon)

�� ACFA representatives: ACFA representatives:

RongshengRongsheng XuXu (IHEP Beijing) (IHEP Beijing)

HwanBaeHwanBae Park Park((KyungpookKyungpook Nat Nat’’llUniversity)University)

�� For South America: For South America: Sergio F. NovaesSergio F. Novaes(University of Sao Paulo)(University of Sao Paulo)

LHC Data Grid HierarchyLHC Data Grid Hierarchy

Tier 1

Tier2 Center

Online System

CERN 700k SI95 ~1 PB Disk;Tape Robot

FNAL: 200kSI95; 600 TBIN2P3 Center INFN CenterRAL Center

InstituteInstituteInstituteInstitute~0.25TIPS

Workstations

~100-400MBytes/sec

2.5 Gbps

100 - 1000Mbits/sec

Tens of Petabytes by 2007-8.An Exabyte within ~5 Years later.Physics data cache

~PByte/sec

~2.5 Gbits/sec

Tier2 CenterTier2 CenterTier2 Center~2.5 Gbps

Tier 0 +1

Tier 3

Tier 4

Tier2 CenterTier 2

Experiment

CERN/Outside Resource Ratio ~1:2Tier0/(ΣΣΣΣ Tier1)/(ΣΣΣΣ Tier2) ~1:1:1

Transatlantic Net WG (HN, L. Price) Bandwidth Requirements (2001) [*] Transatlantic Net WG (HN, L. Price) Bandwidth Requirements (2001) [*]

� 2001 2002 2003 2004 2005 2006

CMS 100 200 300 600 800 2500

ATLAS 50 100 300 600 800 2500

BaBar 300 600 1100 1600 2300 3000

CDF 100 300 400 2000 3000 6000

D0 400 1600 2400 3200 6400 8000

BTeV 20 40 100 200 300 500

DESY 100 180 210 240 270 300

CERN BW

155-310

622 2500 5000 10000 20000

[*] [*] Installed BW. Maximum Link Occupancy 50% AssumedInstalled BW. Maximum Link Occupancy 50% AssumedSee http://See http://gate.hep.anl.gov/lprice/TANgate.hep.anl.gov/lprice/TAN

History – One large Research Site

Projections: 0.5 to 24 Tbps by ~2012

Much of the Traffic:SLAC ���� IN2P3/RAL/INFN;

via ESnet+France;Abilene+CERN

ICFA SCIC Meetings[*] and TopicsICFA SCIC Meetings[*] and Topics

�� Focus on the Digital Divide This Year Focus on the Digital Divide This Year➨➨ Identification of problem areas; work on ways to improve Identification of problem areas; work on ways to improve

�� Network Status and Upgrade Plans in Each Country Network Status and Upgrade Plans in Each Country➨➨ Performance (Throughput) Evolution in Each Country, Performance (Throughput) Evolution in Each Country,

and Transatlantic and Transatlantic�� Performance Monitoring World-Overview Performance Monitoring World-Overview (Les Cottrell, IEPM Project) (Les Cottrell, IEPM Project)�� Specific Technical Topics (Examples): Specific Technical Topics (Examples):➨➨ Bulk transfer, Bulk transfer, QoSQoS, Collaborative Systems, Security, VOIP, Collaborative Systems, Security, VOIP

�� Preparation of Reports to ICFA (Lab Directors Preparation of Reports to ICFA (Lab Directors’’ Meetings) Meetings)➨➨ Last Report: World Network Status and Outlook - Feb. 2002 Last Report: World Network Status and Outlook - Feb. 2002➨➨ Next Report: Next Report: Digital Divide, +Digital Divide, + Monitoring, Advanced Monitoring, Advanced

Technologies, and Requirements Evolution Technologies, and Requirements Evolution –– Feb. 2003 Feb. 2003

[*] Six Meetings in 2002; Next at KEK In December 13.[*] Six Meetings in 2002; Next at KEK In December 13.

SCIC Sub-Committees SCIC Sub-CommitteesWeb Page Web Page http://http://cern.chcern.ch/ICFA-SCIC//ICFA-SCIC/

�� Monitoring:Monitoring: Les Cottrell Les Cottrell((http://www.slac.stanford.edu/xorg/icfa/scic-netmonhttp://www.slac.stanford.edu/xorg/icfa/scic-netmon))With Richard Hughes-Jones (Manchester), With Richard Hughes-Jones (Manchester), Sergio Sergio NovaesNovaes (Sao (SaoPaolo); Paolo); SergeiSergei BerezhnevBerezhnev (RUHEP), (RUHEP), FukukoFukuko Yuasa (KEK), Daniel Yuasa (KEK), DanielDavidsDavids (CERN), Sylvain (CERN), Sylvain RavotRavot (Caltech), Shawn McKee (Michigan) (Caltech), Shawn McKee (Michigan)

�� Advanced Technologies:Advanced Technologies: Richard Hughes-Jones w Richard Hughes-Jones with ith VladimirVladimirKorenkovKorenkov (JINR, (JINR, DubnaDubna), Olivier ), Olivier Martin(CERNMartin(CERN), ), Harvey NewmanHarvey Newman

�� End-to-end Connectivity:End-to-end Connectivity: Richard Mount (SLAC) Richard Mount (SLAC)➨➨ With Michael Ernst, Denis With Michael Ernst, Denis LinglinLinglin, , AlexandreAlexandre SztajnbergSztajnberg (Rio, Brazil) (Rio, Brazil)

�� The Digital Divide:The Digital Divide: Alberto Santoro (Rio, Brazil) Alberto Santoro (Rio, Brazil)➨➨ With With SlavaSlava IlyinIlyin, Yukio , Yukio KaritaKarita, David O. Williams, David O. Williams➨➨ Also Also DongchulDongchul Son (Korea), Son (Korea), HafeezHafeez HooraniHoorani (Pakistan), (Pakistan),

SunandaSunanda BanerjeeBanerjee (India), Vicky White (FNAL) (India), Vicky White (FNAL)�� Key Requirements:Key Requirements: Harvey Newman Harvey Newman

➨➨ Also Charlie Young (SLAC)Also Charlie Young (SLAC)

Network Progress andIssues for Major Experiments

Network Progress andIssues for Major Experiments

�� Major R&E Networks have weathered the economic Major R&E Networks have weathered the economic ““stormstorm””�� Backbones & major links advancing rapidly to 10 Backbones & major links advancing rapidly to 10 GbpsGbps range range

➨➨ ““GbpsGbps”” end-to-end throughput data flows have been end-to-end throughput data flows have been tested; will be in production soon (in 1-2 years) tested; will be in production soon (in 1-2 years)

�� Network advances are changing the view of the netNetwork advances are changing the view of the net’’s roless roles➨➨ Progress to and beyond 10 Progress to and beyond 10 GbpsGbps within next few years within next few years➨➨ Likely to have a profound impact on the experimentsLikely to have a profound impact on the experiments’’

Computing Models, and bandwidth requirementsComputing Models, and bandwidth requirements➨➨ More dynamic view: dynamic path provisioning;More dynamic view: dynamic path provisioning;

GB to TB data transactionsGB to TB data transactions�� Net R&D Driven by Advanced integrated applications, suchNet R&D Driven by Advanced integrated applications, such

as Data Grids, that rely on seamless LAN and WAN operationas Data Grids, that rely on seamless LAN and WAN operation➨➨ With reliable, quantifiable (monitored), high performanceWith reliable, quantifiable (monitored), high performance

➜➜ All of the above will further open the Digital Divide chasm.All of the above will further open the Digital Divide chasm.We need to take actionWe need to take action

Signs of the Times: UncertaintyBut No Change in Outlook

Signs of the Times: UncertaintyBut No Change in Outlook

�� While While Key Providers are in BankruptcyKey Providers are in Bankruptcy➨➨ KPNQwestKPNQwest, , TeleglobeTeleglobe,, Global Crossing, FLAG; Global Crossing, FLAG; WorldcomWorldcom

�� A&R Nets and Major Links Have Continued Their Advance; A&R Nets and Major Links Have Continued Their Advance;Switched Vendors where neededSwitched Vendors where needed➨➨ E.g. T-Systems (Deutsche Telecom) for US-CERNE.g. T-Systems (Deutsche Telecom) for US-CERN

�� Telecom Market Outlook Telecom Market Outlook➨➨ Good pricing continues, in Good pricing continues, in ““open competitionopen competition”” situations situations➨➨ MCI/MCI/WorldcomWorldcom networks continue to operate: networks continue to operate:

20 M customers in US; UK academic & research network20 M customers in US; UK academic & research network➨➨ Level3 market share advance; perhaps also Tyco, GXLevel3 market share advance; perhaps also Tyco, GX

�� Link Donations !: NY-Amsterdam OC192 (Tyco to IEEAF) Link Donations !: NY-Amsterdam OC192 (Tyco to IEEAF)�� Overall: Strong Outlook in R&E Nets Overall: Strong Outlook in R&E Nets➨➨ Regional Progress, and Visions; E.g. IWIRE;Regional Progress, and Visions; E.g. IWIRE;

CALREN: CALREN: ““1 1 GbpsGbps to Every Californian by 2010 to Every Californian by 2010””

ICFA SCIC: R&E Backbone andInternational Link Progress

ICFA SCIC: R&E Backbone andInternational Link Progress

�� GEANT Pan-European BackboneGEANT Pan-European Backbone ( (http://www.dante.net/geanthttp://www.dante.net/geant))➨➨ Now interconnects >31 countries; many trunks Now interconnects >31 countries; many trunks 2.5 and 10 2.5 and 10 GbpsGbps

�� UKUK: : SuperJANETSuperJANET Core at Core at 10 10 GbpsGbps➨➨ 2.5 2.5 GbpsGbps NY-London, with 622 Mbps to NY-London, with 622 Mbps to ESnetESnet and Abilene and Abilene

�� France (IN2P3): France (IN2P3): 2.5 2.5 GbpsGbps RENATER backbone by mid-October RENATER backbone by mid-October➨➨ Lyon-CERN Link being Upgraded to 1 Lyon-CERN Link being Upgraded to 1 GbpsGbps Ethernet Ethernet➨➨ Proposal for dark fiber to CERN by end 2003Proposal for dark fiber to CERN by end 2003

�� SuperSINETSuperSINET (Japan): (Japan): 10 10 GbpsGbps IP and 10 IP and 10 GbpsGbps Wavelength Wavelength Core Core➨➨ Tokyo to NY Links: Now 2 X 2.5 Tokyo to NY Links: Now 2 X 2.5 GbpsGbps; Need to get to Starlight; Need to get to Starlight

�� CA*net4 (Canada):CA*net4 (Canada): Interconnect customer-owned dark fiber Interconnect customer-owned dark fiber nets across Canada at nets across Canada at 10 10 GbpsGbps, started July 2002, started July 2002➨➨ ““Lambda-GridsLambda-Grids”” by ~2004-5 by ~2004-5

�� GWIN (Germany):GWIN (Germany): 2.5 2.5 GbpsGbps Core Core; Connect to US at 2 X 2.5 ; Connect to US at 2 X 2.5 GbpsGbps;;Support for SILK Project: Satellite links to FSU RepublicsSupport for SILK Project: Satellite links to FSU Republics

�� Russia: Russia: Now 155 Mbps links to MoscowNow 155 Mbps links to Moscow➨➨ Moscow-Starlight Link to Moscow-Starlight Link to 155 Mbps155 Mbps (US NSF + Russia Support) (US NSF + Russia Support)➨➨ Moscow-Stockholm Link; connect to GEANT at Moscow-Stockholm Link; connect to GEANT at 155 Mbps155 Mbps

R&E Backbone and Int’l Link ProgressR&E Backbone and Int’l Link Progress

�� Abilene (Internet2) UpgradeAbilene (Internet2) Upgrade from 2.5 to from 2.5 to 10 10 GbpsGbps Underway Underway➨➨ Encourage high throughput use for targeted applicationsEncourage high throughput use for targeted applications

�� ESNET: UpgradeESNET: Upgrade to 10 to 10 GbpsGbps ““NowNow””�� US-CERNUS-CERN➨➨ to 622 Mbps in August; Move to STARLIGHTto 622 Mbps in August; Move to STARLIGHT➨➨ 2.5G Research Triangle from August 20022.5G Research Triangle from August 2002

STARLIGHT-CERN-NL; to 10G in 2003STARLIGHT-CERN-NL; to 10G in 2003�� SLAC + IN2P3 (SLAC + IN2P3 (BaBarBaBar))➨➨ To ~100 Mbps throughput over US-CERN and To ~100 Mbps throughput over US-CERN and RenaterRenater links links➨➨ 600 Mbps Throughput is 600 Mbps Throughput is BaBarBaBar Target for this Year Target for this Year

(with (with ESnetESnet and IN2P3 Link Upgrades) and IN2P3 Link Upgrades)�� FNALFNAL➨➨ ESnetESnet Link Upgrade to 622 Mbps Link Upgrade to 622 Mbps➨➨ Plans for dark fiber to STARLIGHT, planned for this FallPlans for dark fiber to STARLIGHT, planned for this Fall

��NY-Amsterdam NY-Amsterdam Donation Donation from Tyco, from Tyco, September 2002: September 2002:Arranged by IEEAF: Arranged by IEEAF: 622 Gbps+10 622 Gbps+10 GbpsGbps Research Wavelength Research Wavelength

NNNNaaaattttiiiioooonnnnaaaallll RRRReeeesssseeeeaaaarrrrcccchhhh NNNNeeeettttwwwwoooorrrrkkkkssssiiiinnnn JJJJaaaappppaaaannnn

NNNNaaaattttiiiioooonnnnaaaallll RRRReeeesssseeeeaaaarrrrcccchhhh NNNNeeeettttwwwwoooorrrrkkkkssssiiiinnnn JJJJaaaappppaaaannnn

SuperSINETSuperSINET➨➨ Started operation January 4, 2002 Started operation January 4, 2002➨➨ Support for 5 important areas: Support for 5 important areas:

HEP,HEP, Genetics, Genetics, NanoNano-Technology,-Technology,Space/Astronomy, Space/Astronomy, GRIDsGRIDs

➨➨ Provides 10 Provides 10 λλλλλλλλ’’s:s:�� 10 10 GbpsGbps IP connection IP connection�� Direct Direct intersiteintersite GbEGbE links links�� 7 Univ. Connected; 7 Univ. Connected; Add 3 More this Month Add 3 More this Month

HEPnetHEPnet-J-J➨➨ Reconstructed with Reconstructed with MPLS-VPN in MPLS-VPN in SuperSINETSuperSINET

Proposal: Two Proposal: Two TransPacificTransPacific 2.5 2.5 GbpsGbps Wavelengths, and Wavelengths, andJapan-CERN Grid Japan-CERN Grid TestbedTestbed by ~2003 by ~2003

Tokyo

Osaka

Nagoya

Internet

Osaka U

Kyoto U

ICRKyoto-U

Nagoya U

NIFS

NIG

KEK

Tohoku U

IMSU-Tokyo

NAO

U Tokyo

NII Hitot.

NII Chiba

IP

WDM path

IP router

OXC

ISAS

National R&E Network ExampleGermany: DFN Transatlantic Connectivity 2002

STM 4

STM 16

STM 1

6

�� FSU Connections via satellite:FSU Connections via satellite:Yerevan, Minsk, Yerevan, Minsk, AlmatyAlmaty, Baikal, Baikal➨➨ Speeds of 32 - 512 kbps Speeds of 32 - 512 kbps

�� SILK Project (2002): NATO fundingSILK Project (2002): NATO funding➨➨ Links to Caucasus and Central Links to Caucasus and Central Asia (8 Countries) Asia (8 Countries)➨➨In 2001-2 64-512 kbpsIn 2001-2 64-512 kbps➨➨Proposed VSAT for 10-50 X BW:Proposed VSAT for 10-50 X BW: NATO + State Funding NATO + State Funding

�� 2 X OC48: NY-Hamburg 2 X OC48: NY-Hamburgand NY-Frankfurtand NY-Frankfurt

�� Direct Peering to Abilene (US) and Direct Peering to Abilene (US) and CanarieCanarie (Canada) (Canada)

��UCAID will adding another 2 OC48UCAID will adding another 2 OC48’’s;s;in a Proposed Global Terabitin a Proposed Global TerabitResearch Network (GTRN)Research Network (GTRN)

*

Also see http://www-iepm.slac.stanford.edu/monitoring/bulk/;and the Internet2 E2E Initiative: http://www.internet2.edu/e2e

Max. Sustained TCP Throughputon Transatlantic Links

Max. Sustained TCP Throughputon Transatlantic Links

�� 8/10/01 105 Mbps reached with 30 Streams: SLAC-IN2P38/10/01 105 Mbps reached with 30 Streams: SLAC-IN2P3�� 9/1/01 102 Mbps in One Stream: CIT-CERN 9/1/01 102 Mbps in One Stream: CIT-CERN�� 11/5/01 125 Mbps in One Stream (modified kernel): CIT-CERN 11/5/01 125 Mbps in One Stream (modified kernel): CIT-CERN�� 1/09/02 190 Mbps for One stream shared on 2 155 Mbps links 1/09/02 190 Mbps for One stream shared on 2 155 Mbps links�� 3/11/02 120 Mbps 3/11/02 120 Mbps Disk-to-DiskDisk-to-Disk with One Stream on 155 Mbps with One Stream on 155 Mbps

link (Chicago-CERN)link (Chicago-CERN)�� 5/20/02 450-600 Mbps SLAC-Manchester on OC12 with ~100 Streams 5/20/02 450-600 Mbps SLAC-Manchester on OC12 with ~100 Streams�� 6/1/02 290 Mbps Chicago-CERN One Stream on OC12 (mod. Kernel) 6/1/02 290 Mbps Chicago-CERN One Stream on OC12 (mod. Kernel)�� 9/02 850 Mbps Chicago-CERN One Stream on OC48 (mod. 9/02 850 Mbps Chicago-CERN One Stream on OC48 (mod. KernetKernet)) 1350+ (1900+) Mbps with 2 (3) Streams on 2 (3) 1350+ (1900+) Mbps with 2 (3) Streams on 2 (3) GbEGbE Ports Ports

US-CERN DataTAG Link Testswith Grid TCP (S.Ravot et al.): 9/02

US-CERN DataTAG Link Testswith Grid TCP (S.Ravot et al.): 9/02

3 Streams;3 GbE Ports3 Streams;3 GbE Ports

1 to 3Streams;

1-3 GbE Ports

1 to 3Streams;

1-3 GbE Ports

1 32

Next: Tests to 10 Gbps; 1 GB/sec disk to disk

TOE;CPU;

Protocol

TOE;CPU;

Protocol

IIIIEEEEPPPPMMMM:::: PPPPiiiinnnnggggEEEERRRRDDDDeeeeppppllllooooyyyymmmmeeeennnnttttIIIIEEEEPPPPMMMM:::: PPPPiiiinnnnggggEEEERRRRDDDDeeeeppppllllooooyyyymmmmeeeennnntttt

�� Measurements fromMeasurements from➨➨ 34 monitors in 14 countries34 monitors in 14 countries➨➨ Over 600 remote hostsOver 600 remote hosts➨➨ Over 72 countriesOver 72 countries➨➨ Over 3300 monitor-remote site pairsOver 3300 monitor-remote site pairs➨➨ Measurements go back to Jan-95Measurements go back to Jan-95➨➨ Reports on RTT, loss, Reports on RTT, loss, reachabilityreachability,,

jitter, reorders, duplicates jitter, reorders, duplicates ……

�� Countries monitoredCountries monitored➨➨Contain 78% ofContain 78% of

world populationworld population➨➨ 99% of online users99% of online users

of the Internetof the Internet➨➨Mainly A&R sitesMainly A&R sites

Monitoring Sites

Remote Sites

History - Throughput Quality Improvements from USBandwidth of TCP < MSS/(RTT*Sqrt(Loss)) (1)

(1) Macroscopic Behavior of the TCP Congestion Avoidance Algorithm1) Macroscopic Behavior of the TCP Congestion Avoidance Algorithm , , MatthisMatthis,,SemkeSemke, , MahdaviMahdavi, , OttOtt, Computer Communication Review 27(3), July 1997, Computer Communication Review 27(3), July 1997

80% annualimprovementFactor ~100/8 yr

Progress: but Digital Divide is Maintained

Work on the Digital Divide:Several Perspectives

Work on the Digital Divide:Several Perspectives

�� Identify & Help Solve Technical Problems:Identify & Help Solve Technical Problems: NatNat’’l, Regional, Last 10/1/0.1 kml, Regional, Last 10/1/0.1 km

�� Inter-Regional Proposals (Example: Brazil)Inter-Regional Proposals (Example: Brazil)➨➨ US NSF Proposal (10/2002); possible EU LIS ProposalUS NSF Proposal (10/2002); possible EU LIS Proposal

�� Work on Policies and/or Pricing: Work on Policies and/or Pricing: pkpk, in, , in, brbr, , cncn, SE Europe, , SE Europe, ……➨➨ E.g. E.g. RoEduNetRoEduNet (2-6 Mbps); Pricing not so different (2-6 Mbps); Pricing not so different

from US-CERN price in 2002 for a few from US-CERN price in 2002 for a few GbpsGbps➨➨ Find Ways to work with vendors, Find Ways to work with vendors, NRENsNRENs, and/or , and/or GovGov’’tsts

�� Use Model Cases: Installation of new advanced fiberUse Model Cases: Installation of new advanced fiberinfrastructures; Convince Neighboring Countriesinfrastructures; Convince Neighboring Countries➨➨ Poland (to 5k km Fiber); Hungary; Slovakia; IrelandPoland (to 5k km Fiber); Hungary; Slovakia; Ireland

�� Exploit One-off Solutions: Exploit One-off Solutions: E.g. extend the SILK ProjectE.g. extend the SILK Project(DESY/FSU satellite links) to a SE European site(DESY/FSU satellite links) to a SE European site

�� Work with other organizations:Work with other organizations: TerenaTerena, Internet2, AMPATH,, Internet2, AMPATH,IEEAF, UN, etc. IEEAF, UN, etc. to help with technical and/or political to help with technical and/or political solsol’’nsns

GEANT

155Mbps

34Mbps 34Mbps

34Mbps

34Mbps

34MbpsAnnual Cost

> 1 MEuro

Romania: 155 Mbps to GEANT and Bucharest;Inter-City Links of 2-6 Mbps

Romania: 155 Mbps to GEANT and Bucharest;Inter-City Links of 2-6 Mbps

Digital Divide Committee

HUNGARY

NREN Core Network Size (Mbps-km):http://www.terena.nl/compendium/2002

NREN Core Network Size (Mbps-km):http://www.terena.nl/compendium/2002

Logarithmic Scale

1k

100k

100

100M

10M

1M

10kRo

It

PlGrIr

Ukr

Hu Cz

Es

Nl

Fi

Ch

Lagging

In Transition

Leading

Advanced

RNP Brazil (to 20 Mbps)

FIU Miami from So. America (to 80 Mbps)

Note: Auger (AG), ALMA (Chile),CMS-Tier1 (Brazil)

Digital Divide ActivitiesDigital Divide Activities�� Questionnaire Being Distributed (Discuss At ICFA Meeting) Questionnaire Being Distributed (Discuss At ICFA Meeting)➨➨ CERN/IT to Assist with Web Form; Online SubmissionCERN/IT to Assist with Web Form; Online Submission

�� Plan on Project to Build HENP World Network Map; Plan on Project to Build HENP World Network Map; Updated and Maintained on a Web Site, Backed by Database: Updated and Maintained on a Web Site, Backed by Database:➨➨ Systematize and Track Needs and Status Systematize and Track Needs and Status➨➨ Information: Link Bandwidths, Utilization, Quality, Pricing, Information: Link Bandwidths, Utilization, Quality, Pricing,

Local Infrastructure, Last Mile Problems, Vendors, etc. Local Infrastructure, Last Mile Problems, Vendors, etc.➨➨ Identify Urgent Cases; Focus on Opportunities to Help Identify Urgent Cases; Focus on Opportunities to Help

�� First ICFA SCIC Workshop: Focus on the Digital DivideFirst ICFA SCIC Workshop: Focus on the Digital Divide➨➨ Target Date February 2004 in Rio de Janeiro (LISHEP) Target Date February 2004 in Rio de Janeiro (LISHEP)➨➨ Organization Meeting July 2003 Organization Meeting July 2003➨➨ Plan Statement at the WSIS, Geneva (December 2003) Plan Statement at the WSIS, Geneva (December 2003)➨➨ Install and Leave Behind a Good Network Install and Leave Behind a Good Network➨➨ Then 1 (to 2) Workshops Per Year, at Sites that Need Help Then 1 (to 2) Workshops Per Year, at Sites that Need Help

Digital Divide Sub-Committee: Questionnaire Response Extract:

34 Mbps Link toGEANT.Universitypays for Network

HEP group intends to build a type T2 or T3 Grid node, andcontribute to MC Production. Need to upgrade Network toGigabit/sec. In principle there is no access limit theNetwork now. But daily traffic load is the real limitation.

University ofCYPRUS

Will maintain a farm for Monte Carlo studies and a Tier 3Grid node; need more bandwidth

IFT-UNESPBrazil

Will have a Tier3 Grid node; Need network bandwidthupgrades at reasonable price (Telecom monopoly)

TIFR, India

Also plans for T2at NUST

In terms of bandwidth need to upgrade but no last mileconnection problem. High Prices

QAU, Pakistan

Additional bandwidth needed for HEP; now 2 Mbps for > 1000 people

JSI, Slovenia

a) LAN upgrade to 100 Mbps; b) LAN-to-WAN upgrade to 4 Mbps

UNLP, Argentina

Dedicated 2 Mbps link for HEP group Needed(now 2 X 2 Mbps for Whole Community; Not Enough)

Cinvestav,Mexico

Wait for approvedfunds to build aT3; In 2005/6 a T1

HEPGRID PROJECT presented for financial support to workon CMS; T3 then T1. Last Mile Problem: Need to reach RNPwith good routing from UERJ

UERJ, Brazil

OtherComputing / networking needs related to HEPInstitution

Groningen CarrierHotel: March 2002

“Cultivate and promotepractical solutions todelivering scalable,

universally available andequitable access to

suitable bandwidth andnecessary network

resources in support ofresearch and education

collaborations.”

http://www.ieeaf.org

NY-AMS 9/02CA-Tokyo by ~1/03

(Research)

CN

SG

PERTH

GHANA

BuenosAires/San

Paolo

St. Petersburg

KazakhstanUzbekistan

ChenaiNavi

Mumbai

Barcelona GreeceMD

NL

CA

Global Quilt Initiative – GMRE Initiative - 001

Global Medical Research Exchange Initiative Bio-Medicine and Health Sciences

Layer 1 – Spoke & Hub Sites

Layer 2 – Spoke & Hub Sites

Layer 3 – Spoke & Hub Sites

Propose Global Research and Education Network for Physics

Networking for HENPNetworking for HENP

�� Networks are vital for global collaboration, andNetworks are vital for global collaboration, andthe present and future of Our Field; But: the present and future of Our Field; But: Digital DivideDigital Divide

What ICFA and the HENP Community Can DoWhat ICFA and the HENP Community Can Do�� Spread the message: ICFA SCIC is there to helpSpread the message: ICFA SCIC is there to help�� Help identify and highlight specific needs (to Work On)Help identify and highlight specific needs (to Work On)

➨➨Policy problems; Last Mile problems; etc.Policy problems; Last Mile problems; etc.�� Encourage Joint programs [such as in Encourage Joint programs [such as in DESYDESY’’ss Silk project; Silk project;

Japanese links to SE Asia and China; AMPATH to So. America]Japanese links to SE Asia and China; AMPATH to So. America]➨➨ NSF & LIS Proposals: US and EU to South America NSF & LIS Proposals: US and EU to South America

�� Make direct contacts, arrange discussions with Make direct contacts, arrange discussions with govgov’’tt officials officials➨➨ ICFA SCIC is prepared to participate ICFA SCIC is prepared to participate

�� Help Start, or Get Support for Workshops on Networks (& Grids)Help Start, or Get Support for Workshops on Networks (& Grids)➨➨ Discuss & Create opportunities Discuss & Create opportunities➨➨ Encourage, help form funded programs Encourage, help form funded programs

�� Help form Regional support & training groups (requires funding)Help form Regional support & training groups (requires funding)

Some Extra

Slides Follow

Some Extra

Slides Follow

Networks, Grids and HENPNetworks, Grids and HENP�� Next generation 10 Next generation 10 GbpsGbps network backbones are network backbones are

almost here: in the US, Europe and Japanalmost here: in the US, Europe and Japan➨➨ First stages arriving, starting now First stages arriving, starting now

�� Major transoceanic links at 2.5 - 10 Major transoceanic links at 2.5 - 10 GbpsGbps in 2002-3 in 2002-3�� Getting high (reliable; Grid) application performanceGetting high (reliable; Grid) application performance

across networks means!across networks means!➨➨ End-to-end monitoring; a coherent approach End-to-end monitoring; a coherent approach➨➨ Getting high performance (TCP) toolkits in users Getting high performance (TCP) toolkits in users’’ hands hands➨➨ Working in concert with AMPATH, Internet2, Working in concert with AMPATH, Internet2, TerenaTerena;; DataTAGDataTAG, the Grid projects and the Global Grid Forum, the Grid projects and the Global Grid Forum

�� Network improvements are especially needed inNetwork improvements are especially needed inSE Europe, So. America; SE Asia, and Africa:SE Europe, So. America; SE Asia, and Africa:➨➨ Key Examples: India, Pakistan, China; Brazil; Romania Key Examples: India, Pakistan, China; Brazil; Romania

�� Removing regional, Last mile bottlenecks and compromisesRemoving regional, Last mile bottlenecks and compromisesin network quality are now (in all world regions)in network quality are now (in all world regions) On the critical pathOn the critical path

0

5000

10000

15000

20000

Link Bandwidth (Mbps)

BW (Mbps) 310 622 2500 5000 10000 20000

FY2001

FY2002

FY2003

FY2004

FY2005

FY2006

Baseline BW for the US-CERN Link: HENP Transatlantic WG (DOE+NSF)Baseline BW for the US-CERN Link: HENP Transatlantic WG (DOE+NSF)

�� DataTAGDataTAG 2.5 2.5 GbpsGbps Research Link since Summer 2002 Research Link since Summer 2002�� 10 10 GbpsGbps Research Link by Approx. Mid-2003 Research Link by Approx. Mid-2003

TransoceanicNetworking

Integrated withthe Abilene,

TeraGrid,Regional Nets

and ContinentalNetwork

Infrastructuresin US, Europe,

Asia, SouthAmerica

Baseline evolution typicalBaseline evolution typicalof major HENPof major HENP

links 2001-2006 links 2001-2006

History – Loss Quality (Cottrell)

�Fewer sites have very poor to dreadful performance

�More have good performance (< 1% Loss)

History – oneResearch ISP

100% growth in traffic/yr for

last 12 years

Continuous upgrades

Increase packet size (bulk

throughput apps)

Key Network Issues &Challenges

Key Network Issues &Challenges

Net Infrastructure Requirements for High ThroughputNet Infrastructure Requirements for High Throughput��Packet Loss must be ~Zero (at and below 10Packet Loss must be ~Zero (at and below 10-6-6))

�� I.e. No I.e. No ““CommodityCommodity”” networks networks�� Need to track down Need to track down uncongesteduncongested packet loss packet loss

��No Local infrastructure bottlenecksNo Local infrastructure bottlenecks�� Multiple Gigabit Ethernet Multiple Gigabit Ethernet ““clear pathsclear paths”” between between

selected host pairs are needed nowselected host pairs are needed now�� To 10 To 10 GbpsGbps Ethernet paths by 2003 or 2004 Ethernet paths by 2003 or 2004

��TCP/IP stack configuration and tuning Absolutely RequiredTCP/IP stack configuration and tuning Absolutely Required�� Large Windows; Possibly Multiple Streams Large Windows; Possibly Multiple Streams�� New Concepts of New Concepts of Fair UseFair Use Must then be Developed Must then be Developed

��Careful Router, Server, Client, Interface configurationCareful Router, Server, Client, Interface configuration�� Sufficient CPU, I/O and NIC throughput sufficient Sufficient CPU, I/O and NIC throughput sufficient

��End-to-endEnd-to-end monitoring and tracking of performance monitoring and tracking of performance��Close collaboration with local and Close collaboration with local and ““regionalregional”” network staffs network staffs

TCP Does Not Scale to the 1-10 TCP Does Not Scale to the 1-10 GbpsGbps Range Range

AssistedAssisted by by LoansLoans: Level3 (OC192) : Level3 (OC192) andand CiscoCisco (10GbE (10GbE andand 16X1GbE) 16X1GbE)

iGrid2002: OC192+OC48 iGrid2002: OC192+OC48 SetupSetupSeptemberSeptember 2002 2002

Short Term Donation from Level 3

Short Term Donation from Level 3

Argonne

Starlight-Chicago

Amsterdam

CERN

HENP Major Links: BW Roadmapin Gbps (HBN); Shown at ICHEP2002

HENP Major Links: BW Roadmapin Gbps (HBN); Shown at ICHEP2002

Year Production Experimental Remarks

2001 0.155 0.622-2.5 SONET/SDH

2002 0.622 2.5 SONET/SDH DWDM; GigE Integ.

2003 2.5 10 DWDM; 1 + 10 GigE Integration

2005 10 2-4 X 10 ???? Switch; ???? Provisioning

2007 2-4 X 10 ~10 X 10; 40 Gbps

1st Gen. ???? Grids

2009 ~10 X 10 or 1 -2 X 40

~5 X 40 or ~20-50 X 10

40 Gbps ???? Switching

2011 ~5 X 40 or

~20 X 10

~25 X 40 or ~100 X 10

2nd Gen ???? Grids Terabit Networks

2013 ~Terabit ~MultiTerabit ~Fill a Fiber

LHCnet Network : Late 2002LHCnet Network : Late 2002

Development and testsDevelopment and tests

Abilene MRENESnet STARTAPNASA

Linux PC forPerformance tests

& Monitoring

CERN -Geneva

GEANT Switch IN2P3 WHO

Linux PC forPerformance tests

& Monitoring

Caltech/DoE PoP – StarLight Chicago

622 Mbps (Prod.)622 Mbps (Prod.)

Cisco 7609CERN

Cisco 7609Caltech(DoE)

Alcatel 7770DataTAG(CERN)

Cisco 7606DataTAG(CERN)

Juniper M10DataTAG(CERN)

Cisco 7606Caltech(DoE)

Juniper M10Caltech(DoE)

Alcatel 7770DataTAG(CERN)

2.5 2.5 GbpsGbps (R&D) (R&D)

Optical Mux/DmuxAlcatel 1670

Optical Mux/DmuxAlcatel 1670

NLNLSURFnet

GENEVA

UKUKSuperJANET4

ABILENEABILENE

ESNETESNET

CALRENCALREN

ItItGARR-B

GEANT

NewYork

FrFrRenater

STAR-TAP

STARLIGHT

DataTAGDataTAG Project Project

�� EU-Solicited Project. EU-Solicited Project. CERNCERN, PPARC (UK), Amsterdam (NL), and INFN (IT);, PPARC (UK), Amsterdam (NL), and INFN (IT);and US (DOE/NSF: UIC, NWU and Caltech) partnersand US (DOE/NSF: UIC, NWU and Caltech) partners

��Main Aims:Main Aims:➨➨ Ensure maximum interoperability between US and EU Grid ProjectsEnsure maximum interoperability between US and EU Grid Projects➨➨ Transatlantic Transatlantic TestbedTestbed for advanced network research for advanced network research

�� 2.5 2.5 GbpsGbps Wavelength Triangle 7/02 (10 Wavelength Triangle 7/02 (10 GbpsGbps Triangle in 2003) Triangle in 2003)

Wave Triangle

The Rapid Pace of NetworkTechnology Advances Continues

The Rapid Pace of NetworkTechnology Advances Continues

Within the Next One to Two YearsWithin the Next One to Two Years�� 10 10 GbpsGbps Ethernet on Switches and Servers; Ethernet on Switches and Servers;

LAN/WAN integration at 10 LAN/WAN integration at 10 GbpsGbps➨➨ Intel 10 Intel 10 GbEGbE server server NICsNICs in Beta Test in Beta Test

�� 40 40 GbpsGbps Wavelengths Being Shown Wavelengths Being Shown�� NextGenNextGen Routers: 100 Routers: 100 MppsMpps forwarding engines, 4 forwarding engines, 4

and More 10 and More 10 GbpsGbps ports per Slot; Terabit/sec ports per Slot; Terabit/secbackplanes etc.backplanes etc.

�� Broadband Wireless [Multiple 3G/4G alternatives]: Broadband Wireless [Multiple 3G/4G alternatives]: the drive to defeat the last mile problem the drive to defeat the last mile problem➨➨ 802.11 802.11 abab, UWB; Mobile: CDMA2000/1xEV-DO, UWB; Mobile: CDMA2000/1xEV-DO

HENP Lambda Grids:Fibers for Physics

HENP Lambda Grids:Fibers for Physics

�� Problem: Extract Problem: Extract ““SmallSmall”” Data Subsets of 1 to 100 Terabytes from 1 Data Subsets of 1 to 100 Terabytes from 1to 1000 to 1000 PetabytePetabyte Data Stores Data Stores

�� Survivability of the HENP Global Grid System, withSurvivability of the HENP Global Grid System, withhundreds of such transactions per day (circa 2007)hundreds of such transactions per day (circa 2007)requires that each transaction be completed in arequires that each transaction be completed in arelatively short time.relatively short time.

�� Example: Take 800 Example: Take 800 secssecs to complete the transaction. Then to complete the transaction. Then Transaction Size (TB)Transaction Size (TB) Net Throughput (Net Throughput (GbpsGbps))

1 10 1 10 10 100 10 100 100 1000 (Capacity of 100 1000 (Capacity of

Fiber Today) Fiber Today)�� Summary: Providing Switching of 10 Summary: Providing Switching of 10 GbpsGbps wavelengths wavelengths

within ~3 years; and Terabit Switching within 5-7 yearswithin ~3 years; and Terabit Switching within 5-7 yearswould enable would enable ““PetascalePetascale Grids with Terabyte transactions Grids with Terabyte transactions””,,as required to fully realize the discovery potential of major HENPas required to fully realize the discovery potential of major HENPprograms, as well as other data-intensive fields.programs, as well as other data-intensive fields.

WSIS-RO: Bucharest Nov. 7-9WSIS is 12/03 & 05 (Geneva & Tunis)

WSIS-RO: Bucharest Nov. 7-9WSIS is 12/03 & 05 (Geneva & Tunis)

�� The Regional Ministerial Conference, based on the UN/ECEThe Regional Ministerial Conference, based on the UN/ECEmembership, is in preparation of the World Summit on themembership, is in preparation of the World Summit on theInformation Society - Bucharest, 7-9 November 2002Information Society - Bucharest, 7-9 November 2002

�� The UN General Assembly adopted in 2001 a resolution endorsingThe UN General Assembly adopted in 2001 a resolution endorsingthe organization of the the organization of the World Summit on the Information SocietyWorld Summit on the Information Society(WSIS), to be convened under the patronage of the United Nations(WSIS), to be convened under the patronage of the United NationsSecretary-General, Secretary-General, KofiKofi AnnanAnnan, with the International, with the InternationalTelecommunication Union taking the lead role in its preparationTelecommunication Union taking the lead role in its preparationalong with interested UN organizations and the host countries.along with interested UN organizations and the host countries.The Summit will offer a unique opportunity for the worldThe Summit will offer a unique opportunity for the worldcommunity to discuss and give shape to the Information Societycommunity to discuss and give shape to the Information Societyby bringing together governments, international organizations,by bringing together governments, international organizations,private sector and civil society. In preparation for this Summitprivate sector and civil society. In preparation for this Summitregional conferences are organized to develop specific proposals.regional conferences are organized to develop specific proposals.

WSIS-RO: Role of New Technologiesin Developing the Information SocietyWSIS-RO: Role of New Technologies

in Developing the Information Society

�� Tony Hey, UK Core E-Science ProgramTony Hey, UK Core E-Science Program

�� Peter Peter ZangliZangli, European Commission, European Commission

�� Hans Hoffmann, CERN Scientific and TT DirectorHans Hoffmann, CERN Scientific and TT Director

�� FabrizioFabrizio GagliardiGagliardi: EDG Project Leader: EDG Project Leader

�� William Johnston, Lawrence Berkeley LabWilliam Johnston, Lawrence Berkeley Lab

�� Dave Reese or Tom West, CENIC, Pacific and NatDave Reese or Tom West, CENIC, Pacific and Nat’’l Light Raill Light Rail

�� Don Riley or Ed Don Riley or Ed FontegrossiFontegrossi, IEEAF, IEEAF

�� Harvey Newman, CaltechHarvey Newman, Caltech

Building Petascale Global Grids:Implications for Society

Building Petascale Global Grids:Implications for Society

�� Meeting the challenges of Meeting the challenges of Petabyte-to-ExabytePetabyte-to-ExabyteGrids, and Gigabit-to-Terabit Networks, willGrids, and Gigabit-to-Terabit Networks, will transform research in science and engineering transform research in science and engineering

�� These developments will create the first trulyThese developments will create the first truly global virtual organizations (GVO) global virtual organizations (GVO)

�� If these developments are successful this couldIf these developments are successful this couldlead to profound advances in industry, commercelead to profound advances in industry, commerceand society at largeand society at large

➨➨ By changing the relationship between people By changing the relationship between people and and ““persistentpersistent”” information in their daily lives information in their daily lives

➨➨ Within the next five to ten years Within the next five to ten years