Harvey B Newman, Caltech Harvey B Newman, Caltech Optical Networks and Grids Meeting the Advanced...
-
Upload
nora-white -
Category
Documents
-
view
219 -
download
0
Transcript of Harvey B Newman, Caltech Harvey B Newman, Caltech Optical Networks and Grids Meeting the Advanced...
Harvey B Newman, CaltechHarvey B Newman, Caltech
Optical Networks and GridsOptical Networks and GridsMeeting the Advanced Network Needs of ScienceMeeting the Advanced Network Needs of Science
May 7, 2002May 7, 2002http://l3www.cern.ch/~newman/HENPGridsNets_I2Mem050702.ppthttp://l3www.cern.ch/~newman/HENPGridsNets_I2Mem050702.ppt
HENP HENP Grids and NetworksGrids and Networks Global Virtual Organizations Global Virtual Organizations
Computing Challenges: Computing Challenges: Petabyes, Petaflops, Global VOsPetabyes, Petaflops, Global VOs
Geographical dispersion:Geographical dispersion: of people and resources of people and resources Complexity:Complexity: the detector and the LHC environment the detector and the LHC environment Scale: Scale: Tens of Petabytes per year of dataTens of Petabytes per year of data
5000+ Physicists 250+ Institutes 60+ Countries
Major challenges associated with:Major challenges associated with:Communication and collaboration at a distanceCommunication and collaboration at a distance
Managing globally distributed computing & data resources Managing globally distributed computing & data resources Cooperative software development and physics analysisCooperative software development and physics analysis
New Forms of Distributed Systems: Data GridsNew Forms of Distributed Systems: Data Grids
Four LHC Experiments: The Four LHC Experiments: The Petabyte to Exabyte Petabyte to Exabyte
ChallengeChallengeATLAS, CMS, ALICE, LHCBATLAS, CMS, ALICE, LHCB
Higgs + New particles; Quark-Gluon Plasma; CP ViolationHiggs + New particles; Quark-Gluon Plasma; CP Violation
Data storedData stored ~40 Petabytes/Year and UP; ~40 Petabytes/Year and UP; CPU CPU 0.30 Petaflops and UP 0.30 Petaflops and UP
0.1 to 1 Exabyte (1 EB = 100.1 to 1 Exabyte (1 EB = 101818 Bytes) Bytes) (2007) (~2012 ?) for the LHC Experiments(2007) (~2012 ?) for the LHC Experiments
All charged tracks with pt > 2 GeV
Reconstructed tracks with pt > 25 GeV
(+30 minimum bias events)
109 events/sec, selectivity: 1 in 1013 (1 person in a thousand world populations)
LHC: Higgs Decay into 4 muons LHC: Higgs Decay into 4 muons (Tracker only); 1000X LEP Data Rate(Tracker only); 1000X LEP Data Rate
LHC Data Grid HierarchyLHC Data Grid Hierarchy
Tier 1
Tier2 Center
Online System
CERN 700k SI95 ~1 PB Disk; Tape Robot
FNAL: 200k SI95; 600 TBIN2P3 Center INFN Center RAL Center
InstituteInstituteInstituteInstitute ~0.25TIPS
Workstations
~100-400 MBytes/sec
2.5 Gbps
0.1–1 GbpsPhysicists work on analysis “channels”
Each institute has ~10 physicists working on one or more channels
Physics data cache
~PByte/sec
~2.5 Gbps
Tier2 CenterTier2 CenterTier2 Center
~2.5 Gbps
Tier 0 +1
Tier 3
Tier 4
Tier2 Center Tier 2
Experiment
CERN/Outside Resource Ratio ~1:2Tier0/( Tier1)/( Tier2) ~1:1:1
Next Generation Networks for Experiments: Goals and NeedsNext Generation Networks for Experiments: Goals and Needs
Providing rapid access to event samples, subsets Providing rapid access to event samples, subsets and analyzed physics results from massive data storesand analyzed physics results from massive data stores From Petabytes by 2002, ~100 Petabytes by 2007, From Petabytes by 2002, ~100 Petabytes by 2007,
to ~1 Exabyte by ~2012. to ~1 Exabyte by ~2012. Advanced integrated applications, such as Data Grids, Advanced integrated applications, such as Data Grids,
rely on seamless operation of our LANs and WANsrely on seamless operation of our LANs and WANs With reliable, monitored, quantifiable high performanceWith reliable, monitored, quantifiable high performance
Providing analyzed results with rapid turnaround, byProviding analyzed results with rapid turnaround, bycoordinating and managing the coordinating and managing the LIMITED LIMITED computing, computing, data handling and data handling and NETWORK NETWORK resources effectivelyresources effectively
Enabling rapid access to the data and the collaborationEnabling rapid access to the data and the collaboration Across an ensemble of networks of varying capabilityAcross an ensemble of networks of varying capability
Large data samples explored and analyzed by thousands of Large data samples explored and analyzed by thousands of globally dispersed scientists, in hundreds of teamsglobally dispersed scientists, in hundreds of teams
Baseline BW for the US-CERN Link:Baseline BW for the US-CERN Link: HENP Transatlantic WG (DOE+NSF HENP Transatlantic WG (DOE+NSF))
US-CERN Link: 622 Mbps this monthUS-CERN Link: 622 Mbps this month DataTAG 2.5 Gbps Research Link in Summer 2002DataTAG 2.5 Gbps Research Link in Summer 2002 10 Gbps Research Link by Approx. Mid-200310 Gbps Research Link by Approx. Mid-2003
Transoceanic Networking
Integrated with the Abilene,
TeraGrid, Regional Nets
and Continental Network
Infrastructuresin US, Europe,
Asia, South America
Baseline evolution typicalBaseline evolution typicalof major HENPof major HENP
links 2001-2006 links 2001-2006
Transatlantic Net WG (HN, L. Price) Bandwidth Requirements [*]
Transatlantic Net WG (HN, L. Price) Bandwidth Requirements [*]
2001 2002 2003 2004 2005 2006
CMS 100 200 300 600 800 2500
ATLAS 50 100 300 600 800 2500
BaBar 300 600 1100 1600 2300 3000
CDF 100 300 400 2000 3000 6000
D0 400 1600 2400 3200 6400 8000
BTeV 20 40 100 200 300 500
DESY 100 180 210 240 270 300
CERN BW
155-310
622 2500 5000 10000 20000
[*] [*] Installed BW. Maximum Link Occupancy 50% Installed BW. Maximum Link Occupancy 50%
AssumedAssumed
See http://gate.hep.anl.gov/lprice/TANSee http://gate.hep.anl.gov/lprice/TAN
AMS-IX Internet Exchange Throughput Accelerating Growth in Europe (NL)
AMS-IX Internet Exchange Throughput Accelerating Growth in Europe (NL)
Monthly Traffic2X Growth from 8/00 - 3/01;2X Growth from 8/01 - 12/01 ↓
2.0 Gbps
4.0 Gbps
6.0 GbpsHourly Traffic
3/22/02
Emerging Emerging Data GridData Grid User Communities User Communities
NSF Network for Earthquake Engineering NSF Network for Earthquake Engineering Simulation (NEES)Simulation (NEES) Integrated instrumentation, Integrated instrumentation,
collaboration, simulationcollaboration, simulation Grid Physics Network (GriPhyN)Grid Physics Network (GriPhyN)
ATLAS, CMS, LIGO, SDSSATLAS, CMS, LIGO, SDSS Access Grid; VRVS: supporting Access Grid; VRVS: supporting
group-based collaborationgroup-based collaboration
AndAnd Genomics, Proteomics, ...Genomics, Proteomics, ... The Earth System Grid and EOSDISThe Earth System Grid and EOSDIS Federating Brain DataFederating Brain Data Computed MicroTomography Computed MicroTomography …… Virtual Observatories Virtual Observatories
Upcoming Grid Challenges: SecureWorkflow Management and Optimization
Upcoming Grid Challenges: SecureWorkflow Management and Optimization
Maintaining a Maintaining a Global ViewGlobal View of Resources and System State of Resources and System State Coherent end-to-end System MonitoringCoherent end-to-end System Monitoring Adaptive Learning: new paradigms for execution Adaptive Learning: new paradigms for execution
optimization (eventually automated)optimization (eventually automated) Workflow Management,Workflow Management, Balancing Policy Versus Balancing Policy Versus
Moment-to-moment Capability to Complete TasksMoment-to-moment Capability to Complete Tasks Balance High Levels of Usage of Limited Resources Balance High Levels of Usage of Limited Resources
Against Better Turnaround Times for Priority JobsAgainst Better Turnaround Times for Priority Jobs Matching Resource Usage to Policy Over the Long Matching Resource Usage to Policy Over the Long
TermTerm Goal-Oriented Algorithms; SteeringGoal-Oriented Algorithms; Steering Requests Requests
According to (Yet to be Developed) MetricsAccording to (Yet to be Developed) Metrics Handling User-Grid Interactions: Guidelines; AgentsHandling User-Grid Interactions: Guidelines; Agents Building Higher Level Services, and an IntegratedBuilding Higher Level Services, and an Integrated
Scalable (Agent-Based) User Environment for the AboveScalable (Agent-Based) User Environment for the Above
((Physicists’) Application CodesPhysicists’) Application Codes Experiments’ Software Framework LayerExperiments’ Software Framework Layer
Modular and Grid-aware: Architecture able to Modular and Grid-aware: Architecture able to interact effectively with the lower layers interact effectively with the lower layers
Grid Applications LayerGrid Applications Layer (Parameters and algorithms that govern system operations)(Parameters and algorithms that govern system operations)
Policy and priority metricsPolicy and priority metrics Workflow evaluation metricsWorkflow evaluation metrics Task-Site Coupling proximity metricsTask-Site Coupling proximity metrics
Global End-to-End System Services LayerGlobal End-to-End System Services Layer Monitoring and Tracking Component performanceMonitoring and Tracking Component performance Workflow monitoring and evaluation mechanismsWorkflow monitoring and evaluation mechanisms Error recovery and redirection mechanismsError recovery and redirection mechanisms System self-monitoring, evaluation and System self-monitoring, evaluation and
optimizationoptimization mechanisms mechanisms
Application Architecture: Application Architecture: Interfacing to the GridInterfacing to the Grid
COJAC: CMS ORCA Java Analysis Component: Java3D Objectivity JNI
Web Services
COJAC: CMS ORCA Java Analysis Component: Java3D Objectivity JNI
Web Services
Demonstrated Caltech-Riode Janeiro (Feb.) and ChileDemonstrated Caltech-Riode Janeiro (Feb.) and Chile
The simulation program developed within MONARC (MModels odels OOf f NNetworked etworked AAnalysis At nalysis At RRegional egional CCenters) enters) uses a process- oriented approach for discrete event simulation, and provides a realistic modelling tool for large scale distributed systems.
Modeling and Simulation:Modeling and Simulation:MONARC System (I. Legrand)MONARC System (I. Legrand)
SIMULATION of Complex Distributed Systems for LHC
MONARC SONN: 3 Regional Centres MONARC SONN: 3 Regional Centres Learning to Export Jobs (Day 9)Learning to Export Jobs (Day 9)
NUST20 CPUs
CERN30 CPUs
CALTECH25 CPUs
1MB/s ; 150 ms RTT
1.2 MB
/s
150 ms R
TT
0
.8 M
B/s
200
ms
RTT
Day = 9
<E> = 0.73
<E> = 0.66
<E> = 0.83
By I. Legrand
TeraGrid (www.teragrid.org)TeraGrid (www.teragrid.org)NCSA, ANL, SDSC, CaltechNCSA, ANL, SDSC, Caltech
NCSA/UIUC
ANL
UIC Multiple Carrier Hubs
Starlight / NW UnivStarlight / NW Univ
Ill Inst of Tech
Univ of ChicagoIndianapolis (Abilene NOC)
I-WIRE
Caltech
San Diego
DTF Backplane: 4 X 10 Gbps
AbileneChicago
Indianapolis
Urbana
OC-48 (2.5 Gb/s, Abilene)Multiple 10 GbE (Qwest)Multiple 10 GbE (I-WIRE Dark Fiber)
Source: Charlie Catlett, Argonne
A Preview of the Grid Hierarchyand Networks of the LHC Era
NLNLSURFnet
GENEVA
UKUKSuperJANET4
ABILENE
ABILENE
ESNETESNET
CALREN
CALREN
ItItGARR-B
GEANT
NewYork
FrFrRenater
STAR-TAP
STARLIGHT
DataTAG ProjectDataTAG Project
EU-Solicited Project. EU-Solicited Project. CERNCERN, PPARC (UK), Amsterdam (NL), and INFN (IT);, PPARC (UK), Amsterdam (NL), and INFN (IT);and US (DOE/NSF: UIC, NWU and Caltech) partnersand US (DOE/NSF: UIC, NWU and Caltech) partners
Main Aims: Main Aims: Ensure maximum interoperability between US and EU Grid ProjectsEnsure maximum interoperability between US and EU Grid ProjectsTransatlantic Testbed for advanced network researchTransatlantic Testbed for advanced network research
2.5 Gbps Wavelength Triangle 7/02 (10 Gbps Triangle in 2003)2.5 Gbps Wavelength Triangle 7/02 (10 Gbps Triangle in 2003)
Wave
Triangle
RNP Brazil (to 20 Mbps)
FIU Miami/So. America (to 80 Mbps)
A Short List: Revolutions in Information Technology (2002-7)
A Short List: Revolutions in Information Technology (2002-7)
Managed Managed Global Data Grids (As Above)Global Data Grids (As Above) Scalable Data-Intensive Metro and Long HaulScalable Data-Intensive Metro and Long Haul
Network TechnologiesNetwork Technologies DWDM: 10 Gbps then 40 Gbps per DWDM: 10 Gbps then 40 Gbps per ; ;
1 to 10 Terabits/sec per fiber 1 to 10 Terabits/sec per fiber 10 Gigabit Ethernet (See www.10gea.org) 10 Gigabit Ethernet (See www.10gea.org)
10GbE / 10 Gbps LAN/WAN integration 10GbE / 10 Gbps LAN/WAN integration Metro Buildout and Optical Cross ConnectsMetro Buildout and Optical Cross Connects Dynamic Provisioning Dynamic Provisioning Dynamic Path Building Dynamic Path Building
““Lambda GridsLambda Grids” ” Defeating the “Last Mile” ProblemDefeating the “Last Mile” Problem
(Wireless; or Ethernet in the First Mile) (Wireless; or Ethernet in the First Mile) 3G and 4G Wireless Broadband (from ca. 2003);3G and 4G Wireless Broadband (from ca. 2003);
and/or Fixed Wireless “Hotspots” and/or Fixed Wireless “Hotspots” Fiber to the HomeFiber to the Home Community-Owned NetworksCommunity-Owned Networks
Key Network Issues & Challenges
Key Network Issues & Challenges
Net Infrastructure Requirements for High ThroughputNet Infrastructure Requirements for High Throughput Packet Loss must be ~Zero (at and below 10Packet Loss must be ~Zero (at and below 10-6-6))
I.e. No “Commodity” networksI.e. No “Commodity” networks Need to track down uncongested packet lossNeed to track down uncongested packet loss
No Local infrastructure bottlenecksNo Local infrastructure bottlenecks Multiple Gigabit Ethernet “clear paths” between Multiple Gigabit Ethernet “clear paths” between
selected host pairs are needed nowselected host pairs are needed now To 10 Gbps Ethernet paths by 2003 or 2004To 10 Gbps Ethernet paths by 2003 or 2004
TCP/IP stack configuration and tuning Absolutely RequiredTCP/IP stack configuration and tuning Absolutely Required Large Windows; Possibly Multiple StreamsLarge Windows; Possibly Multiple Streams New Concepts of New Concepts of Fair UseFair Use Must then be Developed Must then be Developed
Careful Router, Server, Client, Interface configuration Careful Router, Server, Client, Interface configuration Sufficient CPU, I/O and NIC throughput sufficientSufficient CPU, I/O and NIC throughput sufficient
End-to-endEnd-to-end monitoring and tracking of performance monitoring and tracking of performance Close collaboration with local and “regional” network staffsClose collaboration with local and “regional” network staffs
TCP Does Not Scale to the 1-10 Gbps RangeTCP Does Not Scale to the 1-10 Gbps Range
True End to End Experience
User perception ApplicationOperating systemHost IP stackHost network cardLocal Area Network Campus backbone
networkCampus link to regional
network/GigaPoPGigaPoP link to
Internet2 national backbones
International connections
EYEBALL
APPLICATION
STACK
JACK
NETWORK
. . .
. . .
. . .
. . .
Internet2 HENP WG [*]Internet2 HENP WG [*] Mission: To help ensure that the requiredMission: To help ensure that the required
National and international network infrastructuresNational and international network infrastructures(end-to-end)(end-to-end)
Standardized tools and facilities for high performance Standardized tools and facilities for high performance and end-to-end monitoring and tracking, andand end-to-end monitoring and tracking, and
Collaborative systemsCollaborative systems are developed and deployed in a timely manner, and used are developed and deployed in a timely manner, and used
effectively to meet the needs of the US LHC and other major effectively to meet the needs of the US LHC and other major HENP Programs, as well as the at-large scientific community.HENP Programs, as well as the at-large scientific community. To carry out these developments in a way that is To carry out these developments in a way that is
broadly applicable across many fields broadly applicable across many fields Formed an Internet2 WG as a suitable framework: Formed an Internet2 WG as a suitable framework:
Oct. 26 2001 Oct. 26 2001 [*] Co-Chairs: S. McKee (Michigan), H. Newman (Caltech);[*] Co-Chairs: S. McKee (Michigan), H. Newman (Caltech);
Sec’y J. Williams (Indiana) Sec’y J. Williams (Indiana) Website: Website: http://www.internet2.edu/henphttp://www.internet2.edu/henp; also see the Internet2; also see the Internet2
End-to-end Initiative: End-to-end Initiative: http://www.internet2.edu/e2ehttp://www.internet2.edu/e2e
A Short List: Coming Revolutions in Information Technology
A Short List: Coming Revolutions in Information Technology
Storage VirtualizationStorage Virtualization Grid-enabled Storage Resource Middleware (SRM)Grid-enabled Storage Resource Middleware (SRM) iSCSI (Internet Small Computer Storage Interface);iSCSI (Internet Small Computer Storage Interface);
Integrated with 10 GbE Integrated with 10 GbE Global File Systems Global File Systems Internet Information Software TechnologiesInternet Information Software Technologies
Global Information “Broadcast” ArchitectureGlobal Information “Broadcast” ArchitectureE.g the Multipoint Information Distribution E.g the Multipoint Information Distribution
Protocol ([email protected])Protocol ([email protected]) Programmable Coordinated Agent ArchitecturesProgrammable Coordinated Agent Architectures
E.g. Mobile Agent Reactive Spaces (MARS)E.g. Mobile Agent Reactive Spaces (MARS) by Cabri et al., University of Modena by Cabri et al., University of Modena
The “Data Grid” - Human InterfaceThe “Data Grid” - Human Interface Interactive monitoring and control of Grid resources Interactive monitoring and control of Grid resources
By authorized groups and individualsBy authorized groups and individualsBy Autonomous AgentsBy Autonomous Agents
HENP Major Links: Bandwidth Roadmap (Scenario) in Gbps
HENP Major Links: Bandwidth Roadmap (Scenario) in Gbps
Year Production Experimental Remarks
2001 0.155 0.622-2.5 SONET/SDH
2002 0.622 2.5 SONET/SDH DWDM; GigE Integ.
2003 2.5 10 DWDM; 1 + 10 GigE Integration
2004 10 2-4 X 10 Switch; Provisioning
2005 2-4 X 10 ~10 X 10; 40 Gbps
1st Gen. Grids
2006 ~10 X 10 or 1-2 X 40
~5 X 40 or ~20-50 X 10
40 Gbps Switching
2008 ~5 X 40 or
~20 X 10
~25 X 40 or ~100 X 10
2nd Gen Grids Terabit Networks
2010 ~Terabit ~MultiTerabit ~Fill One Fiber or Use a Few Fibers
HENP Scenario Limitations:Technologies and Costs
HENP Scenario Limitations:Technologies and Costs
Router Technology and Costs Router Technology and Costs (Ports and Backplane) (Ports and Backplane)
Computer CPU, Disk and I/O ChannelComputer CPU, Disk and I/O Channel Speeds to Send and Receive Data Speeds to Send and Receive Data
Link Costs: Unless Dark Fiber (?)Link Costs: Unless Dark Fiber (?) MultiGigabit Transmission Protocols MultiGigabit Transmission Protocols
End-to-End End-to-End ““100 GbE” Ethernet (or something else) by100 GbE” Ethernet (or something else) by
~2006: for LANs to match WAN speeds ~2006: for LANs to match WAN speeds
HENP Lambda Grids:Fibers for Physics
HENP Lambda Grids:Fibers for Physics
Problem: Extract “Small” Data Subsets of 1 to 100 Terabytes Problem: Extract “Small” Data Subsets of 1 to 100 Terabytes from 1 to 1000 Petabyte Data Storesfrom 1 to 1000 Petabyte Data Stores
Survivability of the HENP Global Grid System, with Survivability of the HENP Global Grid System, with hundreds of such transactions per day (circa 2007)hundreds of such transactions per day (circa 2007)requires that each transaction be completed in a requires that each transaction be completed in a relatively short time. relatively short time.
Example: Take 800 secs to complete the transaction. ThenExample: Take 800 secs to complete the transaction. Then Transaction Size (TB)Transaction Size (TB) Net Throughput (Gbps)Net Throughput (Gbps) 1 101 10 10 10010 100 100 1000 (Capacity of 100 1000 (Capacity of
Fiber Today) Fiber Today) Summary: Providing Switching of 10 Gbps wavelengthsSummary: Providing Switching of 10 Gbps wavelengths
within 2-3 years; and Terabit Switching within 5-7 yearswithin 2-3 years; and Terabit Switching within 5-7 yearswould enable “Petascale Grids with Terabyte transactions”,would enable “Petascale Grids with Terabyte transactions”,as required to fully realize the discovery potential of major HENP as required to fully realize the discovery potential of major HENP programs, as well as other data-intensive fields.programs, as well as other data-intensive fields.
10614 Hosts;6003 Registered Users in 60 Countries 41 (7 I2) Reflectors Annual Growth 2 to 3X
Networks, Grids and HENPNetworks, Grids and HENP Next generation 10 Gbps network backbones are Next generation 10 Gbps network backbones are
almost here: in the US, Europe and Japanalmost here: in the US, Europe and Japan First stages arriving, starting nowFirst stages arriving, starting now
Major transoceanic links at 2.5 - 10 Gbps in 2002-3Major transoceanic links at 2.5 - 10 Gbps in 2002-3 Network improvements are especially needed in Network improvements are especially needed in
Southeast Europe, So. America; and some other regions:Southeast Europe, So. America; and some other regions: Romania, Brazil; India, Pakistan, China; Africa Romania, Brazil; India, Pakistan, China; Africa
Removing regional, last mile bottlenecks and Removing regional, last mile bottlenecks and compromises in network quality are now compromises in network quality are now All on the critical pathAll on the critical path
Getting high (reliable; Grid) application performance Getting high (reliable; Grid) application performance across networks means!across networks means!
End-to-end monitoring; a coherent approach End-to-end monitoring; a coherent approach Getting high performance (TCP) toolkits in users’ Getting high performance (TCP) toolkits in users’
handshands Working in concert with AMPATH, Internet E2E, I2 Working in concert with AMPATH, Internet E2E, I2
HENP WG, DataTAG; the Grid projects and the GGFHENP WG, DataTAG; the Grid projects and the GGF
Some Extra Some Extra
Slides FollowSlides Follow
HENP Related Data Grid HENP Related Data Grid ProjectsProjects
ProjectsProjects PPDG IPPDG I USAUSA DOEDOE $2M$2M 1999-20011999-2001 GriPhyNGriPhyN USAUSA NSFNSF $11.9M + $1.6M$11.9M + $1.6M2000-20052000-2005 EU DataGridEU DataGrid EUEU ECEC €10M€10M 2001-20042001-2004 PPDG II (CP)PPDG II (CP) USAUSA DOEDOE $9.5M$9.5M 2001-20042001-2004 iVDGLiVDGL USAUSA NSFNSF $13.7M + $2M$13.7M + $2M 2001-20062001-2006 DataTAGDataTAG EUEU ECEC €4M€4M 2002-20042002-2004 GridPP GridPP UKUK PPARCPPARC >$15M>$15M 2001-20042001-2004 LCG (Ph1)LCG (Ph1) CERN MSCERN MS 30 MCHF30 MCHF 2002-20042002-2004
Many Other Projects of interest to HENPMany Other Projects of interest to HENP Initiatives in US, UK, Italy, France, NL, Germany, Japan, …Initiatives in US, UK, Italy, France, NL, Germany, Japan, … US and EU networking initiatives: US and EU networking initiatives: AMPATH, I2, DataTAG AMPATH, I2, DataTAG US Distributed Terascale Facility: US Distributed Terascale Facility:
($53M, 12 TeraFlops, 40 Gb/s network)($53M, 12 TeraFlops, 40 Gb/s network)
Mobile Agents: (Semi)-Autonomous, Mobile Agents: (Semi)-Autonomous, Goal Driven, AdaptiveGoal Driven, Adaptive Execute AsynchronouslyExecute Asynchronously Reduce Network Load: Local ConversationsReduce Network Load: Local Conversations Overcome Network Latency; Some OutagesOvercome Network Latency; Some Outages Adaptive Adaptive Robust, Fault Tolerant Robust, Fault Tolerant Naturally Heterogeneous Naturally Heterogeneous Extensible Concept: Extensible Concept: Coordinated Agent Coordinated Agent
Architectures Architectures
Beyond Traditional Architectures:Beyond Traditional Architectures:Mobile AgentsMobile Agents
““Agents are objects with rules and legs” -- D. TaylorAgents are objects with rules and legs” -- D. Taylor
Application
Se
rvic
e
Ag
entAgent
Ag
ent A
gen
tA
gen
t
Ag
ent
Ag
ent
Farm Monitor
Client(other service)
LookupService
LookupService
Registration
Farm Monitor
Discovery
Proxy
Component Factory
GUI marshaling
Code Transport
RMI data access
Push & Pullrsh & ssh
scripts; snmp
Globally Scalable Monitoring ServiceGlobally Scalable Monitoring Service
I. Legrand
RCMonitorService
National R&E Network ExampleGermany: DFN TransAtlanticConnectivity Q1 2002
STM 4
STM 16
STM 16
2 X OC12 Now: NY-Hamburg 2 X OC12 Now: NY-Hamburg and NY-Frankfurtand NY-Frankfurt
ESNet peering at 34 MbpsESNet peering at 34 Mbps Upgrade to 2 X OC48 expected Upgrade to 2 X OC48 expected
in Q1 2002 in Q1 2002 Direct Peering to Abilene and Direct Peering to Abilene and
Canarie expectedCanarie expected UCAID will add (?) another 2 UCAID will add (?) another 2
OC48’s; Proposing a Global OC48’s; Proposing a Global Terabit Research Network (GTRN) Terabit Research Network (GTRN)
FSU Connections via satellite:FSU Connections via satellite:Yerevan, Minsk, Almaty, BaikalYerevan, Minsk, Almaty, Baikal Speeds of 32 - 512 kbpsSpeeds of 32 - 512 kbps
SILK Project (2002): NATO fundingSILK Project (2002): NATO funding Links to Caucasus and CentralLinks to Caucasus and Central Asia (8 Countries) Asia (8 Countries)
Currently 64-512 kbpsCurrently 64-512 kbpsPropose VSAT for 10-50 X BW:Propose VSAT for 10-50 X BW: NATO + State Funding NATO + State Funding
National Research Networks in Japan
National Research Networks in Japan
SuperSINET SuperSINET Started operation January 4, 2002Started operation January 4, 2002 Support for 5 important areas:Support for 5 important areas:
HEP,HEP, Genetics, Nano-Technology, Genetics, Nano-Technology,Space/Astronomy, Space/Astronomy, GRIDsGRIDs
Provides 10 Provides 10 ’s:’s: 10 Gbps IP connection 10 Gbps IP connection Direct intersite GbE linksDirect intersite GbE links Some connections to Some connections to 10 GbE 10 GbE in JFY2002in JFY2002
HEPnet-J HEPnet-J Will be re-constructed with Will be re-constructed with MPLS-VPN in SuperSINET MPLS-VPN in SuperSINET
Proposal: Two TransPacific Proposal: Two TransPacific 2.5 Gbps Wavelengths, and 2.5 Gbps Wavelengths, and Japan-CERN Grid Testbed by ~2003 Japan-CERN Grid Testbed by ~2003
Tokyo
Osaka
Nagoya
Internet
Osaka U
Kyoto U
ICR
Kyoto-U
Nagoya U
NIFS
NIG
KEK
Tohoku U
IMS
U-TokyoNAO
U Tokyo
NII Hitot.
NII Chiba
IP
WDM path
IP router
OXC
ISAS
ICFA SCIC Meeting March 9 at CERN: Updates from Members
ICFA SCIC Meeting March 9 at CERN: Updates from Members
Abilene UpgradeAbilene Upgrade from 2.5 to 10 Gbps from 2.5 to 10 Gbps Additional scheduled lambdas planned Additional scheduled lambdas planned
for targeted applications for targeted applications US-CERNUS-CERN
Upgrade On Track: 2 X 155 to 622 Mbps in April; Upgrade On Track: 2 X 155 to 622 Mbps in April; Move to STARLIGHTMove to STARLIGHT
2.5G Research Lambda by this Summer: STARLIGHT-CERN2.5G Research Lambda by this Summer: STARLIGHT-CERN 2.5G Triangle between STARLIGHT (US), SURFNet (NL), 2.5G Triangle between STARLIGHT (US), SURFNet (NL),
CERNCERN SLAC + IN2P3 (BaBar) SLAC + IN2P3 (BaBar)
Getting 100 Mbps over 155 Mbps CERN-US LinkGetting 100 Mbps over 155 Mbps CERN-US Link 50 Mbps Over RENATER 155 Mbps Link, Limited by ESnet50 Mbps Over RENATER 155 Mbps Link, Limited by ESnet 600 Mbps Throughput is BaBar Target for this Year600 Mbps Throughput is BaBar Target for this Year
FNALFNAL Expect ESnet Upgrade to 622 Mbps this MonthExpect ESnet Upgrade to 622 Mbps this Month Plans for dark fiber to STARLIGHT, could be done in ~6 Plans for dark fiber to STARLIGHT, could be done in ~6
Months; Railway and Electric Co. providers consideredMonths; Railway and Electric Co. providers considered
ICFA SCIC: A&R Backbone and International Link Progress
ICFA SCIC: A&R Backbone and International Link Progress
GEANT Pan-European BackboneGEANT Pan-European Backbone ( (http://www.dante.net/geanthttp://www.dante.net/geant)) Now interconnects 31 countriesNow interconnects 31 countries Includes many trunks at 2.5 and 10 GbpsIncludes many trunks at 2.5 and 10 Gbps
UKUK 2.5 Gbps NY-London, with 622 Mbps to ESnet and Abilene2.5 Gbps NY-London, with 622 Mbps to ESnet and Abilene
SuperSINET (Japan):SuperSINET (Japan): 10 Gbps IP and 10 Gbps Wavelength 10 Gbps IP and 10 Gbps Wavelength Upgrade to Two 0.6 Gbps Links, to Chicago and SeattleUpgrade to Two 0.6 Gbps Links, to Chicago and Seattle Plan upgrade to 2 X 2.5 Gbps Connection to Plan upgrade to 2 X 2.5 Gbps Connection to
US West Coast by 2003US West Coast by 2003 CA*net4 (Canada):CA*net4 (Canada): Interconnect customer-owned dark fiber Interconnect customer-owned dark fiber
nets across Canada at 10 Gbps, starting July 2002 nets across Canada at 10 Gbps, starting July 2002 ““Lambda-Grids” by ~2004-5Lambda-Grids” by ~2004-5
GWIN (Germany):GWIN (Germany): Connection to Abilene Connection to Abilene to 2 X 2.5 Gbps in 2002 to 2 X 2.5 Gbps in 2002
RussiaRussia Start 10 Mbps link to CERN and ~90 Mbps to US NowStart 10 Mbps link to CERN and ~90 Mbps to US Now
[*] See “Macroscopic Behavior of the TCP Congestion Avoidance Algorithm,” Matthis, Semke, Mahdavi, Ott, Computer Communication Review 27(3), 7/1997
Throughput quality improvements:BWTCP < MSS/(RTT*sqrt(loss)) [*]
Throughput quality improvements:BWTCP < MSS/(RTT*sqrt(loss)) [*]
China Recent Improvement
80% Improvement/Year
Factor of 10 In 4 Years
Eastern Europe Keeping Up
TCP Protocol Study: Limits We determined Precisely
The parameters which limit the throughput over a high-BW, long delay (170 msec) network
How to avoid intrinsic limits; unnecessary packet loss
Methods Used to Improve TCP Linux kernel programming in order
to tune TCP parameters We modified the TCP algorithm A Linux patch will soon be available
Result: The Current State of the Art for Reproducible Throughput
125 Mbps between CERN and Caltech190 Mbps (one stream) between CERN
and Chicago shared on 2 155 Mbps links
Status: Ready for Tests at Higher BW (622 Mbps) this Month
Congestion window behavior of a TCP connection over the transatlantic line
3) Back to slow start(Fast Recovery couldn’t repair the lostThe packet lost is detected by timeout => go back to slow start cwnd = 2 MSS)
2) Fast Recovery (Temporary state to repair the lost)1) A packet is
lost New loss
Losses occur when the cwnd is larger than 3,5 Mbyte
TCP performance between CERN and Caltech
0
20
40
60
80
100
120
140
1 2 3 4 5 6 7 8 9 10 11
Connection number
Mbp
s
Without tunning
By tunning theSSTHRESHparameter
Maximizing US-CERN TCP Maximizing US-CERN TCP Throughput (S.Ravot, Caltech)Throughput (S.Ravot, Caltech)
Reproducible 125 Mbps Between
CERN and Caltech/CACR
Rapid Advances of Nat’l Backbones:Next Generation Abilene
Rapid Advances of Nat’l Backbones:Next Generation Abilene
Abilene partnership with Qwest extended Abilene partnership with Qwest extended through 2006through 2006
Backbone to be upgraded to 10-Gbps in phases, Backbone to be upgraded to 10-Gbps in phases, to be Completed by October 2003to be Completed by October 2003 GigaPoP Upgrade started in February 2002GigaPoP Upgrade started in February 2002
Capability for flexible Capability for flexible provisioning in support provisioning in support of future experimentation in optical networkingof future experimentation in optical networking In a multi- In a multi- infrastructure infrastructure
US CMS TeraGrid Seamless US CMS TeraGrid Seamless PrototypePrototype
Caltech/Wisconsin Condor/NCSA ProductionCaltech/Wisconsin Condor/NCSA Production Simple Job Launch from CaltechSimple Job Launch from Caltech
Authentication Using Globus Security Infrastructure (GSI)Authentication Using Globus Security Infrastructure (GSI) Resources Identified Using Globus Information Resources Identified Using Globus Information
Infrastructure (GIS)Infrastructure (GIS) CMSIM Jobs (Batches of 100, 12-14 Hours, 100 GB Output)CMSIM Jobs (Batches of 100, 12-14 Hours, 100 GB Output)
Sent to the Wisconsin Condor Flock Using Condor-G Sent to the Wisconsin Condor Flock Using Condor-G Output Files Automatically Stored in NCSA Unitree (Gridftp)Output Files Automatically Stored in NCSA Unitree (Gridftp)
ORCA Phase: Read-in and Process Jobs at NCSAORCA Phase: Read-in and Process Jobs at NCSA Output Files Automatically Stored in NCSA UnitreeOutput Files Automatically Stored in NCSA Unitree
Future: Multiple CMS Sites; Storage in Caltech HPSS Also,Future: Multiple CMS Sites; Storage in Caltech HPSS Also,Using GDMP (With LBNL’s HRM).Using GDMP (With LBNL’s HRM).
Animated Flow Diagram of the DTF Prototype:Animated Flow Diagram of the DTF Prototype: http://cmsdoc.cern.ch/~wisniew/infrastructure.htmlhttp://cmsdoc.cern.ch/~wisniew/infrastructure.html