Grid Technology: The Rough Guide
description
Transcript of Grid Technology: The Rough Guide
![Page 1: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/1.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
Grid Technology: The Rough Guide
Ashok Adiga (TACC) &Victor Bolet (Georgia State
University)
![Page 2: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/2.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
Outline• Introduction• Workshop Objectives• Grids 101• Grids in action• Workshop Outline• Open Discussions
![Page 3: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/3.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
Introduction• Workshop announcements
– Wireless access available– Dinner on Thursday– Ride back to hotel after Workshop on
Friday?• Introductions
– Speaker Introductions– Attendee introductions
• What would you like to get out of this workshop?
![Page 4: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/4.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
Workshop Objectives• Grid Technology is gaining acceptance
– Still evolving; but many success stories– New middleware/tools are becoming
available all the time• Target audience is technical people who
are interested in building grids but not sure how best to begin
• Objective is to provide an overview of current grid technology landscape via– Presentations & demonstrations– Hands-on session building “classroom grid”– Detailed coverage of selected topics,
overview of other topics with links to details
![Page 5: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/5.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
What is a Grid? “Resource sharing & coordinated
problem solving in dynamic … virtual organizations”
1. Enable integration of distributed service & resources2. Using general-purpose protocols & infrastructure3. To achieve useful qualities of service
“The Anatomy of the Grid”, Foster, Kesselman, Tuecke, 2001
![Page 6: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/6.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
What is a Grid? (cont’d) 1. Integrate and coordinate resources not
subject to centralized control …– addresses the issues of security, policy,
payment, membership, etc.2. … using standard, open, general-purpose
protocols and interfaces …– for authentication, authorization, resource
discovery, and resource access.3. … to deliver nontrivial qualities of service.
– relating to response time, throughput, availability, and security, and/or co-allocation of multiple resource types to meet complex user demands,
![Page 7: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/7.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
Grid Applications
Grid Technologies for Resource Integration &
Management
Grid Resources
inte
grat
ioninteroperability
DB Access
PDB portal
App Scheduler
PSE
portalUser-level Middleware and Tools
System-level Common Infrastructure
![Page 8: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/8.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
Grid Applications
Grid Services Architecture• Agree on interfaces, services
– Common infrastructure services act like a “grid OS”– Users interact with the Grid through higher-level, user-
friendly middleware layer
User-focusedmiddleware & tools
(commercial opportunities)
Grid Resources
DB Federation
PDB portal
App Scheduler
PSE
Chem portal
Common infrastructure
services(many open source)
Authentication, information, resource
access, resource mgmt, negotiation,
scheduling, monitoring, data transfer, etc., etc.
![Page 9: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/9.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
Available Grid Technologies• Several Grid solutions & toolkits exist
– Globus Toolkit (www.globus.org)• A toolkit of grid services and tools jointly developed by the Globus
Alliance in the US.– Gridlab (www.gridlab.org)
• A set of application-oriented Grid services and toolkits jointly developed by academic & commercial companies primarily based in Europe
– Gridbus (www.gridbus)• Grid computing and Business technologies developed at the University
of Melbourne.– Unicore (www.unicore.org)
• Grid solution created by the Unicore Forum, a developed by leading European Computing Centers and supporting hardware vendors
– Open Middleware Infrastructure Institute (OMII) (www.omii.ac.uk)• UK e-Science Grid solution
– Condor (www.cs.wisc.edu/condor)• Complete grid solution developed at the University of Wisconsin
– Grid MP (www.ud.com)• Commercial grid solution developed by United Devices
• We will primarily discuss the use of the Globus Toolkit & Condor
![Page 10: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/10.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
What Is the Globus Toolkit?• A Grid development environment
– Develop new OGSA-compliant Web Services– Develop applications using Java or C/C++ Grid APIs– Secure applications using basic security mechanisms
• A set of basic Grid services– Job submission/management– File transfer (individual, queued)– Database access– Data management (replication, metadata)– Monitoring/Indexing system information
• Tools and Examples• The prerequisite for many Grid community tools
![Page 11: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/11.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
“Standard Plumbing” for the Grid
• Not turnkey solutions, but building blocks and tools for application developers and system integrators.– Some components (e.g., file transfer) go
farther than others (e.g., remote job submission) toward end-user relevance.
• Since these solutions exist and others are already using them (and they’re free), it’s easier to reuse than to reinvent.– And compatibility with other Grid systems
comes for free!
![Page 12: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/12.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
WSAuthenticationAuthorization
Pre-WSAuthenticationAuthorization
Data MgmtSecurity CommonRuntime
Execution Mgmt
Information Services
ReliableFile
Transfer(RFT)
GridFTP
OGSA-DAI[Tech Preview]
Python WS Core[contribution]
C WS Core
CommunitySchedulerFramework
[contribution]
GridResource
Allocation Mgmt(WS GRAM)
Monitoring& Discovery
System(MDS4)
Java WS Core
Web ServicesComponents
Non-WS Components
CredentialManagement
ReplicaLocationService(RLS)
GridResource
Allocation Mgmt(Pre-WS GRAM)
Monitoring& Discovery
System(MDS2)
C CommonLibraries
XIO
DelegationService
CAS
GT2
GT3
GT3GT4
GT4
Globus Toolkit Components
![Page 13: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/13.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
Areas of Competence• “Connectivity Layer” Solutions
– Service Management (WSRF)– Monitoring/Discovery (WSRF and MDS)– Security (GSI and WS-Security)– Communication (XIO)
• “Resource Layer” Solutions– Computing / Processing Power (GRAM)– Data Access/Movement (GridFTP, OGSA-DAI)– In development: Telecontrol (NTCP/GTCP)
• “Collective Layer” Solutions– Data Management (RLS, MCS, OGSA-DAI)– Monitoring/Discovery (MDS)– Security (CAS)
![Page 14: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/14.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
Evolution of the GridIn
crea
sed
func
tiona
lity,
stan
dard
izat
ion
Time
Globus Toolkit
Open GridServices Arch
GGF: OGSI, WSRF, …(leveraging OASIS, W3C, IETF)
Multiple implementations,including Globus Toolkit
Defacto standardsGGF: GridFTP, GSI(leveraging IETF)
Customsolutions
X.509,LDAP,FTP, …
Web services
App-specificServices
![Page 15: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/15.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
Standards: Open Grid Services Architecture
• Define a service-oriented architecture…– the key to effective virtualization
• …to address vital Grid requirements– AKA utility, on-demand, system
management, collaborative computing, etc.• …building on Web service standards.
– extending those standards when needed
![Page 16: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/16.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
Standards Compliance• Web services: WS-I compliance
– All interfaces support WS-I Basic Profile, modulo use of WS-Addressing
• Securitya) WS-I Basic Security Profile (plaintext)b) IETF RFC 3820 Proxy Certificate
• GridFTP– GGF GFD 020
• Others in progress & being tracked– WSRF (OASIS), WS-Addressing (W3C), OGSA-DAI
(GGF), RLS (GGF)
![Page 17: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/17.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
Grid and Web Services Convergence
The definition of WSRF means that the Grid and Web services communities can move forward on a common base.
![Page 18: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/18.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
Grid Planning Considerations• All Grid technology is evolving rapidly.
– Web services standards– Grid interfaces– Grid implementations– Grid hosting services (ASP, SSP, etc.)
• Community is important!– Best practices (GGF, OASIS, etc.)– Open source (Linux, Axis, Globus, etc.)
• Applying community standards is vital.– Increases leverage– Mitigates (a bit) effects of rapid evolution– Paves the way for future integration/partnership
![Page 19: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/19.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
What End Users NeedSecure, reliable, on-demand access to data,software, people, and other resources(ideally all via a Web Browser!)
![Page 20: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/20.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
How it Really Happens
WebBrowser
ComputeServer
DataCatalog
DataViewer
Tool
Certificateauthority
ChatTool
CredentialRepository
WebPortal
ComputeServer
Resources implement standard access & management interfaces
Collective services aggregate &/or
virtualize resources
Users work with client applications
Application services organize VOs & enable
access to other services
Databaseservice
Databaseservice
Databaseservice
SimulationTool
Camera
CameraTelepresence
Monitor
RegistrationService
![Page 21: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/21.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
How it Really Happens• Implementations are provided by a mix
of– Application-specific code– “Off the shelf” tools and services– Tools and services from the Globus Toolkit– Tools and services from the Grid community
(compatible with GT)• Glued together by…
– Application development– System integration
![Page 22: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/22.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
How it Really Happens (without the Grid)
WebBrowser
ComputeServer
DataCatalog
DataViewer
Tool
Certificateauthority
ChatTool
CredentialRepository
WebPortal
ComputeServer
Resources implement standard access & management interfaces
Collective services aggregate &/or
virtualize resources
Users work with client applications
Application services organize VOs & enable
access to other services
Databaseservice
Databaseservice
Databaseservice
SimulationTool
Camera
CameraTelepresence
Monitor
RegistrationService
A
B
C
D
E
Application Developer
10
Off the Shelf 12
Globus Toolkit 0
Grid Community
0
![Page 23: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/23.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
How it Really Happens (with the Grid)
WebBrowser
ComputeServer
GlobusMCS/RLS
DataViewer
Tool
CertificateAuthority
CHEF ChatTeamlet
MyProxy
CHEF
ComputeServer
Resources implement standard access & management interfaces
Collective services aggregate &/or
virtualize resources
Users work with client applications
Application services organize VOs & enable
access to other services
Databaseservice
Databaseservice
Databaseservice
SimulationTool
Camera
CameraTelepresence
Monitor
Globus IndexService
GlobusGRAM
GlobusGRAM
GlobusDAI
GlobusDAI
GlobusDAI
Application Developer
2
Off the Shelf 9
Globus Toolkit 4
Grid Community
4
![Page 24: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/24.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
Examples of Grids in Action• Campus level grid projects
– UT Grid (University of Texas)– MGrid (University of Michigan)– …
• Regional initiatives– SURAgrid
• National/International Grids– NSF Teragrid– OSG (Open Sciences Grid)– EGEE (Enabling Grids for E-Science in Europe)
• Volunteer grids– SETI@home– World Community Grid– Grid.org
![Page 25: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/25.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
NSF TeraGrid• 40+ teraflops
compute • 1+ petabyte
online storage• 10-40Gb/s
networking
• Heterogeneous compute, storage, visualization resources• Globus-based grid middleware• Any US researcher can apply for allocations• http://www.teragrid.org for more information
![Page 26: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/26.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
The TeraGrid Strategy• Creating a unified user
environment…– Single user support
resources.– Single authentication point– Common software
functionality– Common job management
infrastructure– Globally-accessible data
storage• …across heterogeneous
resources– 7+ computing architectures– 5+ visualization resources– diverse storage technologies
• Create a unified national HPC infrastructure that is both heterogeneous and extensible
![Page 27: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/27.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
Current TeraGrid Usage Scenarios
• “Traditional” massively parallel jobs– Tightly-coupled interprocessor communication– storing vast amounts of data remotely– remote visualization
• Thousands of independent jobs– Automatically scheduled amongst many
TeraGrid machines– Use data from a distributed data collection
• Multi-site parallel jobs– Compute upon many TeraGrid sites
simultaneouslyTeraGrid is working to enable more!
![Page 28: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/28.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
SURAgrid• A “beyond regional” initiative in support of
SURA regional strategy“Mini-About” SURA– SURA region: 16 states & DC, from Delaware to Texas– SURA membership: 62 research universities, mostly
within the region– SURA mission: Foster excellence in scientific research,
strengthen capabilities, provide training opportunities• Evolved from the NMI Testbed Grid
project, an outgrowth of SURA’s management of the NMI Integration Testbed Program– http://www1.sura.org/3000/NMI-Testbed.html
![Page 29: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/29.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
SURAgrid GoalsSURAgrid: Organizations collaborating to bring
grids to the level of seamless, shared infrastructure
Goals:To develop scalable infrastructure that
leverages local institutional identity and authorization while managing access to shared resources
To promote the use of this infrastructure for the broad research and education community
To provide a forum for participants to gain additional experience with grid technology, and participate in collaborative project development
![Page 30: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/30.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
SURAgrid Participants• University of Alabama at
Birmingham*• University of Alabama in
Huntsville*• University of Arkansas*• University of Florida*• George Mason University*• Georgia State University* • Great Plains Network• University of Kentucky*• University of Louisiana at
Lafayette*• Louisiana State University*• University of Michigan• Mississippi Center for
SuperComputing Research*
• University of North Carolina, Charlotte
• North Carolina State University*
• Old Dominion University*• University of South Carolina*• University of Southern
California• Southeastern Universities
Research Association (SURA)• Texas A&M University*• Texas Advanced Computing
Center (TACC)*• Texas Tech• Tulane University* • Vanderbilt University* • University of Virginia*
*SURA member
![Page 31: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/31.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
SURAgrid Resources
![Page 32: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/32.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
SURAgrid Applications• Multiple Genome Alignment (GSU, UAB, UVA) • Task Farming (LSU)• Muon Detector Grid (GSU)• BLAST (UAB)• ENDYNE (TTU)• SCOOP/ADCIRC (UNC, RENCI, MCNC,
SCOOP partners, SURAgrid partners)• … Potential applications…
![Page 33: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/33.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
Workshop Agenda• Introduction & Overview• Configuring Resources for the Grid• Authentication, Authorization, &
Identity Issues in Grids• Level Grid Services• High Level Grid Services• Grid Packages• User Interfaces• Grid Application Toolkits• Hands-on Session
![Page 34: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/34.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
Hands-on Session• Build Classroom Grid during workshop
– Using some of the technologies described in the presentations
– Grid services pre-installed or installed during sessions
– Laptop “servers” installed & configured during hands-on session
• Try out some basic grid operations on this grid– Security, data & job management, resource
monitoring, grid portals
![Page 35: Grid Technology: The Rough Guide](https://reader036.fdocuments.in/reader036/viewer/2022062521/5681548a550346895dc29a16/html5/thumbnails/35.jpg)
December 8 & 9, 2005, Austin, TXSURA Cyberinfrastructure Workshop Series: Grid Technology: The Rough Guide
Questions?