Post on 19-Jan-2016
description
The Globus Toolkit:Description and
Applications Review
Steve Tuecke & Ian FosterArgonne National Laboratory
The University of Chicago
Globus Co-PI: Carl Kesselman, USC/ISI
Overview
The need for Grid services The Globus toolkit Globus application case studies
– Microtomography: on-line instrumentation
– SF-Express and Overflow: distributed supercomputing
– CAVERNsoft: collaborative engineering
– Nimrod-G: high-throughput computing
– ECCE’: problem solving environment Summary
Creating a Usable Grid :Grid Services (“Middleware”)
Standard grid services that– Provide uniform, high-level access to a wide
range of resources (including networks)
– Address interdomain issues of security, policy, etc.
– Permit application-level management and monitoring of end-to-end performance
Middleware-level and higher-level APIs and tools targeted at application programmers– Map between application and Grid
Grid Services Architecture
Grid FabricLayer
Applications
Transport Multicast
Instrumentation Control interfaces QoS mechanisms
Grid ServicesLayer
Information Resource mgmt
Security Data access Fault detection
. . .
. . .
High-energyphysics data
analysis Regionalclimate studies
Collaborativeengineering
Parameterstudies
On-lineinstrumentation
ApplicationToolkit Layer
Distributedcomputing
Data-intensive
Collab.design
Remoteviz
Remote control
The Globus Project: Argonne, USC/ISI, NCSA, Aerospace, NASA Ames, LBNL,
others Basic research in grid-related technologies
– Resource management, security, adaptation, etc.
Development of Globus toolkit– Core services for grid-enabled tools & applns
Construction of large grid testbed: GUSTO– Largest grid testbed in terms of sites & apps
Application experiments– Tele-immersion, distributed computing, etc.
GUSTO Testbed Map
Globus Grid Services
The Globus toolkit provides a range of basic Grid services– Security, information, fault detection,
communication, resource management, ... These services are simple and orthogonal
– Can be used independently, mix and match
– Programming model independent For each there are well-defined APIs Standards are used extensively
– E.g., LDAP, GSS-API, X.509, ...
Grid Services Layer (1)
Grid Security Infrastructure– Single-sign on, run anywhere [if authorized]
– PKI, X.509 certificates
– Identity/credential mapping at each resource
– Allows programs to act as user for limited period: delegation of rights
Grid Services Layer (2)
Grid Information Service– Currently an LDAP-based directory service
– Publish structure and state info, dynamic performance info, software info, etc., etc.
– Resource discovery: “find me an X with property Y available at time T”
– Auto-configuration: “tell me what I need to know to use A efficiently/securely/...”
– Gateways to other data sources required
– Example of integrating “middleware” service
Grid Services Layer (3)
Access to remote data (GASS)– Uniform access to diverse storage management
systems
– Cache management
– Integration with SRB, DPSS, HPSS Communication (Nexus)
– Application-level interfaces to comm services
– Multiple methods: reliable/unreliable, IP/other, unicast/multicast
– QoS interfaces
Grid Services Layer (4)
Globus Resource Allocation Manager (GRAM)– Uniform interface to resource management
Globus Arch. for Reservation and Allocation– Co-allocation of compute resources
– Immediate and advance reservation of network and computers in prototype form
Fault detection service Network measurement tools Code management and distribution infrastructure
Application Toolkit Layer: e.g.
Message Passing Interface– Multi-method communication, specialized
CAVERNsoft– Shared state for collaborative environments
Condor, Nimrod-G– High-throughput computing
Parallel Application Workspace (PAWS)– High-speed parallel transfers for coupled apps
Globus Progress
Selected “Grid Services” are being migrated into the infrastructure– Grid information service
– Grid security infrastructure
– Grid resource management services Simultaneously these and other Globus services
are being applied to develop– Grid-enabled tools
– Grid-enabled applications An ongoing iterative refinement process
tomographic reconstruction
real-timecollection
wide-areadissemination
desktop & VR clients with shared controls
Advanced Photon Source
Case Study 1:Online Instrumentation
archival storage
DOE X-ray source grand challenge: ANL, USC/ISI, NIST, U.Chicago
CMT Processing
+ GRAM Component startup
+ GSI Uniform access to instrument, vizand processing machines
+ Nexus Communication between pipelinestates and within stages
+ GASS Access to remote archived data,storage of processed data
+ Nexus (2) Pipeline control
+ GASS (2) Executable staging
+ HBM Monitor status of pipeline
Now
Additional Opportunities
End-to-end advance reservation of network, storage, computers
Dynamic discovery and allocation of supercomputers, networks, etc.
Adaptive determination of display resolution, reconstruction fidelity, etc., etc.
Reliable multicast for data, control, video Access control and discovery for collaborative
sessions Integration with mass storage systems
Case Study 2:Distributed Supercomputing
SF-Express Distributed Interactive Simulation: Caltech, USC/ISI
Starting point: SF-Express parallel simulation code
Globus mechanisms for– Resource allocation
– Distributed startup
– I/O and configuration
– Fault detection 100K vehicles (2002 goal) using 13
computers, 1386 nodes, 9 sites
NCSAOrigin
CaltechExemplar
CEWESSP
MauiSP
OVERFLOW with latency-tolerant algorithms
MPICH-G “Grid-enabled” message passing
Globus services
Security Directory Scheduling
Process mgmt Communication
ARC SGI O2000(California)
Argonne SGI O2000(Illinois)
OVERFLOW simulation: NASA Ames
Case Study 3:Collaborative Engineering
CAVERNsoft: UIC Electronic Visualization Laboratory
Manipulate shared virtual space, with– Simulation components
– Multiple flows: Control, Text, Video, Audio, Database, Simulation, Tracking, Haptics, Rendering
Uses Globus comms: (un)reliable uni/multicast
Future: Security, QoS, allocation, reservation
Case Study 4:High-Throughput Computing
Nimrod-G: Monash University
CostDeadline
AvailableMachines
Schedule many independent tasks (e.g., parameter study)
Uses Globus security, discovery, data access, scheduling
Future: Reservation, accounting, code management, etc.
Problem solving environment for comp. chemistry
Globus services used for authentication, remote job submission, monitoring, and control
Future: distributed data archive, resource discovery, charging
Case Study 5:Problem Solving Environment
ECCE’: Pacific Northwest National Laboratory
Summary
Grids require Grid services that make resources accessible and usable and Grid toolkits for application development
The Globus project is building essential services and partnering with tool developers
Significant success stories in a range of problem classes
We’re looking forward to working with applications throughout the community!