Based on - medicalbioinformatics.de · More GridFTP features • Auto tuning of parameters •...
Transcript of Based on - medicalbioinformatics.de · More GridFTP features • Auto tuning of parameters •...
4/17/2013
1
1
Introduction to Grid Computing
Based on:Grid Intro and Fundamentals Review
Talk by Gabrielle Allen
Talk by Laura Bright / Bill Howe
2
3
Overview
• Background: What is the Grid?• Related technologies• Grid applications• Communities
• Grid Tools• Case Studies
4/17/2013
2
4
PUMA: Analysis of Metabolism
PUMA Knowledge Base
Information about proteins analyzed against ~2 million gene sequences
Analysis on Grid
Involves millions of BLAST, BLOCKS, and
other processesNatalia Maltsev et al.http://compbio.mcs.anl.gov/puma2 4
5
Initial driver: High Energy Physics
Tier2 Centre ~1 TIPS
Online System
Offline Processor Farm
~20 TIPS
CERN Computer Centre
FermiLab ~4 TIPSFrance Regional Centre
Italy Regional Centre
Germany Regional Centre
InstituteInstituteInstituteInstitute ~0.25TIPS
Physicist workstations
~100 MBytes/sec
~100 MBytes/sec
~622 Mbits/sec
~1 MBytes/sec
There is a “bunch crossing” every 25 nsecs.
There are 100 “triggers” per second
Each triggered event is ~1 MByte in size
Physicists work on analysis “channels”.
Each institute will have ~10 physicists working on one or more channels; data for these channels should be cached by the institute server
Physics data cache
~PBytes/sec
~622 Mbits/sec or Air Freight (deprecated)
Tier2 Centre ~1 TIPS
Tier2 Centre ~1 TIPS
Tier2 Centre ~1 TIPS
Caltech ~1 TIPS
~622 Mbits/sec
Tier 0
Tier 1
Tier 2
Tier 4
1 TIPS is approximately 25,000
SpecInt95 equivalents
Image courtesy Harvey Newman, Caltech 5
6
Computing clusters have commoditized supercomputing
Cluster Management“frontend”
Tape Backup robots
I/O Servers typically RAID fileserver
Disk Arrays Lots of WorkerNodes
A few Headnodes, gatekeepers and
other service nodes
6
4/17/2013
3
7
What is a Grid?• Many definitions exist in the literature• Early defs: Foster and Kesselman, 1998
“A computational grid is a hardware and software infrastructure that provides dependable, consistent, pervasive, and inexpensive access to high-end computational facilities”
• Kleinrock 1969:“We will probably see the spread of ‘computer utilities’, which, like present electric and telephone utilities, will service individual homes and offices across the country.”
8
3-point checklist (Foster 2002)
1. Coordinates resources not subject to centralized control
2. Uses standard, open, general purpose protocols and interfaces
3. Deliver nontrivial qualities of service• e.g., response time, throughput, availability,
security
9
Grid Architecture
Autonomous, globally distributed computers/clusters
4/17/2013
4
10
Why do we need Grids?
• Many large-scale problems cannot be solved by a single computer
• Globally distributed data and resources
11
Background: Related technologies
• Cluster computing• Peer-to-peer computing• Internet computing
12
Cluster computing
• Idea: put some PCs together and get them to communicate
• Cheaper to build than a mainframe supercomputer
• Different sizes of clusters• Scalable – can grow a cluster by adding
more PCs
4/17/2013
5
13
Cluster Architecture
14
Peer-to-Peer computing
• Connect to other computers• Can access files from any computer on the
network• Allows data sharing without going through
central server• Decentralized approach also useful for
Grid
15
Peer to Peer architecture
4/17/2013
6
16
Internet computing
• Idea: many idle PCs on the Internet• Can perform other computations while not
being used• “Cycle scavenging” – rely on getting free
time on other people’s computers• Example: SETI@home• What are advantages/disadvantages of
cycle scavenging?
17
Some Grid Applications
• Distributed supercomputing• High-throughput computing• On-demand computing• Data-intensive computing
• Collaborative computing
18
Distributed Supercomputing
• Idea: aggregate computational resources to tackle problems that cannot be solved by a single system
• Examples: climate modeling, computational chemistry
• Challenges include:– Scheduling scarce and expensive resources– Scalability of protocols and algorithms– Maintaining high levels of performance across
heterogeneous systems
4/17/2013
7
19
High-throughput computing
• Schedule large numbers of independent tasks
• Goal: exploit unused CPU cycles (e.g., from idle workstations)
• Unlike distributed computing, tasks loosely coupled
• Examples: parameter studies, cryptographic problems
20
On-demand computing
• Use Grid capabilities to meet short-term requirements for resources that cannot conveniently be located locally
• Unlike distributed computing, driven by cost-performance concerns rather than absolute performance
• Dispatch expensive or specialized computations to remote servers
21
Data-intensive computing
• Synthesize data in geographically distributed repositories
• Synthesis may be computationally and communication intensive
• Examples:– High energy physics generate terabytes of
distributed data, need complex queries to detect “interesting” events
– Distributed analysis of Sloan Digital Sky Survey data
4/17/2013
8
22
Collaborative computing
• Enable shared use of data archives and simulations
• Examples:– Collaborative exploration of large geophysical
data sets
• Challenges:– Real-time demands of interactive applications
– Rich variety of interactions
23
Grid Communities• Who will use Grids?• Broad view
– Benefits of sharing outweigh costs
– Universal, like a power Grid
• Narrow view– Cost of sharing across institutional boundaries
is too high
– Resources only shared when incentive to do so
– Grid will be specialized to support specific communities with specific goals
24
Grid Users
• Many levels of users– Grid developers
– Tool developers
– Application developers
– End users
– System administrators
4/17/2013
9
25
Some Grid challenges
• Data movement• Data replication• Resource management• Job submission
26
Some Grid-Related Projects
• Globus• Condor• Nimrod-G
27
What is a grid made of ? Middleware.
Grid Protocols
Grid Resources dedicatedby UC, IU, Boston
GridStorage
GridMiddleware
Com
putingC
luster
Grid resource time purchasedfrom commercial provider
GridMiddleware
Com
putingC
luster
Grid resources sharedby OSG, LCG, NorduGRID
GridMiddleware
Com
putingC
luster
Grid Client
ApplicationUser
Interface
GridMiddleware
Resource,WorkflowAnd DataCatalogs
GridStorage
GridStorage
• Security to control access and protect communication (GSI)• Directory to locate grid sites and services: (VORS, MDS)• Uniform interface to computing sites (GRAM)• Facility to maintain and schedule queues of work (Condor-G)• Fast and secure data set mover (GridFTP, RFT)• Directory to track where datasets live (RLS)
27
4/17/2013
10
28
Globus Grid Toolkit• Open source toolkit for building Grid systems
and applications
• Enabling technology for the Grid • Share computing power, databases, and other
tools securely online • Facilities for:
– Resource monitoring
– Resource discovery
– Resource management
– Security
– File management
29
Data Management in Globus Toolkit
• Data movement– GridFTP
– Reliable File Transfer (RFT)
• Data replication– Replica Location Service (RLS)
– Data Replication Service (DRS)
30
GridFTP• High performance, secure, reliable data
transfer protocol
• Optimized for wide area networks• Superset of Internet FTP protocol• Features:
– Multiple data channels for parallel transfers– Partial file transfers
– Third party transfers
– Reusable data channels
– Command pipelining
4/17/2013
11
31
More GridFTP features
• Auto tuning of parameters• Striping
– Transfer data in parallel among multiple senders and receivers instead of just one
• Extended block mode– Send data in blocks
– Know block size and offset
– Data can arrive out of order
– Allows multiple streams
32
Striping Architecture
• Use “Striped” servers
33
Limitations of GridFTP
• Not a web service protocol (does not employ SOAP, WSDL, etc.)
• Requires client to maintain open socket connection throughout transfer– Inconvenient for long transfers
• Cannot recover from client failures
4/17/2013
12
34
GridFTP
35
Reliable File Transfer (RFT)
• Web service with “job-scheduler” functionality for data movement
• User provides source and destination URLs• Service writes job description to a database
and moves files• Service methods for querying transfer status
36
RFT
4/17/2013
13
37
Replica Location Service (RLS)
• Registry to keep track of where replicas exist on physical storage system
• Users or services register files in RLS when files created
• Distributed registry– May consist of multiple servers at different sites
– Increase scale
– Fault tolerance
38
Replica Location Service (RLS)• Logical file name – unique identifier for contents of file
• Physical file name – location of copy of file on storage system
• User can provide logical name and ask for replicas
• Or query to find logical name associated with physical file location
39
Data Replication Service (DRS)• Pull-based replication capability• Implemented as a web service• Higher-level data management service built on
top of RFT and RLS• Goal: ensure that a specified set of files exists
on a storage site• First, query RLS to locate desired files• Next, creates transfer request using RFT
• Finally, new replicas are registered with RLS
4/17/2013
14
40
Condor
• Original goal: high-throughput computing• Harvest wasted CPU power from other
machines• Can also be used on a dedicated cluster• Condor-G – Condor interface to Globus
resources
41
Condor• Provides many features of batch systems:
– job queueing
– scheduling policy
– priority scheme
– resource monitoring
– resource management
• Users submit their serial or parallel jobs • Condor places them into a queue• Scheduling and monitoring• Informs the user upon completion
42
Nimrod-G• Tool to manage execution of parametric studies
across distributed computers
• Manages experiment– Distributing files to remote systems
– Performing the remote computation
– Gathering results
• User submits declarative plan file– Parameters, default values, and commands
necessary for performing the work
• Nimrod-G takes advantage of Globus toolkit features
4/17/2013
15
43
Nimrod-G Architecture
Security
44
Grid security is a crucial component
• Resources are typically valuable• Problems being solved might be sensitive• Resources are located in distinct
administrative domains– Each resource has own policies, procedures,
security mechanisms, etc.
• Implementation must be broadly available & applicable– Standard, well-tested, well-understood
protocols; integrated with wide variety of tools45
4/17/2013
16
Security Services• Forms the underlying communication
medium for all the services
• Secure Authentication and Authorization• Single Sign-on
– User explicitly authenticates only once – then single sign-on works for all service requests
• Uniform Credentials• Example: GSI (Grid Security
Infrastructure)
46
• Authentication stops imposters• Examples of authentication:
– Username and password
– Passport
– ID card
– Public keys or certificates
– Fingerprint
Authentication means identifying that you are whom you claim to be
47
• Is this device allowed to access to this service?
• Read, write, execute permissions in Unix• Access conrol lists (ACLs) provide more
flexible control• Special “callouts” in the grid stack in job
and data management perform authorization checks.
Authorization controls what you are allowed to do.
48
4/17/2013
17
Job and resource management
49
Job Management Services provide a standard interface to remote resources
• Includes CPU, Storage and Bandwidth
• Globus component is Globus Resource Allocation Manager (GRAM)
• The primary Condor grid client component isCondor-G
• Other needs:– scheduling– monitoring– job migration– notification
50
GRAM provides a uniform interface to diverse resource scheduling systems.
User
Grid
VO
VO
VO
VO
Condor
PBS
LSF
UNIX fork()
GRAM
Site A
Site B
Site C
Site D
51
4/17/2013
18
GRAM: What is it?• Globus Resource Allocation Manager• Given a job specification:
– Create an environment for a job
– Stage files to and from the environment
– Submit a job to a local resource manager
– Monitor a job
– Send notifications of the job state change
– Stream a job’s stdout/err during execution
52
A “Local Resource Manager” is a batch system for running jobs across a computing cluster
• In GRAM• Examples:
– Condor– PBS– LSF– Sun Grid Engine
• Most systems allow you to access “fork”– Default behavior– It runs on the gatekeeper:
• A bad idea in general, but okay for testing
53
Managing your jobs• We need something more than just the basic
functionality of the globus job submission commands
• Some desired features– Job tracking– Submission of a set of inter-dependant jobs– Check-pointing and Job resubmission capability– Matchmaking for selecting appropriate resource for
executing the job• Options: Condor, PBS, LSF, …
54
4/17/2013
19
Grid Workflow
55
56
A typical workflow pattern in image analysis runs many filtering apps.
3a.h
align_warp/1
3a.i
3a.s.h
softmean/9
3a.s.i
3a.w
reslice/2
4a.h
align_warp/3
4a.i
4a.s.h 4a.s.i
4a.w
reslice/4
5a.h
align_warp/5
5a.i
5a.s.h 5a.s.i
5a.w
reslice/6
6a.h
align_warp/7
6a.i
6a.s.h 6a.s.i
6a.w
reslice/8
ref.h ref.i
atlas.h atlas.i
slicer/10 slicer/12 slicer/14
atlas_x.jpg
atlas_x.ppm
convert/11
atlas_y.jpg
atlas_y.ppm
convert/13
atlas_z.jpg
atlas_z.ppm
convert/15
Workflow courtesy James Dobson, Dartmouth Brain Imaging Center 56
Workflows can process vast datasets.• Many HEP and Astronomy experiments consist of:
– Large datasets as inputs (find datasets)– “Transformations” which work on the input datasets
(process)– The output datasets (store and publish)
• The emphasis is on the sharing of the large datasets
• Transformations are usually independent and can be parallelized. But they can vary greatly in duration.
Montage Workflow: ~1200 jobs, 7 levelsMosaic of M42 created on TeraGrid
= DataTransfer
= ComputeJob 57
4/17/2013
20
58
Virtual data model enables workflow to abstract grid details.
58
59
Grid Case Studies
• Earth System Grid• LIGO• TeraGrid
60
Earth System Grid
• Provide climate studies scientists with access to large datasets
• Data generated by computational models – requires massive computational power
• Most scientists work with subsets of the data
• Requires access to local copies of data
4/17/2013
21
61
ESG Infrastructure
• Archival storage systems and disk storage systems at several sites
• Storage resource managers and GridFTP servers to provide access to storage systems
• Metadata catalog services• Replica location services• Web portal user interface
62
Earth System Grid
63
Earth System Grid Interface
4/17/2013
22
64
Laser Interferometer Gravitational Wave Observatory (LIGO)
• Instruments at two sites to detect gravitational waves
• Each experiment run produces millions of files• Scientists at other sites want these datasets on
local storage• LIGO deploys RLS servers at each site to
register local mappings and collect info about mappings at other sites
65
Large Scale Data Replication for LIGO
• Goal: detection of gravitational waves• Three interferometers at two sites• Generate 1 TB of data daily• Need to replicate this data across 9 sites
to make it available to scientists• Scientists need to learn where data items
are, and how to access them
66
LIGO
4/17/2013
23
67
LIGO Solution
• Lightweight data replicator (LDR)• Uses parallel data streams, tunable TCP
windows, and tunable write/read buffers• Tracks where copies of specific files can
be found • Stores descriptive information (metadata)
in a database – Can select files based on description rather
than filename
68
TeraGrid
• NSF high-performance computing facility• Nine distributed sites, each with different
capability , e.g., computation power, archiving facilities, visualization software
• Applications may require more than one site
• Data sizes on the order of gigabytes or terabytes
69
TeraGrid
4/17/2013
24
70
TeraGrid
• Solution: Use GridFTP and RFT with front end command line tool (tgcp)
• Benefits of system:– Simple user interface
– High performance data transfer capability
– Ability to recover from both client and server software failures
– Extensible configuration
71
TGCP Details
• Idea: hide low level GridFTP commands from users
• Copy file smallfile.dat in a working directory to another system:tgcp smallfile.dat tg-login.sdsc.teragrid.org:/users/ux454332
• GridFTP command:globus-url-copy -p 8 -tcp-bs 1198372 \gsiftp://tg-gridftprr.uc.teragrid.org:2811/home/navarro/smallfile.dat \
gsiftp://tg-login.sdsc.teragrid.org:2811/users/ux454332/smallfile.dat
72
The reality
• We have spent a lot of time talking about “The Grid”
• There is “the Web” and “the Internet”• Is there a single Grid?
4/17/2013
25
73
The reality
• Many types of Grids exist• Private vs. public• Regional vs. Global• All-purpose vs. particular scientific
problem
74
Conclusion: Why Grids?
• New approaches to inquiry based on– Deep analysis of huge quantities of data
– Interdisciplinary collaboration
– Large-scale simulation and analysis
– Smart instrumentation
– Dynamically assemble the resources to tackle a new scale of problem
• Enabled by access to resources & services without regard for location & other barriers
74
75
Grids: Because Science Takes a Village …
• Teams organized around common goals– People, resource, software, data, instruments…
• With diverse membership & capabilities– Expertise in multiple areas required
• And geographic and political distribution– No location/organization possesses all required skills
and resources
• Must adapt as a function of the situation– Adjust membership, reallocate responsibilities,
renegotiate resources
75