INFSO-RI-508833
Enabling Grids for E-sciencE
www.eu-egee.org
Concepts of grid computing
Concepts of Grid Computing, Towards e-Research, St Andrews 2
Enabling Grids for E-sciencE
INFSO-RI-508833
Acknowledgements
• This talk was prepared by Mike Mineter of NeSC and includes slides from previous tutorials and talks delivered by:– Dave Berry, Richard Hopkins, Guy Warner (National e-Science Centre)– the EDG training team– Ian Foster, Argonne National Laboratories– Jeffrey Grethe, SDSC– EGEE colleagues – Mark Baker, The Distributed Systems Group, University of Portsmouth,
http://dsg.port.ac.uk/mab
• Talks at 3rd EGEE conference by– Kyriakos Baxevanidis,Deputy Head,Unit of Research
Infrastructures,European Commission, DG INFSO– Dr Spyros Konidaris, European Commission – DG INFSO
Concepts of Grid Computing, Towards e-Research, St Andrews 3
Enabling Grids for E-sciencE
INFSO-RI-508833
Goals of this module
• To introduce the concepts of Grid computing assuming no previous knowledge
Concepts of Grid Computing, Towards e-Research, St Andrews 4
Enabling Grids for E-sciencE
INFSO-RI-508833
Contents
• “The Grid” vision• What is “a grid” ?• Drivers of grid computing• Current status of grids• The basis: authentication, authorisation, security
Concepts of Grid Computing, Towards e-Research, St Andrews 5
Enabling Grids for E-sciencE
INFSO-RI-508833
The Grid Metaphor
GRID
MIDDLEWARE
Mobile Access
Supercomputer, PC-Cluster
Data-storage, Sensors, Experiments
Workstation
Visualising
Internet, networks
Concepts of Grid Computing, Towards e-Research, St Andrews 6
Enabling Grids for E-sciencE
INFSO-RI-508833
The grid vision• The grid vision is of “Virtual
computing” (+ information services to locate computation, storage resources)
– Compare: The web: “virtual documents” (+ search engine to locate them)
• MOTIVATION: collaboration through sharing resources (and expertise) to expand horizons of – Research – Commerce – engineering, …
“the knowledge economy”– Public service – health,
environment,…
Concepts of Grid Computing, Towards e-Research, St Andrews 7
Enabling Grids for E-sciencE
INFSO-RI-508833
Contents
• “The Grid” vision• What is “a grid” ?
Concepts of Grid Computing, Towards e-Research, St Andrews 8
Enabling Grids for E-sciencE
INFSO-RI-508833
“A grid”• The initial vision: “The Grid”• The present reality: Many
“grids”• Each grid is an infrastructure
enabling one or more “virtual organisations” to share computing resources
• What’s a VO?– People in different
organisations seeking to cooperate and share resources across their organisational boundaries
• Why establish a Grid?– Share data– Pool computers– Collaborate
Institute A
VO
Institute CInstitute B
Institute D
Concepts of Grid Computing, Towards e-Research, St Andrews 9
Enabling Grids for E-sciencE
INFSO-RI-508833
The Single Computer
• The Operating System enables easy use of – Input devices– Processor– Disks– Display– Any other attached devices
Disks, Processor, Memory, …
Application Software
Operating
System
Concepts of Grid Computing, Towards e-Research, St Andrews 10
Enabling Grids for E-sciencE
INFSO-RI-508833
Resources on a Local Area Network
User just perceives “shared resources”, with no regard to location in the organisation:
- Authenticated by username / password
- Authorised to use own files,…
Resources connected by a LAN
Application Software
Operating System on each computer
Middleware for sharingcomputers, servers, printers, …
Concepts of Grid Computing, Towards e-Research, St Andrews 11
Enabling Grids for E-sciencE
INFSO-RI-508833
Resources on a grid
Resources connected by internet
Application Software
Operating System on each resource
Grid Middleware on each resource
Grid Middleware: “collective services”
Interface between app. and grid
Concepts of Grid Computing, Towards e-Research, St Andrews 12
Enabling Grids for E-sciencE
INFSO-RI-508833
A grid• Grid middleware
runs on each shared resource– Data storage– (Usually) batch
jobs on pools of processors
• Users join VO’s• Virtual organisation
negotiates with sites to agree access to resources
• Distributed services (both people and middleware) enable the grid
INTERNET
Concepts of Grid Computing, Towards e-Research, St Andrews 13
Enabling Grids for E-sciencE
INFSO-RI-508833
What characterises a grid?
• Co-ordinated resource sharing – No centralised point of control – Different administrative domains.
• Standard, open, general-purpose protocols and interfaces– NOT specific to an application– EGEE, NGS support multiple VO’s
• Delivering non-trivial qualities of service – Co-ordinated to deliver combined services,
greater than sum of the individual components
• http://www.gridtoday.com/02/0722/100136.html
Concepts of Grid Computing, Towards e-Research, St Andrews 14
Enabling Grids for E-sciencE
INFSO-RI-508833
The components of a Grid
• Resources– networking, computers, storage, data, instruments, …
• Grid Middleware– the “operating system of the grid”
• Operations infrastructure– Run enabling services (people + software)
• Virtual Organization management– Procedures for gaining access to resources
Concepts of Grid Computing, Towards e-Research, St Andrews 15
Enabling Grids for E-sciencE
INFSO-RI-508833
Key concepts
• Virtual organisation: people and resources collaborating - across admin, organisational boundaries
• Single sign-on– I connect to one machine – some sort of “digital credential” is
passed on to any other resource I use, basis of:Authentication: How do I identify myself to a resource without username/password for each resource I use?Authorisation: what can I do? Determined by
• My membership of VO • VO negotiations with resource providers
• Grid middleware runs on each resource• User just perceives “shared resources” with no
concern for location or owning organisation
Concepts of Grid Computing, Towards e-Research, St Andrews 16
Enabling Grids for E-sciencE
INFSO-RI-508833
Contents
• “The Grid” vision• What is “a grid” ?• Drivers of grid computing
Concepts of Grid Computing, Towards e-Research, St Andrews 17
Enabling Grids for E-sciencE
INFSO-RI-508833
The first driver: e-Science
• What is e-Science? Collaborative science that is made possible by the sharing across the Internet of resources (data, instruments, computation, people’s expertise...)– Often very compute intensive– Often very data intensive (both creating new data and accessing
very large data collections) – data deluges from new technologies
– Crosses organisational boundaries
• Examples….
Concepts of Grid Computing, Towards e-Research, St Andrews 18
Enabling Grids for E-sciencE
INFSO-RI-508833
Astronomy
No. & sizes of data sets as of mid-2002,grouped by wavelength
• 12 waveband coverage of large areas of the sky
• Total about 200 TB data• Doubling every 12 months• Largest catalogues near 1B objects
Data and images courtesy Alex Szalay, John Hopkins University
Concepts of Grid Computing, Towards e-Research, St Andrews 19
Enabling Grids for E-sciencE
INFSO-RI-508833
Large Hadron Collider at CERN
• Data Challenge:– 10 Petabytes/year of data !!!– 20 million CDs each year!
• Simulation, reconstruction, analysis:– LHC data handling requires computing
power equivalent to ~100,000 of today's fastest PC processors!
• Operational challenges– Reliable and scalable through project
lifetime of decades
Mont Blanc(4810 m)
Downtown Geneva
Concepts of Grid Computing, Towards e-Research, St Andrews 20
Enabling Grids for E-sciencE
INFSO-RI-508833
BLAST gridification
DB
BLASTSeq1 > dcscdssdcsdcdscbscdsbcbjbfvbfvbvfbvbvbhvbhsvbhdvbhfdbvfd
Seq2 > bvdfvfdvhbdfvbbhvdsvbhvbhdvrefghefgdscgdfgcsdycgdkcsqkc
…
Seqn > bvdfvfdvhbdfvbbhvdsvbhvbhdvrefghefgdscgdfgcsdycgdkcsqkchdsqhfduhdhdhqedezhhezldhezhfehflezfzejfv
DB
BLAST
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
DB
BLAST
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
DBBLAST
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
RESULTdedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbfvbfvbvfbvbvbhvbhsvbhdvbhfdbvfdbvdfvfdvhbdfvbhdbhvdsvbhvbhdvrefghefgdscgdfgcsdycgdkcsqkcqhdsqhfduhdhdhqedezhdhezldhezhfehflezfzeflehfhezfhehfezhflezhflhfhfelhfehflzlhfzdjazslzdhfhfdfezhfehfizhflqfhduhsdslchlkchudcscscdscdscdscsddzdzeqvnvqvnq! Vqlvkndlkvnldwdfbwdfbdbdwdfbfbndblnblkdnblkdbdfbwfdbfn
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
UI
Computingelement
Computingelement
Inputfile
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
Seq1 >dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbdfndfjvbndfbnbnfbjnbjxbnxbjk:nxbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
Seq2 >dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbdfndfjvbndfbnbnfbjnbjxbnxbjk:nxbf
dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf
Seqn >dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbdfndfjvbndfbnbnfbjnbjxbnxbjk:nxbf
Concepts of Grid Computing, Towards e-Research, St Andrews 21
Enabling Grids for E-sciencE
INFSO-RI-508833
Engine flight data
Airline office
Maintenance Centre
European data center
London AirportNew York Airport
American data center
Grid
Diagnostics Centre
•“A Significant factor in the success of the Rolls-Royce campaign to power the Boeing 7E7 with the Trent 1000 was the emphasis on the new aftermarket support service for the engines provided via DS&S. Boeing personnel were shown DAME as an example of the new ways of gathering and processing the large amounts of data that could be retrieved from an advanced aircraft such as the 7E7, and they were very impressed”, DS&S 2004
Engine Model
Case Based Reasoning
DAME: Grid based tools and Infer-structure for Aero-Engine Diagnosis and Prognosis
XTO
Universities:York,Leeds,Sheffield, Oxford
Companies:Rolls-RoyceDS&S Cybula
Concepts of Grid Computing, Towards e-Research, St Andrews 22
Enabling Grids for E-sciencE
INFSO-RI-508833
Political drivers
• Entering the “knowledge society” from the “industrial society”– industrial society: also enabled by communications infrastructure
• Lisbon strategy: Research and Innovation will be the most important factors in determining Europe’s success through the next decades
• THE GOAL: “UNLEASH CREATIVITY”- by investment in– Human skills– Infrastructures
• Growth of e-infrastructure (= networks + grid + operations)– phase 1: mainly academia, some in industry: “an elite, privileged to do
this job”– phase 2: ordinary people doing distributed work; SMEs, adopt, adapt
and use– phase 3: the next generations
will transform e-infrastructure and its usesWe don’t know how others will use what we devise
Concepts of Grid Computing, Towards e-Research, St Andrews 23
Enabling Grids for E-sciencE
INFSO-RI-508833
Grids and e-Infrastructure
• “Campus grids”: internal to an institute / university:– “High throughput” – harvesting compute time– Not really ‘a grid’ unless crossing administrative domains– Can become a resource on a grid– Example: Condor– http://www.nesc.ac.uk/esi/events/556/
• Grids: cross administrative boundaries– National scale: in UK, NGS – International scale: in Europe, EGEE
Concepts of Grid Computing, Towards e-Research, St Andrews 24
Enabling Grids for E-sciencE
INFSO-RI-508833
EGEE – building e-infrastructure
EGEE is building a large-scale production grid service to:
• Underpin research, technology and public service
• Link with and build on national, regional and international initiatives
• Foster international cooperation both in the creation and the use of the e-infrastructure
Network infrastructure& Resource
centres
Ope
ratio
ns,
Supp
ort a
nd
trai
ning
Collaboration
Pan-European Grid
23rd November 2004 To change: View -> Header and Footer
25
Empowerment in the Information AgeDr Spyros Konidaris
European Commission – DG INFSO, 3rd EGEE Conference
Information Age
1781 2000
Empowerment
Industrial Age
(Networks+ Tools)
“GRIDS”
e-Infra
structure
20
Concepts of Grid Computing, Towards e-Research, St Andrews 26
Enabling Grids for E-sciencE
INFSO-RI-508833
Commercial drivers
•European organisations could save €4.5 Billion by adopting basic Grid technologies
•Industry Analysts Gartner and Giga both estimate that standardisation and consolidation can save between 8.5% and 20% of IT budgets
• www.oracle.com/technology/tech/grid
Concepts of Grid Computing, Towards e-Research, St Andrews 27
Enabling Grids for E-sciencE
INFSO-RI-508833
Contents
• “The Grid” vision• What is “a grid” ?• Drivers of grid computing• Some examples • Current status of grids
Concepts of Grid Computing, Towards e-Research, St Andrews 28
Enabling Grids for E-sciencE
INFSO-RI-508833
… then where are we now?
If “The Grid”vision leads us here…
Concepts of Grid Computing, Towards e-Research, St Andrews 29
Enabling Grids for E-sciencE
INFSO-RI-508833
Grid projects
Many Grid development efforts — all over the world
•NASA Information Power Grid•DOE Science Grid•NSF National Virtual Observatory•NSF GriPhyN•DOE Particle Physics Data Grid•NSF TeraGrid•DOE ASCI Grid•DOE Earth Systems Grid•DARPA CoABS Grid•NEESGrid•DOH BIRN•NSF iVDGL
•DataGrid (CERN, ...)•EuroGrid (Unicore)•DataTag (CERN,…)•Astrophysical Virtual Observatory•GRIP (Globus/Unicore)•GRIA (Industrial applications)•GridLab (Cactus Toolkit)•CrossGrid (Infrastructure Components)•EGSO (Solar Physics)
•UK – OGSA-DAI, RealityGrid, GeoDise, Comb-e-Chem, DiscoveryNet, DAME, AstroGrid, GridPP, MyGrid, GOLD, eDiamond, Integrative Biology, …•Netherlands — VLAM, PolderGrid•Germany — UNICORE, Grid proposal•France — Grid funding approved•Italy — INFN Grid•Eire — Grid proposals•Switzerland - Network/Grid proposal•Hungary — DemoGrid, Grid proposal•Norway, Sweden - NorduGrid
Concepts of Grid Computing, Towards e-Research, St Andrews 30
Enabling Grids for E-sciencE
INFSO-RI-508833
Grids: where are we now?
• Many key concepts identified and known• Many grid projects have tested, and benefit from, these• Major efforts now on establishing:
– Standards (a slow process) (e.g. Global Grid Forum, http://www.gridforum.org/ )
– Production Grids for multiple VO’s“Production” = Reliable, sustainable, with commitments to quality of service
• In Europe, EGEE• In UK, National Grid Service• In US, Teragrid
One stack of middleware that serves many research (and other!!!)communities Operational procedures and services (people!, policy,..)
– New user communities• … whilst research & development continues
Concepts of Grid Computing, Towards e-Research, St Andrews 31
Enabling Grids for E-sciencE
INFSO-RI-508833
The key for new VO’s
• The tools, services used by the VO’s applications
• Application development environment, portals, semantics
• Insulate applications from changing middleware
Basic Grid services:AA, job submission, info, …
Middleware: “collective services”
Applicationtoolkits, standards
Application
Concepts of Grid Computing, Towards e-Research, St Andrews 32
Enabling Grids for E-sciencE
INFSO-RI-508833
The vision of 2001: convergence of Web Services and Grids
OGSIGrid prototypes
web developments
Web services
“big Science” research
INTERNET
World-wide web
Massively parallel computing
High-end computing
High throughput-computing
Open Grid Services Architecture
Concepts of Grid Computing, Towards e-Research, St Andrews 33
Enabling Grids for E-sciencE
INFSO-RI-508833
Contents
• “The Grid” vision• What is “a grid” ?• Drivers of grid computing• Current status of grids• The basis: authentication, authorisation, security
Concepts of Grid Computing, Towards e-Research, St Andrews 34
Enabling Grids for E-sciencE
INFSO-RI-508833
Grid security and trust -1• Providers of resources (computers, databases,..) need risks to
be controlled: they are asked to trust users they do not know – They trust a VO– The VO trusts its users
• User’s need – single sign-on: to be able to logon to a machine that can pass the
user’s identity to other resources– To trust owners of the resources they are using
• Build middleware on layer providing:– Authentication: who wants to use/provide resource– Authorisation: what the user is allowed to do– Security: reduce vulnerability, e.g. from outside the firewall– Non-repudiation: knowing who did what
• Digital credentials and the “Grid Security Infrastructure”middleware are the basis of production grids
Concepts of Grid Computing, Towards e-Research, St Andrews 35
Enabling Grids for E-sciencE
INFSO-RI-508833
Grid security and trust -2
• Currently, achieved by Certification:– User’s identity has to be certified by one of the national
Certification Authorities (CAs)mutually recognized http://www.gridpma.org/, for EU go via here to http://marianne.in2p3.fr/datagrid/ca/ca-table-ca.html to find your CA
•E.g. In UK go to http://www.grid-support.ac.uk/ca/ralist.htm– Resources are also certified by CAs
• User– User joins a VO– Digital certificate is basis of AA– Identity passed to other resources you use, where it is
mapped to a local account – the mapping is maintained by the VO
• Common agreed policies establish rights for a Virtual Organization to use resources
Concepts of Grid Computing, Towards e-Research, St Andrews 36
Enabling Grids for E-sciencE
INFSO-RI-508833
Grid security and trust -3
• Certification and GSI provides– Authentication
Resource can trust userUser can trust the resource provider…. So long as certificates are protected – they are your grid identity
– A basis for Authorisationso a VO can manage access to resourcesResource providers trust the VOThe VO trusts the user
– Mechanism for checking message integrityMessages are passed between machinesPublic/private key pairs protect message integrity as well as authentication
•Not (usually) encrypted but message-integrity is checked
Concepts of Grid Computing, Towards e-Research, St Andrews 37
Enabling Grids for E-sciencE
INFSO-RI-508833
Summary of grid computing concepts
• Flexible collaboration across multiple administrative domains – sharing data, computers, instruments, application software,..
• Single sign-on to resources in multiple organisations– Authorisation, authentication
• Need for people-services as well as middleware services – credential authorities, VO managers, support
• Drives are towards– Production services (reliable, sustainable,… – against which
research projects can plan with confidence)In Europe, EGEEIn UK, National Grid Service
– Standards– Empowering new user communities
Top Related