Guy Warner [email protected] - University of St...

INFSO-RI-508833

Enabling Grids for E-sciencE

www.eu-egee.org

Concepts of grid computing

Guy [email protected]

Concepts of Grid Computing, Towards e-Research, St Andrews 2


INFSO-RI-508833

Acknowledgements

• This talk was prepared by Mike Mineter of NeSC and includes slides from previous tutorials and talks delivered by:– Dave Berry, Richard Hopkins, Guy Warner (National e-Science Centre)– the EDG training team– Ian Foster, Argonne National Laboratories– Jeffrey Grethe, SDSC– EGEE colleagues – Mark Baker, The Distributed Systems Group, University of Portsmouth,

http://dsg.port.ac.uk/mab

• Talks at 3rd EGEE conference by– Kyriakos Baxevanidis,Deputy Head,Unit of Research

Infrastructures,European Commission, DG INFSO– Dr Spyros Konidaris, European Commission – DG INFSO

http://dsg.port.ac.uk/mab



INFSO-RI-508833

Goals of this module

• To introduce the concepts of Grid computing assuming no previous knowledge



INFSO-RI-508833

Contents

• “The Grid” vision• What is “a grid” ?• Drivers of grid computing• Current status of grids• The basis: authentication, authorisation, security



INFSO-RI-508833

The Grid Metaphor

GRID

MIDDLEWARE

Mobile Access

Supercomputer, PC-Cluster

Data-storage, Sensors, Experiments

Workstation

Visualising

Internet, networks



INFSO-RI-508833

The grid vision• The grid vision is of “Virtual

computing” (+ information services to locate computation, storage resources)

– Compare: The web: “virtual documents” (+ search engine to locate them)

• MOTIVATION: collaboration through sharing resources (and expertise) to expand horizons of – Research – Commerce – engineering, …

“the knowledge economy”– Public service – health,

environment,…



INFSO-RI-508833

Contents

• “The Grid” vision• What is “a grid” ?



INFSO-RI-508833

“A grid”• The initial vision: “The Grid”• The present reality: Many

“grids”• Each grid is an infrastructure

enabling one or more “virtual organisations” to share computing resources

• What’s a VO?– People in different

organisations seeking to cooperate and share resources across their organisational boundaries

• Why establish a Grid?– Share data– Pool computers– Collaborate

Institute A

VO

Institute CInstitute B

Institute D



INFSO-RI-508833

The Single Computer

• The Operating System enables easy use of – Input devices– Processor– Disks– Display– Any other attached devices

Disks, Processor, Memory, …

Application Software

Operating

System



INFSO-RI-508833

Resources on a Local Area Network

User just perceives “shared resources”, with no regard to location in the organisation:

- Authenticated by username / password

- Authorised to use own files,…

Resources connected by a LAN


Operating System on each computer

Middleware for sharingcomputers, servers, printers, …



INFSO-RI-508833

Resources on a grid

Resources connected by internet


Operating System on each resource

Grid Middleware on each resource

Grid Middleware: “collective services”

Interface between app. and grid



INFSO-RI-508833

A grid• Grid middleware

runs on each shared resource– Data storage– (Usually) batch

jobs on pools of processors

• Users join VO’s• Virtual organisation

negotiates with sites to agree access to resources

• Distributed services (both people and middleware) enable the grid

INTERNET



INFSO-RI-508833

What characterises a grid?

• Co-ordinated resource sharing – No centralised point of control – Different administrative domains.

• Standard, open, general-purpose protocols and interfaces– NOT specific to an application– EGEE, NGS support multiple VO’s

• Delivering non-trivial qualities of service – Co-ordinated to deliver combined services,

greater than sum of the individual components

• http://www.gridtoday.com/02/0722/100136.html

http://www.gridtoday.com/02/0722/100136.html



INFSO-RI-508833

The components of a Grid

• Resources– networking, computers, storage, data, instruments, …

• Grid Middleware– the “operating system of the grid”

• Operations infrastructure– Run enabling services (people + software)

• Virtual Organization management– Procedures for gaining access to resources



INFSO-RI-508833

Key concepts

• Virtual organisation: people and resources collaborating - across admin, organisational boundaries

• Single sign-on– I connect to one machine – some sort of “digital credential” is

passed on to any other resource I use, basis of:Authentication: How do I identify myself to a resource without username/password for each resource I use?Authorisation: what can I do? Determined by

• My membership of VO • VO negotiations with resource providers

• Grid middleware runs on each resource• User just perceives “shared resources” with no

concern for location or owning organisation



INFSO-RI-508833

Contents

• “The Grid” vision• What is “a grid” ?• Drivers of grid computing



INFSO-RI-508833

The first driver: e-Science

• What is e-Science? Collaborative science that is made possible by the sharing across the Internet of resources (data, instruments, computation, people’s expertise...)– Often very compute intensive– Often very data intensive (both creating new data and accessing

very large data collections) – data deluges from new technologies

– Crosses organisational boundaries

• Examples….



INFSO-RI-508833

Astronomy

No. & sizes of data sets as of mid-2002,grouped by wavelength

• 12 waveband coverage of large areas of the sky

• Total about 200 TB data• Doubling every 12 months• Largest catalogues near 1B objects

Data and images courtesy Alex Szalay, John Hopkins University



INFSO-RI-508833

Large Hadron Collider at CERN

• Data Challenge:– 10 Petabytes/year of data !!!– 20 million CDs each year!

• Simulation, reconstruction, analysis:– LHC data handling requires computing

power equivalent to ~100,000 of today's fastest PC processors!

• Operational challenges– Reliable and scalable through project

lifetime of decades

Mont Blanc(4810 m)

Downtown Geneva



INFSO-RI-508833

BLAST gridification

DB

BLASTSeq1 > dcscdssdcsdcdscbscdsbcbjbfvbfvbvfbvbvbhvbhsvbhdvbhfdbvfd

Seq2 > bvdfvfdvhbdfvbbhvdsvbhvbhdvrefghefgdscgdfgcsdycgdkcsqkc

…

Seqn > bvdfvfdvhbdfvbbhvdsvbhvbhdvrefghefgdscgdfgcsdycgdkcsqkchdsqhfduhdhdhqedezhhezldhezhfehflezfzejfv

DB

BLAST

dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbf




DB

BLAST



DBBLAST













RESULTdedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbfvbfvbvfbvbvbhvbhsvbhdvbhfdbvfdbvdfvfdvhbdfvbhdbhvdsvbhvbhdvrefghefgdscgdfgcsdycgdkcsqkcqhdsqhfduhdhdhqedezhdhezldhezhfehflezfzeflehfhezfhehfezhflezhflhfhfelhfehflzlhfzdjazslzdhfhfdfezhfehfizhflqfhduhsdslchlkchudcscscdscdscdscsddzdzeqvnvqvnq! Vqlvkndlkvnldwdfbwdfbdbdwdfbfbndblnblkdnblkdbdfbwfdbfn



















UI

Computingelement

Computingelement

Inputfile












Seq1 >dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbdfndfjvbndfbnbnfbjnbjxbnxbjk:nxbf




Seq2 >dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbdfndfjvbndfbnbnfbjnbjxbnxbjk:nxbf


Seqn >dedzedzdzedezdzecdscsdcscdssdcsdcdscbscdsbcbjbdfndfjvbndfbnbnfbjnbjxbnxbjk:nxbf



INFSO-RI-508833

Engine flight data

Airline office

Maintenance Centre

European data center

London AirportNew York Airport

American data center

Grid

Diagnostics Centre

•“A Significant factor in the success of the Rolls-Royce campaign to power the Boeing 7E7 with the Trent 1000 was the emphasis on the new aftermarket support service for the engines provided via DS&S. Boeing personnel were shown DAME as an example of the new ways of gathering and processing the large amounts of data that could be retrieved from an advanced aircraft such as the 7E7, and they were very impressed”, DS&S 2004

Engine Model

Case Based Reasoning

DAME: Grid based tools and Infer-structure for Aero-Engine Diagnosis and Prognosis

XTO

Universities:York,Leeds,Sheffield, Oxford

Companies:Rolls-RoyceDS&S Cybula



INFSO-RI-508833

Political drivers

• Entering the “knowledge society” from the “industrial society”– industrial society: also enabled by communications infrastructure

• Lisbon strategy: Research and Innovation will be the most important factors in determining Europe’s success through the next decades

• THE GOAL: “UNLEASH CREATIVITY”- by investment in– Human skills– Infrastructures

• Growth of e-infrastructure (= networks + grid + operations)– phase 1: mainly academia, some in industry: “an elite, privileged to do

this job”– phase 2: ordinary people doing distributed work; SMEs, adopt, adapt

and use– phase 3: the next generations

will transform e-infrastructure and its usesWe don’t know how others will use what we devise



INFSO-RI-508833

Grids and e-Infrastructure

• “Campus grids”: internal to an institute / university:– “High throughput” – harvesting compute time– Not really ‘a grid’ unless crossing administrative domains– Can become a resource on a grid– Example: Condor– http://www.nesc.ac.uk/esi/events/556/

• Grids: cross administrative boundaries– National scale: in UK, NGS – International scale: in Europe, EGEE



INFSO-RI-508833

EGEE – building e-infrastructure

EGEE is building a large-scale production grid service to:

• Underpin research, technology and public service

• Link with and build on national, regional and international initiatives

• Foster international cooperation both in the creation and the use of the e-infrastructure

Network infrastructure& Resource

centres

Ope

ratio

ns,

Supp

ort a

nd

trai

ning

Collaboration

Pan-European Grid

23rd November 2004 To change: View -> Header and Footer

25

Empowerment in the Information AgeDr Spyros Konidaris

European Commission – DG INFSO, 3rd EGEE Conference

Information Age

1781 2000

Empowerment

Industrial Age

(Networks+ Tools)

“GRIDS”

e-Infra

structure

20



INFSO-RI-508833

Commercial drivers

•European organisations could save €4.5 Billion by adopting basic Grid technologies

•Industry Analysts Gartner and Giga both estimate that standardisation and consolidation can save between 8.5% and 20% of IT budgets

• www.oracle.com/technology/tech/grid

http://www.oracle.com/technology/tech/grid



INFSO-RI-508833

Contents

• “The Grid” vision• What is “a grid” ?• Drivers of grid computing• Some examples • Current status of grids



INFSO-RI-508833

… then where are we now?

If “The Grid”vision leads us here…



INFSO-RI-508833

Grid projects

Many Grid development efforts — all over the world

•NASA Information Power Grid•DOE Science Grid•NSF National Virtual Observatory•NSF GriPhyN•DOE Particle Physics Data Grid•NSF TeraGrid•DOE ASCI Grid•DOE Earth Systems Grid•DARPA CoABS Grid•NEESGrid•DOH BIRN•NSF iVDGL

•DataGrid (CERN, ...)•EuroGrid (Unicore)•DataTag (CERN,…)•Astrophysical Virtual Observatory•GRIP (Globus/Unicore)•GRIA (Industrial applications)•GridLab (Cactus Toolkit)•CrossGrid (Infrastructure Components)•EGSO (Solar Physics)

•UK – OGSA-DAI, RealityGrid, GeoDise, Comb-e-Chem, DiscoveryNet, DAME, AstroGrid, GridPP, MyGrid, GOLD, eDiamond, Integrative Biology, …•Netherlands — VLAM, PolderGrid•Germany — UNICORE, Grid proposal•France — Grid funding approved•Italy — INFN Grid•Eire — Grid proposals•Switzerland - Network/Grid proposal•Hungary — DemoGrid, Grid proposal•Norway, Sweden - NorduGrid



INFSO-RI-508833

Grids: where are we now?

• Many key concepts identified and known• Many grid projects have tested, and benefit from, these• Major efforts now on establishing:

– Standards (a slow process) (e.g. Global Grid Forum, http://www.gridforum.org/ )

– Production Grids for multiple VO’s“Production” = Reliable, sustainable, with commitments to quality of service

• In Europe, EGEE• In UK, National Grid Service• In US, Teragrid

One stack of middleware that serves many research (and other!!!)communities Operational procedures and services (people!, policy,..)

– New user communities• … whilst research & development continues

http://www.gridforum.org/



INFSO-RI-508833

The key for new VO’s

• The tools, services used by the VO’s applications

• Application development environment, portals, semantics

• Insulate applications from changing middleware

Basic Grid services:AA, job submission, info, …

Middleware: “collective services”

Applicationtoolkits, standards

Application



INFSO-RI-508833

The vision of 2001: convergence of Web Services and Grids

OGSIGrid prototypes

web developments

Web services

“big Science” research

INTERNET

World-wide web

Massively parallel computing

High-end computing

High throughput-computing

Open Grid Services Architecture



INFSO-RI-508833

Contents

• “The Grid” vision• What is “a grid” ?• Drivers of grid computing• Current status of grids• The basis: authentication, authorisation, security



INFSO-RI-508833

Grid security and trust -1• Providers of resources (computers, databases,..) need risks to

be controlled: they are asked to trust users they do not know – They trust a VO– The VO trusts its users

• User’s need – single sign-on: to be able to logon to a machine that can pass the

user’s identity to other resources– To trust owners of the resources they are using

• Build middleware on layer providing:– Authentication: who wants to use/provide resource– Authorisation: what the user is allowed to do– Security: reduce vulnerability, e.g. from outside the firewall– Non-repudiation: knowing who did what

• Digital credentials and the “Grid Security Infrastructure”middleware are the basis of production grids



INFSO-RI-508833

Grid security and trust -2

• Currently, achieved by Certification:– User’s identity has to be certified by one of the national

Certification Authorities (CAs)mutually recognized http://www.gridpma.org/, for EU go via here to http://marianne.in2p3.fr/datagrid/ca/ca-table-ca.html to find your CA

•E.g. In UK go to http://www.grid-support.ac.uk/ca/ralist.htm– Resources are also certified by CAs

• User– User joins a VO– Digital certificate is basis of AA– Identity passed to other resources you use, where it is

mapped to a local account – the mapping is maintained by the VO

• Common agreed policies establish rights for a Virtual Organization to use resources

http://www.gridpma.org/

http://marianne.in2p3.fr/datagrid/ca/ca-table-ca.html





INFSO-RI-508833

Grid security and trust -3

• Certification and GSI provides– Authentication

Resource can trust userUser can trust the resource provider…. So long as certificates are protected – they are your grid identity

– A basis for Authorisationso a VO can manage access to resourcesResource providers trust the VOThe VO trusts the user

– Mechanism for checking message integrityMessages are passed between machinesPublic/private key pairs protect message integrity as well as authentication

•Not (usually) encrypted but message-integrity is checked



INFSO-RI-508833

Summary of grid computing concepts

• Flexible collaboration across multiple administrative domains – sharing data, computers, instruments, application software,..

• Single sign-on to resources in multiple organisations– Authorisation, authentication

• Need for people-services as well as middleware services – credential authorities, VO managers, support

• Drives are towards– Production services (reliable, sustainable,… – against which

research projects can plan with confidence)In Europe, EGEEIn UK, National Grid Service

– Standards– Empowering new user communities

Guy Warner [email protected] - University of St...

Documents

Transcript of Guy Warner [email protected] - University of St...