Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft European and German Grid Computing...

Forschungszentrum Karlsruhein der Helmholtz-Gemeinschaft

European and German Grid Computing Projects

Marcel Kunze

Forschungszentrum Karlsruhe

[email protected]

DESY Seminar

December 2002

Marcel Kunze, DESY Seminar

Grid Computing Potential

1980 2005


What is Grid Computing?

Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations


Grid Computing: Diverse Areas of Application

• Computational GridHigh Performance Computing (HPC)Perspective: Parallelization of programs, as fast as possible

• Data GridHigh Throughput Computing (HTC)Perspective : Parallelization of data, as much as possible

• Gaming GridCommunication between playersPerspective: LAN + WAN-Party, as entertaining as possible


Grid Computing: Commercial Aspects

• Huge potential of Grid Computing in the field of information technology:– Cost reduction through more efficient/diverse use of system resources

– „Resource on demand“Added flexibility, e.g. dynamically satisfy increased computing demand in bank accounting sector at the end of an accounting period

– „Business continuity“ Proliferation of mission-critical IT-Services to ease disaster recovery

– Possibility of system consolidation: Reduction of system complexity by standardization of services


Grid Computing: Scientific Aspects

• Enables formation of „critical mass“ of resources and expertise to allow for interdisciplinary projects in so-called “Virtual Organizations” (VO)– Tele-Collaboration

– Instrument Sharing

– Computing Resource Sharing

– Data Sharing

• Potential fields of application – Bio-Informatics and Medicine

– Astronomy

– Earth Sciences

– High Energy Physics (HEP)


Biomedical Applications

• Data mining on genomic databases (exponential growth)

• Indexing of medical databases (Tb/hospital/year)

• Collaborative framework for large scale experiments (e.g. epidemiological studies)

• Parallel processing for

–Databases analysis–Complex 3D modelling

Fabrizio

Gagliardi


Astronomy: Past and Future of the Universe

New phase of astronomy, storing, searching and analysing Petabytes of data:

• Virtual Observatories – GAVO, NVO, AVO, AstroGrid– Store all wavelengths, need

distributed joins– NVO 500 TB/yr from 2004

• Grid Computing might help to– Master the data streams– Federate databases with different

schema– Maintain meta data (information)

and provenance data (history)

Crab Nebula viewed

At four different

wavelengths: X-ray,

optical, infrared, radio.


Earth Observation

ESA missions:

• about 100 Gbytes of data per day (ERS 1/2)

• 500 Gbytes, for the next ENVISAT mission (launched March 1st)

EO requirements for the Grid:

• enhance the ability to access high level products

• allow reprocessing of large historical archives

• improve Earth science complex applications (data fusion, data mining, modelling …)

Fabrizio

Gagliardi


What is Grid Computing?

Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations


Elements of the Problem

• Resource sharing– Computers, storage, sensors, networks, …

– Heterogeneity of device, mechanism, policy

– Sharing conditional: negotiation, payment, …

• Coordinated problem solving– Integration of distributed resources

– Compound quality of service requirements

• Dynamic, multi-institutional virtual orgs– Dynamic overlays on classic org structures

– Map to underlying control mechanisms


The Grid World: Current Status

• Dozens of major Grid projects in scientific & technical computing/research & education– Deployment, application, technology

• Considerable consensus on key concepts and technologies– Open source Globus Toolkit™ a de facto standard for major

protocols & services

– Far from complete or perfect, but out there, evolving rapidly, and large tool/user base

• Global Grid Forum a significant force• Industrial interest emerging rapidly


“Web Services”

• Increasingly popular standards-based framework for accessing network applications– W3C standardization; Microsoft, IBM, Sun, others

• WSDL: Web Services Description Language– Interface Definition Language for Web services

• SOAP: Simple Object Access Protocol– XML-based RPC protocol; common WSDL target

• WS-Inspection– Conventions for locating service descriptions

• UDDI: Universal Description, Discovery & Integration – Directory for Web services


Open Grid Services Architecture(OGSA)

• Platform independent industrial and scientific standard to construct Grid computing components

• Service orientation to virtualize resources• From Web services:

– Standard interface definition mechanisms: multiple protocol bindings, multiple implementations, local/remote transparency

• Building on Globus Toolkit:– Grid service: semantics for service interactions

– Management of transient instances (& state)

– Factory, Registry, Discovery, other services

– Reliable and secure transport

• Multiple hosting targets: J2EE, .NET, “C”, …


The Grid Service

• A (potentially transient) Web service with specified interfaces & behaviors, including– Creation (Factory)

– Global naming (GSH) & references (GSR)

– Lifetime management

– Registration & Discovery

– Authorization

– Notification

– Concurrency

– Manageability


Grid Projects


GRID Projects World Wide

• EU– EDG (EU-IST) – R&D EU GRID project [ www.edg.org ]– CrossGRID QoS – Real Time apps. [ www.crossgrid.org ]– DataTAG GLUE (EU-USA) [ www.datatag.org ]– LCG The LHC Computing GRID – Deployment [ cern.ch/lcg ]– The new 16,2 B Euro EU VI Framework Prog. GEANT based GRID projects

• USA– GriPhyN iVDGL-VDTv1 PPDG ( NSF, DoE )[ www.griphyn.org ] [ www.idvgl.org ] [ www.ppdg.org ]

• Asia– ApGrid Pragma (USA-Asia)[ www.apgrid.org ]


GRIDLAB

GRIA

EGSO

DATATAG

CROSSGRID

DATAGRID

Applications

GRIP EUROGRID

DAMIENMiddleware

& Tools

Underlying Infrastructures ScienceIndustry / business

- Links with European National efforts

- Links with US projects (GriPhyN, PPDG, iVDGL,…)

IST Grid Project SpaceA

. B

axevanidis

http://www.cultivate-europe.org/funding.html


The EU DataGrid Project

• 21 Partners• 9.8 M Euros EU funding over 3 years• 90% for middleware and applications

(Particle Physics, Earth Obs. and Biomedical)• Three year phased developments & demos

(2001-2003)• Spin-off:

– DataTAG (2002-2003)

– CrossGrid (2002-2004)

– GridStart (2002-2004)

Fabrizio

Gagliardi


Research and Academic Institutes•CESNET (Czech Republic)•Commissariat à l'énergie atomique (CEA) – France•Computer and Automation Research Institute, Hungarian Academy of Sciences (MTA SZTAKI)•Consiglio Nazionale delle Ricerche (Italy)•Helsinki Institute of Physics – Finland•Institut de Fisica d'Altes Energies (IFAE) - Spain•Istituto Trentino di Cultura (IRST) – Italy•Konrad-Zuse-Zentrum für Informationstechnik Berlin - Germany•Royal Netherlands Meteorological Institute (KNMI)•Ruprecht-Karls-Universität Heidelberg - Germany•Stichting Academisch Rekencentrum Amsterdam (SARA) – Netherlands•Swedish Research Council - Sweden

DataGrid Partners

Industrial Partners•Datamat (Italy)•IBM-UK (UK)•CS-SI (France)

Main Partners•CERN – International (Switzerland/France)•CNRS - France•ESA/ESRIN – International (Italy)•INFN – Italy•NIKHEF – The Netherlands•PPARC - UK

Fabrizio

Gagliardi


EU DataGrid Project Objectives

• Use Grid technology to develop a sustainable computing model for effective share of computing resources and data for large scientific communities

• Specific project objectives:– Middleware for fabric & Grid management

– Large scale testbeds

– Production quality demonstrations

• Key products: – Resource broker (Compute elements, storage elements)

– Replica manager

– Virtual Organization manager


EDG: Interaction with Sister Projects

• CrossGrid– Using the same security certs.– Testbed sites install EDG software

• Extending it for needs of intensive interactive applications

– Participating in the EDG testing activities

– Representatives in each projects architecture & management groups

• DataTAG (EDT)– EDT is deploying EDG sw to

investigate inter-operability with US projects (iVDGL, GriPhyN, PPDG)

– Results feedback into EDG software releases (e.g. GLUE compatible information providers/consumers)

• NorduGrid– Using the same security certs.– Involved in EDG architecture work

• Good ideas for gatekeeper and MDS configuration

• Helped develop GDMP and GSI extensions for Replica Catalog

• Involved in GLUE schema work• Security policy

– Middleware testing– Working in WP8 (HEP applications)

• iVDGL/GriPhyN/PPDG– US members in EDG architecture

group– Looking for common packaging and

toolkit usage solutions

Fabrizio

Gagliardi


CrossGrid: An EDG Partner

• New category of Grid enabled applications– Computing and data intensive

– Distributed

– Interactive, near real time response (a person in a loop)

– Layered

• New programming tools• Grid more user friendly, secure and efficient• Interoperability with other Grids• Implementation of standards


CrossGrid Layered Architecture

ApplicationsAnd Supporting

Tools

ApplicationsDevelopment

Support

Grid Common Services

Grid Visualisation

Kernel

Grid Visualisation

Kernel

DataMining on

Grid

DataMining on

Grid

Interactive Distributed

Data Access

Interactive Distributed

Data Access

Globus Replica

Manager

Globus Replica

Manager

Roaming Access

Roaming Access

Grid Resource

Management

Grid Resource

Management

Grid Monitoring

Grid Monitoring

MPICH-GMPICH-G

Distributed Data

Collection

Distributed Data

Collection

User Interaction Service

User Interaction Service

DataGridReplica

Manager

DataGridReplica

Manager

Datagrid Job

Manager

Datagrid Job

Manager

GRAMGRAM GSIGSIReplica CatalogReplica Catalog GASSGASSMDSMDS GridFTPGridFTPGlobus-IOGlobus-IO

Resource Manager

Resource Manager

CPUCPU

ResourceManagerResourceManager

Resource Manager

Resource Manager

Secondary Storage

Secondary Storage


Scientific Instruments

(Medical Scaners, Satelites, Radars)

Scientific Instruments

(Medical Scaners, Satelites, Radars)


DetectorLocal High

Level Trigger

DetectorLocal High

Level Trigger


VR systems(Caves,immerse desks)

VR systems(Caves,immerse desks)


Visualization tools

Visualization tools

Optimization of Data Access

Optimization of Data Access

Tertiary StorageTertiary Storage

Local Resources

BiomedicalApplicationBiomedicalApplication

PortalPortalPerformance

AnalysisPerformance

AnalysisMPI

VerificationMPI

VerificationMetrics and BenchmarksMetrics and Benchmarks

HEP High LevelTriggerHEP High

LevelTriggerFlood

ApplicationFlood

ApplicationHEP Interactive Distributed Data

Access Application

HEP Interactive Distributed Data

Access Application

HEP DataMining on Grid

Application

HEP DataMining on Grid

Application

WeatherForecast

application

WeatherForecast

application


Interactive Treatment Planning

The vascular geometry can be modified using a library of models

Draw interactively Computational geometry


Biomedical Application Sample pulse flow simulation


Through links with sister projects, there is thepotential for a uniform global scientific applications grid

Benefit of StandardizationF

abrizio G

agliardi


Common WorldGrid Demo at IST2002

• WorldGrid Demo together with US and EU partners at IST2002• Seamless interoperation of EDG with US Middleware (via GLUE)

VO centric Ganglia monitor

http://people.cs.uchicago.edu/~cldumitr/Ganglia/overview_simple.pl?ent=WorldGrid/DataTAG%20Grid&ip=oke.uchicago.edu&port=8653&refresh=60&graph=load

http://people.cs.uchicago.edu/~cldumitr/Ganglia/overview_simple.pl?ent=WorldGrid/iVDGL%20Grid&ip=oke.uchicago.edu&port=8653&refresh=30&graph=load


WorldGrid Lessons (+)

• Advantages of the Grid:– Possibility to execute tasks and move files over a

distributed computing infrastructure by using one single personal certificate (no need to memorize dozens of passwords)

– Possibility do distribute the workload adequately and automatically, without logging in explicitly to each remote system

– Possibility to do worldwide production in a perfectly coordinated way, using identical software (RPMs), scripts and databases

• GLUE interfacing works to make the middleware worlds talk to each other. Better solution: Agree on open standards environment like OGSA !


WorldGrid Lessons (-)

• EDG stability very much dependent on the quality of the basic building blocks (Globus, Condor, etc)

• Globus support needs to be strengthened (formal contract, European support center, internal Globus support team)

• Understand impact of OGSA and industrial involvement– EDG Toolkit might need to be refactored


EDG Future Plans

• Concentrate on production quality and real applications

• Educate new users and disseminate results

• Complete the program of work till end of 2003

• Port EDG to other platforms than Linux (Solaris)

• Port EDG to Globus ToolKit version 3 (OGSA)

• Make plans to conserve momentum and assets in the EU FP6


A Proposal to the EU

• EDG has already demonstrated the viability of Grid technology

• EU had a fast start in supporting 18 Grid projects in the last two years!

• RN Geant offers an excellent basis for a large European Grid infrastructure

• The EU FP6 program should encourage and support the deployment and production quality operation of a large international Grid infrastructure open to research and industry in Europe

Fabrizio

Gagliardi


Enabling Grids and e-Science in Europe

• EGEE: Integrated Infrastructure Initiative to support European Research Area

• Vision: to create and deploy Grid technologies to enable the widespread uptake of e-Science applications throughout the European Research Area

• Four key objectives:– integrating Grid technological developments from across Europe;– establishing a Europe-wide Grid infrastructure for science and industry

with a focus on heterogeneity and interoperability;– enabling the creation of e-Science applications from across the scientific

and industrial spectrum; – ensuring the timely delivery of the project’s programme of work, guided

by the needs of academic and industrial partners.

Start by integration of the national Grid initiatives Fabrizio

Gagliardi


GRID Computing Center:

Infrastructure and Services

GRID Computing Center GridKa

Competence Center:

Applications and e-Science


Partners in the HGF24.000 employees, 2.5 Billion Euro yearly budget

Grid Computing perfectly well matches the HGF vision of “Konzertierte Forschung”

• ALFRED-WEGENER-INSTITUT FÜR POLAR- UND MEERESFORSCHUNG AWI• DEUTSCHES ELEKTRONEN-SYNCHROTRON DESY• DEUTSCHES KREBSFORSCHUNGSZENTRUM DKFZ• DEUTSCHES ZENTRUM FÜR LUFT- UND RAUMFAHRT DLR• FORSCHUNGSZENTRUM JÜLICH FZJ• FORSCHUNGSZENTRUM KARLSRUHE FZK• GESELLSCHAFT FÜR BIOTECHNOLOGISCHE FORSCHUNG GBF• GEOFORSCHUNGSZENTRUM POTSDAM GFZ• GKSS-FORSCHUNGSZENTRUM GEESTHACHT GKSS• GSF-FORSCHUNGSZENTRUM FÜR UMWELT UND GESUNDHEIT GSF• GESELLSCHAFT FÜR SCHWERIONENFORSCHUNG GSI• HAHN-MEITNER-INSTITUT BERLIN HMI• MAX-PLANCK-INSTITUT FÜR PLASMAPHYSIK IPP• MAX-DELBRÜCK-CENTRUM FÜR MOLEKULARE MEDIZIN MDC• UFZ-UMWELTFORSCHUNGSZENTRUM LEIPZIG-HALLE UFZ

http://www.awi-bremerhaven.de/

http://www.desy.de/

http://www.dkfz-heidelberg.de/

http://www.dlr.de/


A German Grid InitiativeD-GRID

• Initially driven by the HGF centers and the DFN

• Open to accept further partners in academia and industry

• Aim at a coordination of Grid activities

• Deployment of a new generation networking infrastructure (towards “TeraGrids”)

• Promotion of open standards for interfaces and protocols (GGF)

Stuttgart

Leipzig

Berlin

Frankfurt

Karlsruhe

Garching

Kiel

Braunschweig

Dresden

Aachen

RegensburgKaiserslautern

Augsburg

Bielefeld

Hannover

Erlangen

Heidelberg

Ilmenau

Würzburg

Magdeburg

Marburg

Göttingen

Oldenburg

Essen

St. Augustin

Rostock

Global UpstreamHamburg

10 Gbit/s2,4 Gbit/s2,4 Gbit/s622 Mbit/s


Scientific Program

• Installation of a coordination office for Grid Computing• Foster the enhanced application of IT in the field of

scientific and engineering disciplines• Management of the relationship between the German

Grid competence centers and representation of their common interests in the Global Grid Forum

• Training and education in the field of Grid Computing and e-Science

• Deployment of national Grid resources and development of a program for academic and industrial projects


Technical Program

• Goal: Integration of all kind of resourcese.g. sensors, computing infrastructure, people,…

• Deployment of a state-of-the-art production backbone (2003: 10 GBPS, 2006: 100 GBPS)

• Discussion of safety and trust relationship• Management of German Grid certificates• Selection of middleware components and installation procedures

(Globus, UNICORE)• Infrastructure: Find qualified manpower to offer and support the

additional services• Definition of cost model (Grid-Accounting, credit points)• Collaboration with the German HPC centers

=> D-GRID Kickoff Workshop in January 2003 (Wissenschaftszentrum Bonn)


Where do we go from here??


e-Science (enhanced Science)Konzertierte Forschung

• e-Science is about more than networks, GRIDs, High Performance Computing, cluster computing ...

• e-Science: „e-Science is about global collaboration in key areas of science, and the next generation of infrastructure that will enable it.“ (Dr. John Taylor, director of UK research council)

• Foster the transition of data Grid to semantic Grid– Federation of resources

– Federation of distinguished data sources

– Knowledge retrieval and collaboratories


Knowledge Grid

Semantic Grid: Three Layer Grid Abstraction

Information Grid

Computation/Data GridDat

a to

Kno

wle

dge

Con

trol

Automation

e-Science


Grid-enabled Applications

Prototype Grid Infrastructures

Gèant: World Class Networking

• Where do we need to get to ?– Applications to support an “e-society”

(“Cyber-Infrastructure”)

– An international Grid infrastructure which hides the complexities from the users (“Invisible Computing”)

– A powerful and flexible network infrastructure

• Where do we need to invest ?– Applications targeted at realistic problems

in “e-science”

– Prototypes of Grid infrastructures

– Maintain and improve the GEANT network

The Future

Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft European and German Grid Computing...

Documents

Transcript of Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft European and German Grid Computing...