slides (PPT)

The CrossGrid Project

Marcel Kunze, FZK

representing the X#-Collaboration

ACAT 2002 Moscow Marcel Kunze - FZK

Main Objectives

New category of Grid enabled applications Computing and data intensive Distributed Interactive, near real time response (a person in a loop) Layered

New programming tools

Grid more user friendly, secure and efficient

Interoperability with other Grids

Implementation of standards


Poland:Cyfronet & INP CracowPSNC PoznanICM & IPJ Warsaw

Portugal:LIP Lisbon

Spain:CSIC SantanderValencia & RedIrisUAB BarcelonaUSC Santiago & CESGA

Ireland:TCD Dublin

Italy:DATAMAT

Netherlands:UvA Amsterdam

Germany:FZK KarlsruheTUM MunichUSTU Stuttgart

Slovakia:II SAS Bratislava

Greece:AlgosystemsDemo AthensAuTh Thessaloniki

Cyprus:UCY Nikosia

Austria:U.Linz

21 21 institutinstitutees s

11 11 countriescountries

CrossGrid Collaboration


GRIDLAB

GRIA

EGSO

DATATAG

CROSSGRID

DATAGRID

Applications

GRIP EUROGRID

DAMIENMiddleware

& Tools

Underlying Infrastructures ScienceIndustry / business

- Links with European National efforts

- Links with US projects (GriPhyN, PPDG, iVDGL,…)

IST Grid Project Space


Collaboration with other # Projects

Objective – Exchange of Information Software components

Partners DATAGRID DATATAG GRIDLAB EUROGRID and GRIP

GRIDSTART Participation in GGF


Project Phases

M 1 - 3: requirements definition and merging

M 4 - 12: first development phase: design, 1st prototypes, refinement of requirements

M 13 -24: second development phase: integration of components, 2nd prototypes

M 25 -32: third development phase: complete integration, final code versions

M 33 -36: final phase: demonstration and documentation


Network infrastructure, archivers, HPC/HPV systems,

Labour instruments, etc. Local domain services

Protocols, Authentication, Authorization, Access policy, Resource management, etc.

Remote Data Access

Optimization

Monitoring

Schedulers

Roaming Access

Portals

Interactive simulation and visualisation of a biomedical systemFlooding crisis team support

Distributed Data Analysis in High Energy PhysicsWeather forecast and air pollution modeling

FABRIC FABRIC INFRASTRUCTUREINFRASTRUCTURE

GridGridSERVICESSERVICES

TOOLSTOOLS

APPLICATIONSAPPLICATIONS

GLOBUS TOOLKIT , Condor-G, ...

DATAGRID SET OF TOOLS

Grid Visualization

KernelBenchmarks

Structure Overview


CrossGrid Architecture

ApplicationsAnd Supporting

Tools

ApplicationsDevelopment

Support

Grid Common Services

Grid Visualisation

Kernel

Grid Visualisation

Kernel

DataMining on

Grid

DataMining on

Grid

Interactive Distributed

Data Access

Interactive Distributed

Data Access

Globus Replica

Manager

Globus Replica

Manager

Roaming Access

Roaming Access

Grid Resource

Management

Grid Resource

Management

Grid Monitoring

Grid Monitoring

MPICH-GMPICH-G

Distributed Data

Collection

Distributed Data

Collection

User Interaction Service

User Interaction Service

DataGridReplica

Manager

DataGridReplica

Manager

Datagrid Job

Manager

Datagrid Job

Manager

GRAMGRAM GSIGSIReplica CatalogReplica Catalog GASSGASSMDSMDS GridFTPGridFTPGlobus-IOGlobus-IO

Resource Manager

Resource Manager

CPUCPU

ResourceManagerResourceManager

Resource Manager

Resource Manager

Secondary Storage

Secondary Storage


Scientific Instruments

(Medical Scaners, Satelites, Radars)

Scientific Instruments

(Medical Scaners, Satelites, Radars)


DetectorLocal High

Level Trigger

DetectorLocal High

Level Trigger


VR systems(Caves,immerse desks)

VR systems(Caves,immerse desks)


Visualization tools

Visualization tools

Optimization of Data Access

Optimization of Data Access

Tertiary StorageTertiary Storage

Local Resources

BiomedicalApplicationBiomedicalApplication

PortalPortalPerformance

AnalysisPerformance

AnalysisMPI

VerificationMPI

VerificationMetrics and BenchmarksMetrics and Benchmarks

HEP High LevelTriggerHEP High

LevelTriggerFlood

ApplicationFlood

ApplicationHEP Interactive Distributed Data

Access Application

HEP Interactive Distributed Data

Access Application

HEP DataMining on Grid

Application

HEP DataMining on Grid

Application

WeatherForecast

application

WeatherForecast

application


Layered Structure

Interactive and Data Intensive Applications (WP1) Interactive simulation and visualization of a biomedical system Flooding crisis team support Distributed data analysis in HEP Weather forecast and air pollution modeling

Grid Application Programming Environment (WP2)

MPI code debugging and verification Metrics and benchmarks Interactive and semiautomatic performance evaluation tools

Grid Visualization Kernel Data Mining

New CrossGrid Services (WP3)

Globus Middleware

Fabric Infrastructure (Testbed WP4)

DataGridGriPhyN

...

Services

HLA

Portals and roaming access Grid resource management Grid monitoring Optimization of data access


Scope of Applications

Applications in health and environment Data federation, processing and interpretation in geographically

distributed locations Fast, interactive decision making

Interactive access to distributed Databases Super computers and High Performance Clusters Visualisation engines Medical scanners Environmental data input devices


Application Requirements

High quality presentation

High frame rate

Intuitive interaction

Real-time response

Interactive algorithms

High performance computing and networking

Distributed resources and data


Role of Network Latency

Communication delay and rendering delay are negligible


CrossGrid Application Development (WP1)

Interactive simulation and visualisation of a biomedical system Grid-based system for pre-treatment planning in vascular

interventional and surgical procedures through real-time interactive simulation of vascular structure and flow.

Flooding crisis team supportDistributed interactive data analysis in HEP Focus on LHC experiments (ALICE, ATLAS, CMS and LHCb)

Weather forecast and air pollution modelling Porting distributed/parallel codes on Grid Coupled Ocean/Atmosphere Mesoscale Prediction System STEM-II Air Pollution Code


Interactive Simulation and Visualisation

of a Biomedical SystemGrid-based prototype system for treatment planning in vascular interventional and surgical procedures through near real-time interactive simulation of vascular structure and flow.

The system will consist of a distributed near real-time simulation environment, in which a user interacts in Virtual Reality (VR) and other interactive display environments.

A 3D model of the arteries, derived using medical imaging techniques, will serve as input to a simulation environment for blood flow calculations.

The user will be allowed to change the structure of the arteries, thus mimicking an interventional or surgical procedure.


Current Situation

Observation

Diagnosis & Planning

Treatment


Experimental Setup


Angio w/ Fem-Fem &

Fem-Pop

AFB w/ E-S

Prox.Anast.

Angio w/Fem-Fem

AFB w/ E-E Prox.

Anast.

Preop

Alternatives

Simulation Based Planning and Treatment


VR-Interaction


Flood Crisis Prevention

Support system for establishment and operation of Virtual Organization for Flood Forecasting associating a set of individuals and institutions involved in flood prevention and protection.

The system will employ a Grid technology to seamlessly connect together the experts, data and computing resources needed for quick and correct flood management decisions.

The main component of the system will be a highly automated early warning system based on hydro-meteorological (snowmelt) rainfall-runoff simulations.

System will integrate the advanced communication techniques allowing the crisis management teams to consult the decisions with various experts. The experts will be able to run the simulations with changed parameters and analyze the impact.


Storage systems

databases

surface automatic meteorological and hydrological stations

systems for acquisition and processing of satellite information

meteorological radars

External sources of informationGlobal and regional centers GTSEUMETSAT and NOAAHydrological services of other countries

Data sources

meteorological models

hydrological models

hydraulic models

HPC, HTC

Grid infrastructure

Flood crisis teams

meteorologistshydrologistshydraulic engineers

Usersriver authoritiesenergyinsurance companiesnavigation

mediapublic

Virtual Organization for Flood Forecasting


Váh River Catchment Area: 19700km2, 1/3 of Slovakia

(Inflow point)

Nosice

Strečno

(Outflow point)

Pilot Site Catchment Area: 2500km2

(above Strečno: 5500km2)

Váh River Pilot Site

Flood Crisis Prevention

Water stages/discharges in the real time operating hydrological stations

Mapping of the flooded areas


Flow +

water depth

s

Flood SimulationsResults


Distributed Analysis in High Energy Physics

Challenging points Access to large distributed databases in the Grid Development of distributed data-mining techniques Definition of a layered application structure Integration of user-friendly interactive access

(based on PROOF)

Focus on LHC experiments (ALICE, ATLAS, CMS and LHCb)


PROOF

Local

Remote

Selection

Parameters

Procedure

Proc.C

Proc.C

Proc.C

Proc.C

Proc.C

PROOF

CPU

CPU

CPU

CPU

CPU

CPU

TagDB

RDB

DB1

DB4

DB5

DB6

DB3

DB2


Weather Forecast and Air Pollution Modeling

Integration of distributed databases into Grid

Migration of data mining algorithms to Grid

Porting distributed atmospheric & wave models to Grid

Porting parallel codes for air quality models to Grid

Integration, testing and demonstration of the application in the testbed environment


COAMPSCoupled Ocean/Atmosphere Mesoscale Prediction System:

Atmospheric Components

Complex Data Quality ControlAnalysis:

Multivariate Optimum Interpolation Analysis of Winds and Heights Univariate Analyses of Temperature and Moisture Optimum Interpolation Analysis of Sea Surface Temperature

Initialization: Variational Hydrostatic Constraint on Analysis Increments Digital Filter

Atmospheric Model: Numerics: Nonhydrostatic, Scheme C, Nested Grids, Sigma-z Physics: Convection, Explicit Moist Physics, Radiation, Surface Layer

Features: Globally Relocatable (5 Map Projections) User-Defined Grid Resolutions, Dimensions, and Number of Nested Grids 6 or 12 Hour Incremental Data Assimilation Cycle Can be Used for Idealized or Real-Time Applications Single Configuration Managed System for All Applications Operational at:

7 Areas, Twice Daily, using 81/27/9 km or 81/27 km grids Forecasts to 72 hours

Operational at all Navy Regional Centers (w/GUI Interface)


Status Quo … Quo Vadis ?

Current state (briefly) Simulation done on a single system or local clusters Visualisation on a single system, locally

What we are going to achieve HPC, HTC, HPV in geographically distributed environment Improved interaction with the end user Near real time simulations Different visualisation equipments (adaptive according to the end-user needs), like PDA Workstations VR studio (e.g. CAVE)


Grid Application Programming

Environment (WP2)

MPI code debugging and verification

Metrics and benchmarks

Interactive and semiautomatic performance evaluation tools

Specify, develop, integrate, test tools for HPC and HTC applications on the Grid


New Grid Services and Tools (WP3)

Portals and roaming accessGrid resource managementGrid monitoringOptimisation of data access

Objectives To develop interactive compute- and data-intensive applications To develop user-friendly Grid environments To offer easy access to the applications and Grid To have reasonable trade-off between resource usage efficiency

and application speedup To support management issues while accessing resources


Auth Thessaloniki

U v Amsterdam

FZK Karlsruhe

TCD Dublin

U A Barcelona

LIP Lisbon

CSIC Valencia CSIC Madrid

USC Santiago CSIC Santander

DEMO Athens UCY Nikosia

CYFRONET Cracow

II SAS Bratislava

PSNC Poznan

ICM & IPJ Warsaw

1515 sisitteess

International Testbed Organisation (WP4)

Testbed setup & incremental evolution Integration with DataGridInfrastructure supportVerification & quality control


Summary

Layered structure of all X# applicationsReuse of SW from DataGrid and other # projectsGlobus as the bottom layer of the middlewareHeterogeneous computer and storage systemsDistributed development and testing of SW 12 partners in applications 14 partners in middleware 15 partners in testbeds


1980s: Internet 1990s: Web 2000s: Grid

Where do we need to get to ?Applications to support an “e-society”

(“Cyber-Infrastructure”)An international Grid infrastructure which

hides the complexities from the users (“Invisible Computing”)

A powerful and flexible network infrastructure

Where do we need to invest ?Applications targeted at realistic problems in

“e-science”Prototypes of Grid infrastructuresMaintain and improve the GEANT network

Expression of Interest for EU FP6 program:“Enabling Grids and e-Science in Europe

(EGEE)”

Grid-enabled Applications

Prototype Grid Infrastructures

Gèant: World Class Networking


GRIDLAB

GRIA

EGSO

DATATAG

CROSSGRID

DATAGRID

Applications

GRIP EUROGRID

DAMIENMiddleware

& Tools

Underlying Infrastructures ScienceIndustry / business

EGEE Project Space

Enabling Grids and E-Science in Europe (EGEE)


First results of EGEE Brainstorming

ConsultingPrototypingDeployment

ConsultingTraining Courses

DisseminationForum

Tools and Service Development

Creation and support E- Science centres

Applications inOther Sciences

EI ROforum

S/W HardeningGLOBUS

EuroGrid, Gridlab etc.

Semantic GRI DDatabaseSecurity

European I nfrastructure

Integrated ProjectENABLI NG GRI DS ESCI ENCE EUROPE

EGEE

National eScienceCentres I ndustry

Applications

SMEs developingGrid-enabled Applications

I ndustry Outreach

ModulableTestbeds

R&D Agenda

Deployment withI T I ndustry

ScienceOutreach

C Jon

es,

D O

Will

iam

s, e

t al

slides (PPT)

Documents

Transcript of slides (PPT)