The Knowledge-based Workflow System for Grid Applications Ladislav Hluchý, Viet Tran, Ondrej Habala...

The Knowledge-based The Knowledge-based Workflow System for Grid Workflow System for Grid

ApplicationsApplications

Ladislav HluchLadislav Hluchýý, , Viet Tran, Ondrej HabalaViet Tran, Ondrej Habala

II SAS, SlovakiaII SAS, Slovakia

[email protected]@savba.sk

http://www.ui.sav.sk/http://www.ui.sav.sk/

May 9-11, 2007, ManchesterMay 9-11, 2007, ManchesterEGEE User ForumEGEE User Forum 2

KK-Wf Grid: -Wf Grid: PartnersPartners

Fraunhofer FIRST (Berlin, Germany) Fraunhofer FIRST (Berlin, Germany)

UIBK (Innsbruck, Austria) UIBK (Innsbruck, Austria)

IISAS (Bratislava, Slovak Republic) IISAS (Bratislava, Slovak Republic)

CYFRONET (Cracow, Poland) CYFRONET (Cracow, Poland) LogicDIS S.A. (Athens, Greece)LogicDIS S.A. (Athens, Greece)

Softeco Sismat SpA (Genoa, Italy)Softeco Sismat SpA (Genoa, Italy)

http://www.kwfgrid.euhttp://www.kwfgrid.eu


K-WfK-Wf GridGrid: Consortium: Consortium

www.kwfgrid.netwww.kwfgrid.net

Bratislava

Athens

Genova

Kraków

Innsbruck

Berlin


K-Wf Grid: ObjectivesK-Wf Grid: Objectives

Objectives of K-Wf gridObjectives of K-Wf grid To enable users to To enable users to create complex workflowscreate complex workflows and and

use grid resources without detailed knowledge of griduse grid resources without detailed knowledge of grid To construct workflows optimized for underlying To construct workflows optimized for underlying

infrastructure, infrastructure, using its advantages and avoiding using its advantages and avoiding its bottlenecksits bottlenecks

To (semi-)automatically To (semi-)automatically construct workflows based construct workflows based on user’s requirementson user’s requirements, using semantic annotation , using semantic annotation of services, data, applications and resourcesof services, data, applications and resources

To constantly renew information about the grid by To constantly renew information about the grid by using complex monitoring network – using complex monitoring network – to learn from to learn from experienceexperience

To provide To provide simple, easy-to-use interfacesimple, easy-to-use interface to K-Wf to K-Wf Grid servicesGrid services


Knowledge-based Workflow System for Knowledge-based Workflow System for Grid Applications – K-Wf GridGrid Applications – K-Wf Grid

Execute workflowExecute workflow

Capture knowledgeCapture knowledgeReuse knowledgeReuse knowledge

Monitor environmentMonitor environment

Analyze informationAnalyze information

Construct workflowConstruct workflow


K-Wf Grid: AdvantagesK-Wf Grid: Advantages

Composition of workflow from a set of servicesComposition of workflow from a set of services System composes the workflow for you – just tell him System composes the workflow for you – just tell him

what you want to get at the endwhat you want to get at the end System uses services which are System uses services which are available at the available at the

timetime and which are expected (based on past and which are expected (based on past experience) to provide good results (good = what you experience) to provide good results (good = what you want)want)

UsabilityUsability Less grid language, more application domain Less grid language, more application domain

languagelanguage Integrated collaboration interfacesIntegrated collaboration interfaces

Reuse of componentsReuse of components K-Wf Grid is based on respected standardsK-Wf Grid is based on respected standards


K-Wf Grid: Workflow RefinementK-Wf Grid: Workflow Refinement

Web Service ClassesWeb Service Classes

Web Service OperationsWeb Service Operations

Abstract OperationAbstract Operation

Grid ResourcesGrid Resources

User RequestUser Request““I want this data””

““There is my data!””


K-Wf Grid: Pilot applicationsK-Wf Grid: Pilot applications

PAs as domain-related instances of a typical K-Wf PAs as domain-related instances of a typical K-Wf Grid-based application systemGrid-based application system Common horizontal features / Specific vertical Common horizontal features / Specific vertical

customizationcustomization Flood Forecasting Simulation Cascade (FFSC, PA1)Flood Forecasting Simulation Cascade (FFSC, PA1)

Chain of simulations, targeting hydraulic simulation of Chain of simulations, targeting hydraulic simulation of a flooded areaa flooded area

Stateful WSRF services implementation in Java, Stateful WSRF services implementation in Java, using Globus Toolkit 4using Globus Toolkit 4

Enterprise Resource Planning (ERP, PA2)Enterprise Resource Planning (ERP, PA2) Stateless web services in Java (Tomcat/Axis)Stateless web services in Java (Tomcat/Axis)

Coordinated Traffic Management (CTM, PA3)Coordinated Traffic Management (CTM, PA3) Traffic simulation in Genoa, ItalyTraffic simulation in Genoa, Italy Developed as stateless web services in PerlDeveloped as stateless web services in Perl


K-Wf Grid: PA1 – FFSCK-Wf Grid: PA1 – FFSC

Flood Forecasting Simulation CascadeFlood Forecasting Simulation Cascade a cascade of simulation models for weather a cascade of simulation models for weather

forecasting, hydrological and hydraulic simulations, forecasting, hydrological and hydraulic simulations, able to produce predictions of possible floods in able to produce predictions of possible floods in target areastarget areas

InputInput– map of area, precipitation, river water level etc.map of area, precipitation, river water level etc.

OutputOutput– water flow in target areawater flow in target area

Suggestions / user supportSuggestions / user support– applicability of different models to specific scenarios (QoS)applicability of different models to specific scenarios (QoS)– comparison between stored predictions and real datacomparison between stored predictions and real data– future dispersion forecastingfuture dispersion forecasting


Flood Forecasting ApplicationFlood Forecasting Application(PA1-FFSC)(PA1-FFSC)

Flood forecasting based on a Flood forecasting based on a series of simulationsseries of simulations

Several simulation models forSeveral simulation models for MeteorologyMeteorology HydrologyHydrology HydraulicsHydraulics

2D/3D visualization2D/3D visualization Implementation using GT4 Implementation using GT4

WSRFWSRF Being extendedBeing extended

More modelsMore models More services – towards risk More services – towards risk

managementmanagement


Replica location

Metadata catalog

Ce

ntr

al s

erv

ice

s

Informationindex


Replica location

Metadata catalog

Ce

ntr

al s

erv

ice

s

Informationindex

Monitoringservice


Replica location

Metadata catalog

Ce

ntr

al s

erv

ice

s

Informationindex

Monitoringservice

Knowledgestore

Informationprocessing

Workflowmanagement


Replica location

Metadata catalog

Ce

ntr

al s

erv

ice

s

Informationindex

Monitoringservice

Knowledgestore


Workflowmanagement

Portal


Replica location

Metadata catalog

Ce

ntr

al s

erv

ice

s

Informationindex

Monitoringservice

Knowledgestore


Workflowmanagement

Portal

Execute workflow

Monitor environment

Analyze information

Capture knowledge

Reuse knowledge

Construct workflow


Porting to gLitePorting to gLite

In K-Wf grid, applications are implemented as WSRF In K-Wf grid, applications are implemented as WSRF servicesservices

In gLite, the services are replaced by a set of worker In gLite, the services are replaced by a set of worker jobs that are submited by workflow manager via jobs that are submited by workflow manager via gLite WMSgLite WMS When started, the worker jobs will contact to When started, the worker jobs will contact to

workflow manager for executing tasksworkflow manager for executing tasks


Replica location

Metadata catalog

Ce

ntr

al s

erv

ice

s

Informationindex

Monitoringservice

Knowledgestore


Workflowmanagement

Portal

Execute workflow

Monitor environment

Capture knowledge

Reuse knowledge

Construct workflow

gLite WMS


Int.eu.gridInt.eu.grid

GoalsGoals Deploy and operate an interoperable production-level e-Infrastructure for Deploy and operate an interoperable production-level e-Infrastructure for

demanding interactive applicationsdemanding interactive applications Distributed (MPI-) parallel interactive computing and storage at Tera levelDistributed (MPI-) parallel interactive computing and storage at Tera level User friendly access through a interactive Grid desktopUser friendly access through a interactive Grid desktop Support for virtual organizations at all levels:Support for virtual organizations at all levels:

Setup, Collaborative environment, Grid enhancement of applications, Execution and Setup, Collaborative environment, Grid enhancement of applications, Execution and monitoring toolsmonitoring tools

Project detailsProject details Project start: May 1, 2006Project start: May 1, 2006 Project duration: 24 monthsProject duration: 24 months

Contribution of UI SAVContribution of UI SAV Operation of computer cluster with 32 CPUs and 32GB RAM as part of the project Operation of computer cluster with 32 CPUs and 32GB RAM as part of the project

production testbedproduction testbed User support, organization of tutorials and users’ trainingsUser support, organization of tutorials and users’ trainings Support for environmental applications and their usersSupport for environmental applications and their users Implementation of interactivity to flood simulation application Implementation of interactivity to flood simulation application Support for gridification and interactivity of radiocative pollution simulation aplication Support for gridification and interactivity of radiocative pollution simulation aplication

(owned by MicroStep-MIS)(owned by MicroStep-MIS)

http://www.interactive-grid.eu/


Int.eu.grid - PartnersInt.eu.grid - Partners

BIFI, Zaragoza/SpainBIFI, Zaragoza/Spain CESGA, Santiago de Compostela/SpainCESGA, Santiago de Compostela/Spain CSIC-IFCA, Santander/SpainCSIC-IFCA, Santander/Spain CYFRONET, Cracow/PolandCYFRONET, Cracow/Poland FZK, Karlsruhe/GermanyFZK, Karlsruhe/Germany GUP, Linz/AustriaGUP, Linz/Austria HLRS, Stuttgart/GermanyHLRS, Stuttgart/Germany ICM, Warsaw/PolandICM, Warsaw/Poland LIP, Lisbon/PortugalLIP, Lisbon/Portugal PSNC, Poznan/PolandPSNC, Poznan/Poland TCD, Dublin/IrelandTCD, Dublin/Ireland UAB, Barcelona/SpainUAB, Barcelona/Spain UI SAV, Bratislava/SlovakiaUI SAV, Bratislava/Slovakia


Int.eu.grid - InteractivityInt.eu.grid - Interactivity

Migrating Desktop

Remote Access Server

Job SubmissionServices

gLogin

CrossBroker

Computing Element

Gatekeeper

LRMS

Worker node 0

Worker node 1

Worker node N

…

JDL

Job control

Interactive channels

Interactive input

Interactive output


Control Output/visualizations/state


Control Output/visualizations


Flood Interactivity in Int.eu.gridFlood Interactivity in Int.eu.grid

Job is internally controlled by a workflow managerJob is internally controlled by a workflow manager Job steps can be added/removed during runtimeJob steps can be added/removed during runtime Can be paused/restarted/modifiedCan be paused/restarted/modified

Job outputs/visualized data are available through Job outputs/visualized data are available through the MD interfacethe MD interface

Job may be clonedJob may be cloned State = workflow state + intermediate data, may be State = workflow state + intermediate data, may be

easily transferredeasily transferred Good for parameter studiesGood for parameter studies


ConclusionConclusion

Current research focuses on SOKU (Service Current research focuses on SOKU (Service Oriented Knowledge Utilities)Oriented Knowledge Utilities) K-Wf Grid is one of the early implementations of K-Wf Grid is one of the early implementations of

SOKU conceptsSOKU concepts How to adapt infrastructure research to this shift in How to adapt infrastructure research to this shift in

paradigm?paradigm? Application developers & end users need easy Application developers & end users need easy

access to grid infrastructureaccess to grid infrastructure SOKU is the way to achieve thisSOKU is the way to achieve this How to extend gLite towards SOKU?How to extend gLite towards SOKU?

The Knowledge-based Workflow System for Grid Applications Ladislav Hluchý, Viet Tran, Ondrej Habala...

Documents

Transcript of The Knowledge-based Workflow System for Grid Applications Ladislav Hluchý, Viet Tran, Ondrej Habala...