WS-VLAM workflow

UvA, Amsterdam June 2007 WS-VLAM Introduction presentation

WS-VLAM Introduction

Systems and Network Engineering group

Institute of informatics

University of Amsterdam


Introduction

• This presentation gives a quick introduction to the WS-VLAM workflow management system

• It helps the reader understand the technology push which motivated our design, describes the current state and present the ongoing activities and planned ones.

• NOTE: We are currently still working on the User’s and developer’s

Guide, they are not available at this time. The presentations do not replace the documentation it just give a quick introduction.

Do not hesitate to contact us for further information

Thanks for taking some of your time to read about the WS-VLAM work, any comments are more than welcome

WS-VLAM developers [email protected]


Needed background …

• To be able to follow the content of this presentation you will need to know about:

WSRF= Web Service Reference Framework

GT4= Globus Toolkit version 4

• More detailed description of the presented work can be found in the section of publication of the WS-VLAM home page


Objective of this presentation…

• Why you should look at this presentations You want to know how WS-VLAM works and

interacts with the Grid middleware.

• Why you don’t have to worry if you do not understand the content on this presentation: You are an end-user: it will be enough for you to

read the User’s guide or the WS-VLAM composer presentation

You are an application developer: it will be enough for you to read the Developer’s guide & the WS-VLAM developers presentation


Outline

• Introduction • Architecture of the WS-VLAM• Hierarchical workflow compositions &

execution • Interoperability with other WMS• Conclusions


GVLAM:

• Enable end-users (scientists) to define, execute, and monitor e-science applications. Location independent experimentation. Application Domain portal.

• Provide application developers: Abstract view of the Grid (shields the Grid

details). Easy way to port applications to the Grid.


The Virtual Laboratory for e-Science …

Dutc

h te

lesc

ienc

e

Data

inte

nsive

scie

nce

Med

ical

diag

nosis

Generic e-science framework layer

Application layer

Bio

info

rmat

ics

ASP

Bio

dive

rsity

Food

Info

rmat

ics

Grid layer


Process Management Stack

Before and after OGSA/WSRF

Network & Computing Resources

Grid Middleware: Process & resource management

Application Templates

User Applications

Workflow Management system

Web Service Interface

Data Management Stack

Network & storage Resources

Grid Middleware: Data management



Process Management StackData Management Stack

Workflow Management Systems

Network & Computing Resources



User Applications

Workflow execution

Workflow composition

Network & storage Resources



Web Service Interface Web Service Interface


Data Management Stack Process Management Stack

Grid Service & OGSA-DAI

Network & Computing ResourcesNetwork & storage Resources



Application Users

Workflow execution

GridServices

GridServices

GridServices

OGSA DAI


User Applications





Data Management Stack Process Management Stack

First Application developers …




Workflow execution

GridServicesApplication

web service

GridServices

GridServices


User Applications

Application TemplatesWeb service



OGSA DAI



Process Management Stack

Second the Workflow developers …

Data Management Stack




Workflow execution



GridServices

GridServices

GridServicesApplication Templates

Web service


OGSA DAI

WorkflowEngine

Web service

WorkflowEngine

Web service


Application web service

Distributed Workflow


WS-VLAM WMS…

• Goals Create a WMS which adopt a Service Oriented

Paradigm

• Approach Workflow Engine (RTSM)

o Implemented as a WSRF service using Globus 4 (GT4)

Workflow Composer (GUI)o Thin client

Repository for workflow componentso Implemented also as WSRF service.


Conclusions

• WSRF compliant middleware such as GT4 offers useful basic services to develop a truly grid enabled workflow engine:

Delegation service and ws-notification mechanisms proved to be very useful

• Separating the workflow composition from execution suits better the Grid environments and improve interoperability

• OGSA-WG is collecting cases and input to decide which services should be standardized


http://www.vl-e.nl/


New workflow Components (Under development)

• The workflow components developed the last year, can be still used and will be deployed on the shared workflow component repository on the rapid prototyping environment Ridge-O-Grammar modules from (SP 1.5) Kansk Toolbox from (SP 1.4) AID web services (SP 2.2) Virtual file system Web service (SP 2.4) Generic workflow components (SP 2.5)

o File Reader/Writer, o universal File Reader/Writer (SRB/GFTP/...)o Demos: MRIscanner, Histogram, Floatingball

• We will contact other SPs to integrate their components o Legacy code, or fully integrated code (java, C++, Python)


Current deployment & validation

• Current tests are performed on the rapid prototyping environment RTSM Service and Repository service are

deployed on ds2a.das2.nikhef.nl ws-VLAM Client (3 MB) can be used only on

ds2.das2.nikhef (because GT4 is only available on this node)

ws-VLAM client can also be used a host where Poc R1 is installed

• Tests will be performed on the validation environment for PoC R2


Planned tasks

• Access to the ws-vlam engine from Taverna This will be developed in the context of a use case together

with SP1.5

• Access to the ws-vlam engine from Kepler

• Parameter sharing among workflow components

• Job farming

• Support for more than one job submission mechanisms (GT4-GRAM, Direct, etc)

• Fault tolerance

• Control flow

• Data provenance

• Semantic Service discovery


References and download

• VLAM client can be downloaded from www.science.uva.nl/~gvlam/wsvlam/

• Documentation can be downloaded form (TO BE

DONE VERY SOON) www.science.uva.nl/~gvlam/wsvlam/doc/


Workflow interoperability: Execute a VLAM workflow from Kepler/Taverna

• A predefined Application workflow developed in VLAM can be executed as a single step in Kepler/Taverna (no need to recompose graphically the whole workflow).

• The predefined Application workflow will be executed on any remote computing resource where the VLAM-RTSM GT4 Web service is installed.

• Advantages: Compose workflow where sub-workflows (which require grid

resources) are executed on grid-enabled resources, while the rest of the workflow is either executed using other Kepler actors or taverna processors

It is also more efficient, since it avoid the overhead which will result by wrapping every workflow component as a separated web service or a separate remote grid-execution.


Workflow interoperability: Execute a VLAM workflow from Kepler/Taverna

GRAMservices

GT4 Java Container

RTSMFactory

Delegationservice

Worker nodes

pre-ws-GRAM

RTSM Instance

Workflowcomponents

Kepler/Taverna workbench

RTSM Client

WorkfowDescription

(XML)

Das2 or PoC facilities.

1. Kepler/Taverna users can have access to some of the parameters of the Application workflow to change the default values

2. Kepler/Taverna users have to specify the location of the input data file as URL and will get back a URL if the Sigwin generate data files

3. Graphical output of the Application workflow are handled automatically by the VLAM Taverna processor /Kepler actor.

RTSM-GT4 Web serviceVLAM Actor

or

Taverna processor

(1) Proxy Delegate

(2) Service

Invocation

WS-VLAM workflow

Technology

Transcript of WS-VLAM workflow