June 30 2006 Amsterdam

A Workflow Bus for e-Science Applications

Dr Zhiming ZhaoFaculty of Science, University of AmsterdamVL-e SP 2.5

Outline

Introduction A workflow bus and generic e-Science

framework Prototype and experiment results Discussion Conclusions Future work

Scientific workflow in e-Science

…

Grid infrastructure, E-Science and Scientific workflow

Step1: designing an experiment

Step

2: p

erformin

g th

e experim

ent

Step3: analyzing the experiment

results

Discovery Grid

Scientific Workflows in e-ScienceExperiment processes

Abstract workflows

Executable (concrete workflows)

wo

rkflow

s for ad

min

istration

, e.g.,

AA

A, an

d o

ther issu

es.

A SWMS is able to:

Automate experiment routines

Rapid prototype experimental computing systems

Hide integration details between resources

Manage experiment lifecycle

Insight a Scientific Workflow Management System

In our view, a SWMS at least implements:

A model for describing workflows;

An engine for executing/managing workflows;

Different levels of support for a user to compose, execute and control a workflow.

Workflow (based on certain model)

Engine

User su

pp

ort

resources

Composition

Engine level control

Resource level control

A SWMS

Diversity in SWMSTaverna:

-Web services based language: Scufl;

-FreeFluo: engine

-Graphical viz of workflow

Kepler:

-Actor,director

-MoML

-Execution models

Triana:

-Components

-Task graph

-Data/control flow

DAGMan:

-Computing tasks

-DAG

Pegasus:

-Based on DAGMan

-VDL

-DAG

…

Research context

Virtual LabGrid Layer

Application Layer

Different levels of abstraction

Workflow services: Short term Long term: a

generic and effective workflow management service

Mission

Effectively reuse existing workflow managements systems, and provide a generic e-Science framework for different application domains.

A generic framework can Improve the reuse of workflow components and the

workflows for different experiments Reduce the learning cost for different systems Allow application users to work on a consistent

environment when underlying infrastructure changed

Abstract approach

Extend approach

Aggregate approach

Possible options

SWMS1

SWMS2

SWMS3

SWMSG

SWMS1

SWMS2

SWMS3

SWMSG

SWMS1

SWMS2

SWMS3

SWMS1

SWMS2

SWMS3

SWMS G

Why we choose an aggregation approach? Abstract approach

Build a perfect system Difficult to find a set of systems cover all the required generic

functionality; it requires re-implementation of existing things Extend approach

Incrementally development The solution depends on a specific system

Aggregate approach Maximize the reuse of the existing workflow systems Has to handle interoperability issues; provide customized

interface existing workflow system

A workflow bus paradigm

Workflow bus

Taverna KeplerTriana

Sub workflow 1

Sub workflow 2

Sub workflow 3

Workflow

A workflow bus is a special workflow system for executing meta workflows, in which sub workflows will be executed by different engines.

Architecture

Terminology: The execution of a workflow is one study, and the execution of a

sub-workflow is called a sub-study, or a scenario Basic idea

Study manager schedules sub workflows Scenario managers interface third party workflow engines and

reacts to the Study manager A user interface for composition and execution control.

Network

Scenario MngerScenario MngerScenario Mnger Study Mnger

Taverna

Engine

Triana

Engine

User interface

Requirements

A distributed framework for study and scenario managers

Data input/output of a sub-workflow, description of the workflow can be described and recognized by study and scenario managers

Handle the user interactions which are needed in scenarios

The engine can be decoupled from a SWMS Be fault tolerant

Considerations

From integration point view: study and scenario managers can be coupled by: Web services Object oriented middleware (CORBA, HLA, etc.) Agent based middleware Or an existing workflow system (Kepler, Taverna,

Triana or others) The description of meta workflow The execution model of the meta workflow

A JADE/Ptolemy based prototype

Director

Actor

Actor

Actor

JADE agent framework

Scenario MngerScenario MngerScenario Mnger Study Mnger

Taverna

Engine

Triana

Engine

Ptolemy

User interface

How it works

In user front end: a user defines meta workflow, each actor represents a sub workflow

At runtime, each actor initiates a scenario agent, and passes the workflow description to the scenario manager

A scenario manager controls an engine and execute the sub-workflow

Prototype

Experiment results

Message delayMessage delay

0.001

0.01

0.1

1

10

100

0.001 0.01 0.1 1 10 100 1000 10000 100000

Size of message (KB, 1KB=1024 bytes)

Tim

e (s

eco

nd

s)

Scenario managers

SOAP

Cont.

Overhead

584.6

767.4

1636.1

1776.2

0

200

400

600

800

1000

1200

1400

1600

1800

time in ms

2048x1536 4096x3072Image

Triana workflow execution

Agent

Triana GUI

10~20% performance improvement.

10~20% performance improvement.

584.6

767.4

1636.11776.2

0

200

400

600

800

1000

1200

1400

1600

1800

Milliseconds

2048x1536 4096x3072Image

Triana workflow execution AgentTriana GUI

Discussion

Challenges in supporting scientific workflows Requirements on domain specific experiments Generic workflow support and domain specific

applications Existing workflow management systems are

diverse in functionality, design and user support Related work

Interoperability among workflow systems (sister Link project)

Resource level: e.g., Kepler invokes Taverna’s resources

Applications of workflow bus

Use case 1: A user has workflow in Taverna Some functionality is missing in Taverna but can be

provided by Triana He can develop the workflow in two systems, and run

it via the workflow bus

Use case 2: A user wants to execute a Taverna or Triana workflow

in multiple instances with different input data

Conclusions

A workflow bus is a feasible approach to realize generic e-Science framework

Multi agent technology provides a distributed environment for decomposing and encapsulating control intelligence

Ptolemy II provides different computing paradigms which give user freedom to execute workflows

Future work

Working on developing a scenario manager for Kepler engine.

Synchronized data flow is currently used; more computing modes will be evaluated.

Data provenance for workflow bus.

Referneces

Z. Zhao; A. Belloum; H. Yakali; P.M.A. Sloot and L.O. Hertzberger: Dynamic Workflow in a Grid Enabled Problem Solving Environment, in Proceedings of the 5th International Conference on Computer and Information Technology (CIT2005), pp. 339-345 . IEEE Computer Society Press, Shanghai, China, September 2005.

Z. Zhao; A. Belloum; A. Wibisono; F. Terpstra; P.T. de Boer; P.M.A. Sloot and L.O. Hertzberger: Scientific workflow management: between generality and applicability, in Proceedings of the International Workshop on Grid and Peer-to-Peer based Workflows in conjunction with the 5th International Conference on Quality Software, pp. 357-364. IEEE Computer Society Press, Melbourne, Australia , September 19th-21st 2005.

Z. Zhao; A. Belloum; P.M.A. Sloot and L.O. Hertzberger: Agent Technology and Generic Workflow Management in an e-Science Environment, in Hai Zhuge and G.C. Fox, editors, Grid and Cooperative Computing - GCC 2005: 4th International Conference, Beijing, China, in series Lecture Notes in Computer Science, vol. 3795, pp. 480-485. Springer, November 2005. ISBN 3-540-30510-6. (DOI: 10.1007/11590354_61)

Z. Zhao; A. Belloum; P.M.A. Sloot and L.O. Hertzberger: Agent technology and scientific workflow management in an e-Science environment, in Proceedings of the 17th IEEE International conference on Tools with Artificial Intelligence (ICTAI05), pp. 19-23. IEEE Computer Society Press, Hongkong, China, November 14th-16th 2005.

Acknowledgement Suresh Booms All the members in VL-e SP2.5