O G S A -D A I

OGSA-DAI: Patterns and Functions

Supercomputing ‘06AIST Stand

Malcolm Atkinson

Director e-Science Institute & e-Science Envoy

www.nesc.ac.uk14th November 2006

Outline of Talk Reminder about OGSA-DAI

Functionality, Extensibility & Patterns of Use Users OGSA-DAI Team

The new architecture Motivation Features Status

The Road ahead Engineering Challenges Research Challenges

OGSA-DAI OGSA-DAI provides:

WS framework for wrapping data resources Extensible:

Functionality can be added Client library Move computation to be close to the data

Have a workflow like language Encapsulate multiple interactions in one-interaction

Works with multiple Grid Fabrics OMII-UK container, GT4.*, Tomcat + Axis Soon (OMII-Europe project): Unicore, gLite 3.0

Simple DB operations SQL

Data Resource Accessor

Relational

XMLDBData

Resource Accessor

Data Service

Resource

Data Service

Resource

Data

Service

Data

Service

XPath or XQuery

5

O G S A -D A IExposing Databases

Data Resource Accessor

Relational

XMLDBData

Resource Accessor

Data Resource Accessor

Data Service

Resource

Files

Data Service

Resource

Data Service

Resource

SQLOne

XMLOne

FilesOne

Data

Service

Relational Multi-resourcesMultiple Data Resource

AccessorMulti

Resource

Data Service

Resource

MultiOne

Data

Service

Data

Service

Relational

SQL Query

Multi Results

SQL Query

Results x M

Data Service

Resource

Data

Service

Relational

SQL Query SQL QueryResults Results

SQL QuerySQL Query ResultsResults

SQL Query SQL QueryResults Results

Delegate DAI task completion

consumer

Data

Res

ourc

e

DR messages

Client OGSA-DAIRequest & Response D

ata

Res

ourc

e

DR messages

Data

del

iver

y

OGSA-DAI

Request & Response

Multi-ServiceIntegratione.g. DQP

Client

DR1

DR2

DR3

Data

Res

ourc

e

OGSA-DAI

Data

Res

ourc

eD

ata

Res

ourc

e

DR mes

sage

s

DR messages

DR messages

Data

Res

ourc

e

OGSA-DAI

Data

Res

ourc

eD

ata

Res

ourc

e

DR mes

sage

s

DR messages

DR messages

Data

Res

ourc

e

OGSA-DAI

Data

Res

ourc

eD

ata

Res

ourc

e

DR mes

sage

s

DR messages

DR messages

Request, R

esponse & D

ata Transport

Req

uest

, Res

pons

e & D

ata

Tra

nspo

rt

Request & Response

9

O G S A -D A IClients & Authorisation

Data Resource Accessor

Data

Service

Data Service

ResourceClient

Perform Document

SQLOne

Relational

Perform Document

SQL Query

ResultSet

SQL Query

ResultsResponse Document

Response Document

Authorization points

Used by a number of projects

OGSA-DAI

More info from: http://www.ogsadai.org.uk/about/projects.php

International Cooperation and Recognition

USA:o Globus Allianceo IBM Corporationo caBIGo BIRNo Indiana University o GridSphereo GEONo LEADo MCSo NCSAo Secure Data Grido UNC

Japan:o AISTo BioGrido NAREGI

Europe:o CERNo DataMiningGrido GridMinero GridSphereo inteligrido N2Grido OntoGrido Provenanceo SIMDAT

UK:o OMIIo NGSo NCeSSo NIeeSo AstroGrido BioSimGrido BRIDGESo CancerGrido ConvertGrido eDiaMonDo EDINAo First Group plco Fujitsu Labs Europeo GEDDMo GeneGrido Genomic Technology and Informaticso GOLDo Human Genetics Unito IBM UKo myGrido Oracle UK

China:o CASo ChinaGrido cnGrido INWA

Australia:o Curtin Business Schoolo INWA

TutorialsBoston CambridgeCERN ChicagoEdinburgh LondonSan Francisco SeattleSeoul SingaporeTokyo ISSGC 03 to 06

DIALOGUE workshopsColumbus, Edinburgh, Indiana, Vienna

Chicago, Manchester, San Diego

South Korea:o KISTI

China40%

United Kingdom15%

United States11%

Germany3%

Japan5%

Italy2%

France3%

Austria1%

Others20%

1485 registered users5250+ downloads

Outline of Talk Reminder about OGSA-DAI

Functionality, Extensibility & Patterns of Use Users OGSA-DAI Team

The new architecture Motivation Features Status

The Road ahead Engineering Challenges Research Challenges

Features of Future OGSA-DAI Comprehensive library of Integration Activities

Putting the I in OGSA-DAI You will still add your own

Perhaps less often With more examples to start from With a new OGSA-DAI Engine for support With new Activity base classes

Connected by “pipes” carrying Streams of values – scalability is a major motivator

Each with own threads Exploiting concurrent processing Possibly in different VMs – resilience a major motivator

Support for Iteration Expect Request documents

To contain more sophisticated DAGs Branching and merging activities Synchronous and dynamic data consumption

Auto-iteration over (Lists of) values Iteration activities

Manage the iteration of a controlled graph of activities

17

O G S A -D A IMulti-resource requests

Data Request

Execution Service

Data Request Execution Resource

Client

Data Resource Data

Data Resource Data

Data Resource Data

MyDRER

One

Two

Three

Perform Document

myQuery(One) myQuery(Two)

myIntegrateData

deliverGridFTP

Client-server interaction

Service-resource association

Resource-resource association

Resource-database association

18

O G S A -D A IAn expanded resource model

Data Request

Execution Service

Data Request Execution Resource

Client

Data Resource

Information Service

Data Resource

Session Management

Service

Session

Data

Data Resource Data

Data Resource Data

SessionSession

SessionSessionRequest

Request Management

Service

MySession123

MyDRER

One

Two

Three

MyRequest123456

19

O G S A -D A IPersistence

OGSA-DAI State

Data Request

Execution Service

Data Request Execution ResourceClient

Data Resource

Information Service

Data Resource

Session Management

Service

Session

Data

SessionSession

SessionSessionRequestRequest

Management Service

OGSA-DAI Server

Another OGSA-DAI Server

20

O G S A -D A IPersistence

Caching Reduce overheads of OGSA-DAI-database communications

Allows configuration and state to persist between container shutdowns and restarts

Sticky resources e.g. data sources and sinks In-memory – only available via server that created them

API File-based implementation for backwards compatibility Relational implementation for compatibility with common

OGSA-DAI databases Extensibility point

21

O G S A -D A IScalability and robustness

OGSA-DAI State

Client

OGSA-DAI cluster

Front-end Router

Monitor & Restart

Warning: not in next release – ODE contains the foundations

22

O G S A -D A IScalability and robustness

Release 3 will afford future provision of clustering and load balancing Clustered OGSA-DAI servers

Different JVM Different machine

Load balancer Simple e.g. round robin Complex e.g. decide on basis of current load of each server

Scalable Increases number of requests that OGSA-DAI can handle

concurrently Robust

Seamless operation in the face of errors e.g. server failures Sticky sessions and resources

Session and resource state may be non-serializable or be lightweight in-memory components

Need to dispatch resources to their “owner” JVM

23

O G S A -D A IOther planned features

Support for registries Guidelines on how to best use standard registries

(e.g. GRIMOIRES or GT4 ServiceGroups) and tools to help you do so within OGSA-DAI

Improved support for message-level and transport-level security including delegation

Concurrent or closely-following releases of A new implementation of DAIS

OGSA-DAIR and OGSA-DAIX A re-factored OGSA-DQP with a Java-compliant query

compiler and which runs under Linux, Solaris or Windows

Support for generic activities Powerful general purpose activities

Parameterised by Java methods Any data transformation or integration Task for application partitioned to writing simple methods Iteration and data management provided

ExamplesTupleGenericSort TupleGenericProductSelectProject

TupleGenericSelectProject TupleGenericUnion

TupleGenericMultiSelectProject TupleGenericIntersection

TupleGenericJoinSelectProject TupleGenericDifference

TupleGenericGroupBy

Outline of Talk Reminder about OGSA-DAI

Functionality, Extensibility & Patterns of Use Users OGSA-DAI Team

The new architecture Motivation Features Status

The Road ahead Engineering Challenges Research Challenges

Engineering Challenges Integrating / supporting all activities

Choices of internal data formats OMII-UK & OGSA-DAI code Contributors code

Robust & scalable ODE implementations Continuous operation

deployment, configuration, security & management Supporting but containing users’ code

Integration with other data services Replica & cashing services Grid file services Storage management services

Research Challenges High-level notations

Auto-Optimisation, Partitioning & Parallelisation Meta-data driven DAI

Automatic generation of DAI workflow detail Visual Comprehension

Visual forms of DAI workflows Visualisation of enactment progress Provision of control and diagnosis aids

Relationship with other workflows

28

O G S A -D A IFurther information The OGSA-DAI project site

http://www.ogsadai.org.uk

The DAIS-WG site http://forge.gridforum.org/projects/dais-wg

Recent talkshttp://www.ogsadai.org.uk/documentation/presentations/GW2006GGF18/

Design in Progress

http://www.ogsadai.org.uk/documentation/Design_documents/ OGSA-DAI users mailing list

users@ogsadai.org.uk General discussion on OGSA-DAI, data and the grid

Formal support for OGSA-DAI releases http://www.ogsadai.org.uk/support support@ogsadai.org.uk

OGSA-DAI training courses