OGCE Workflow and LEAD Overview

Suresh Marru, Marlon Pierce

September 2009

Suresh Marru

The LEAD Gateway

Suresh Marru

LEAD Pioneering Technology

Suresh Marru

StreamingObservations

Storms Forming

Forecast Model

Data Mining

Refine forecast grid

Instrument Steering

LEAD Dynamic Adaptive Infrastructure

Suresh Marru

LEAD Scientists and Educational Interactions

DevelopersResearcherss

End UsersLowering the barrier for using complex end-to-end weather

technologies

Democratize

Empower

Facilitate

Suresh Marru

Suresh Marru

Generalize, Harden, Build

Test

Open Grid Computing Environments

Suresh Marru

Middleware Services

Compute & Data Resources

User Interactions

Computational Cloud

GFac Services

Local Lab Resources

Other ClientsXBaya GUI

XBaya Core

Web Portal

Computational Grids

Event Bus

Workflow Engine (ODE) XRegistry

XMCCat Metadata Catalog

Flexible Layered Service Oriented Architecture

Suresh Marru

Application Wrapper Framework• Scientific Applications are wrapped

into web services by filling in a web-form.

• The Application Factory generates a web service for each application with I/O interfaces.

• Registers WSDL for the service with a registry

• Each service generates a stream of notifications that log the service actions back to the XMCCat Metadata Catalog, user monitoring, and provenance tracking tools

cc

Application Factory

Run programRun program& publish events& publish events

App Service

Suresh Marru

Application Wrapper Architecture

Suresh Marru

Service Monitoring via Events• The service output is a stream of

events• I am running your request• I have started to move your input files• I have all the files• I am running your application• The application is finished• I am moving the output to you file

space• I am done

• These are automatically generated by the service using a distributed event system(WS-Eventing / WS-Notification)• Topic based pub-sub system with

a well known “channel”

ApplicationServiceInstance

12

34

56

NotificationChannel

listener publisher

xx

Suresh Marru

Workflow Suite Architecture

Suresh Marru

Composition and

Monitoring Abstract DAG Model

BPEL 1.1

BPEL 2.0

SCUFL

Python

Dynamic Enactor/Interpreter

Jython Based Enactor

GPEL Engine

Apache ODE Engine

Taverna

Python Runtime

Message Bus

Interoperable Xbaya Architecture

Suresh Marru1414

OVP/RST/ MIG

Fault Tolerance/Recovery Service

Application Performance

Models

Resource Reliability

Models

NWS, MDS

BQP

Deadline &Success

Probability

PortalBPEL

Workflow Engine

Application Service

Notification Service

Simple Recovery Architecture

Suresh Marru

Workflow Composer

Suresh Marru

Example: Assimilation Workflow

Suresh Marru

• Domain webs of large genomes– Input list of amino acid sequences– Identify all known domains– Construct webs

Parallel processingCapability-type computing

Ensemble-type processing(minimal network reqs)

Capacity-type computing

Science Independence Example

Suresh Marru

Biology Gateway

Suresh Marru

Polar Grid Example

Suresh Marru

LEADLower entry barrier to using weather analysis toolsImprove detection, analysis & prediction of mesoscale weather

Motif-NetworkTransformation of sequenced genomes to “domain-space”

Cyber-Infrastructure EvaluationPerformance evaluation of future supercomputer architectures

ADAMAlgorithms for feature extraction, data normalization, classification

and normalization

GridChemMolecular Chemistry Grid helping researchers run chemistry

applications on Grid Environment

Example Gateways using OGCE Workflow Suite

Suresh Marru

Live Demo & Questions?