Cracow Grid Workshop 2003

Institute of Computer

Science AGH

A Concept of a Monitoring Infrastructure for Workflow-Based Grid Applications

Bartosz Baliś, Marian Bubak, Włodzimierz Funika, Tomasz

Szepieniec, Marcin Radecki, Roland Wismüller, Tomasz Arodź, Marcin

Kurdziel

Cracow Grid Workshop 2003

Institute of Computer

Science AGH

Plan

Workflow-based applications Application Monitoring Architecture of the monitoring

infrastructure Performance analysis Monitoring of multilingual

applications Summary

Cracow Grid Workshop 2003

Institute of Computer

Science AGH

Workflow-based Grid Applications

Applications composed of a workflow of components

Components are independent Grid services

Cracow Grid Workshop 2003

Institute of Computer

Science AGH

Application Monitoring

Goal: provide monitoring services to • Obtain debug / performance-related

information• Possibly enable manipulations on the

target application• Detect events and program actions to be

executed when they occur

Consumers• Tools for performance analysis / debugging• Other tools and systems, e.g., for fault-

tolerance, load balancing, etc.

Cracow Grid Workshop 2003

Institute of Computer

Science AGH

Monitoring infrastructure for Wf-based apps

Additional monitoring interface for each grid service (component) to be monitored• Obtain monitoring info for that component

Global monitoring service• Separate grid service• One per application• Obtain collective monitoring info

Cracow Grid Workshop 2003

Institute of Computer

Science AGH

Monitoring Wf-based Applications

Client

Monitoring service

Additional monitoring interface (monitoring info related to the component)

Global monitoring (grid) service. Collective

monitoring information.

Cracow Grid Workshop 2003

Institute of Computer

Science AGH

Startup

User submits a workflow application and requests monitoring, possibly subsequently

Monitoring service is created• It must discover the workflow components• Workflow subsystem must provide a

mechanism for this• Workflow registry?

Cracow Grid Workshop 2003

Institute of Computer

Science AGH

Component monitoring interface – how?

Monitoring functionality inherent part of each component

How to provide the additional monitoring interface?• Take monitoring interface into account at

the design stage• Component developers involved?

This approach (hopefully) enables instrumentation of grid services

Cracow Grid Workshop 2003

Institute of Computer

Science AGH

Performance analysis service

Combined with the monitoring service

Supports performance analysis of•Grid infrastructure•Application

•Intra-component•Inter-component

A B C D E F G HSEL ECT ED

O N-LINE

Infrastructuremonitoring

Applicationmonitoring

Inter-component

Inter-

component

Inter-

component

Cracow Grid Workshop 2003

Institute of Computer

Science AGH

Infrastructure and intra-component analysis

Grid infrastructure • Uses existing tools to monitor the status of

grid environment of the components

Intra-component operations• Measure some quantities describing the

status and performance of a service used in the application workflow

• Depend on the monitoring interface defined by the component developers

• Reuse existing approaches to monitoring and analysis

Cracow Grid Workshop 2003

Institute of Computer

Science AGH

Inter-component performance monitoring

Adds a new level to the performance analysis of grid applications

Accounts for the concept of application as a workflow of services

Monitors the state and performance of the cooperation of the services to capture the state and performance of the whole application

Two modes:• Structural, semantic monitoring mode • Activity monitoring mode

Cracow Grid Workshop 2003

Institute of Computer

Science AGH

Structural, semantic monitoring Captures information on workflow components:

• Component status• Component usage• Custom component properties

Uses the intra-component monitoring data on:• Functions• Code regions• Synchronisation objects

Combines the information on various components to describe the status of the whole workflow based-application

Proposed techniques:• Source/byte code instrumentation• Dynamic instrumentation

Cracow Grid Workshop 2003

Institute of Computer

Science AGH

Activity monitoring

Captures information on the behaviour of the workflow

Analysed activity properties:• Caller/callee relationships• End-to-end response time for activity invocation• Success rate of invocation• Synchronisation time in activity• Volume of data exchanged in activity

Proposed techniques:• Distributed instrumentation of workflow components• Insertion of probes

Cracow Grid Workshop 2003

Institute of Computer

Science AGH

Event Service

Automatically sends notifications to other grid services when some performance conditions are met.

Allows for a reaction on undesired changes of grid environment status as well as in the activity of the application.

Will be integrated with performance data provider in a uniform framework.

This integration will enable the notified service to inspect a wide range of performance information before deciding on the actions that should be taken.

Cracow Grid Workshop 2003

Institute of Computer

Science AGH

Two levels of performance information A proven concept form the G-PM tool. Used in both application performance data

provider and event service. Lower level performance information:

• a consistent, well-defined set of metrics,• used, by system administrators, application

developers, users and other parties. Higher level performance information:

• more abstract, numerical, quantitative metrics,• defined on the basis of the lower-level metrics

with a special definition language,• allowing for creation of various metrics suitable for

a particular purpose.

Cracow Grid Workshop 2003

Institute of Computer

Science AGH

Monitoring of multi-lingual applications Services can be implemented using multi-lingual

approach e.g. as a computational kernels written in Fortran and wrapped by Java code.

A higher level abstraction of metrics must be worked out which could handle different programming approaches and languages.

The low-level monitoring system must be designed in a way that enables a simultaneous monitoring of Java, C, C++ and other language-based service parts using an uniform approach:• the specification of monitoring services must be the same

for all languages,• the implementations of the monitoring services may vary.

Cracow Grid Workshop 2003

Institute of Computer

Science AGH

Summary

Workflow-based approach to applications still immature

Architecture for a monitoring infrastructure proposed• Integration of monitoring services into

components• Global grid monitoring service

Performance analysis• Resuse of existing approaches for

infrastructure and intra-component analysis

• New level of inter-component analysis