Architecture Grille Orientée Services ProActive Integration with HP Software in the AGOS project...

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

s

ProActive Integration with HP Software in the AGOS project

Towards the integration of business services within a grid environment supported by a virtualized infrastructure

Grids@Work 2008Sophia Antipolis – October 21st, 2008

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

s

2 April 18, 2023

Project contextPartner use casesDraft architectureProducts mappingHP integrationProActive features

Agenda

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

s

Project context

Putting together experts from grids, services, resources and data to define a simple, coherent and realistic solution

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

sProject context

Selected by the “Pôle de compétitivité Solutions Communicantes Sécurisées”

List of partners: HP, Oracle, Inria, Activeeon Defined scope: integration project

- Definition of an integration methodology- Definition of new software components- Integration of business and service dashboards- Integration of services with virtualized resources- Contribution to standards

Objective: “Integrate GRID technology into a business oriented SOA stack running over a virtualized infrastructure”

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

sProposed solution

A reference architecture putting together:- business intelligence- service oriented components- grid environment- virtualized systems

A library of process and service templates based on standards

A set of tools to easily build such applications A real-time monitoring of business processes,

services and resources A methodology to migrate to the architecture

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

s

Applied onto water operator use case Simulation application to better handle water distribution

network and provision adequate hydraulic meshing Focus is on software which allows the study, the planning

and the control of irrigation- hydraulic simulation software for meshed networks for

distribution of raw water or drinking water under pressure- full network is too large to be simulated at once; split into sub-

network parts that can be studied independently Main interest:

- Parallelization of the application (SPMD approach)- Overlap multiple runs (parameters sweeping) on subnetworks- Better integrate with other applications- Better use of resources to speed up computation

Measured in terms of speed increasing of digital processing provided by the grid architecture

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

sApplied onto global distribution system’s usage use case

Parsing and analysis of web session logs from target applications Focus is on providing massive data analysis:

- Customer session graph analysis or other data mining services- Know how groups of customers behave: the usage / navigation in websites

Main interest- Handle the size and the volume of data generated by core applications- Parsing of logs done massively in parallel- Results need to be stored in a distributed database- Better understand how the products are used- Help to improve the products continuously through the feedback loop

Expectations- Enable an advanced workflow beyond current standards to support processes

and sub processes and/or to compose hybrid processes with services.- Enable streaming results from any parallel activity on a grid to feed a

distributed database- Support operational aspects like monitoring, clustering, scalability, security,

load balancing and failover thus enabling high availability SLA compliance agreements to be respected.

- Enables one button push start, stop and resume on any running grid process

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

s

From building blocks to functional architecture proposal

Draft architecture

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

s

9 April 18, 2023

Base line…

Staged pyramid- Adaptive infrastructure of heterogeneous resources- Service delivery with quality added-value- Business intelligence based on standardized processes

Based on standards- As defined in the related standards documents- Existing implementations (proprietary / open-source)

With grid middleware- Grid environment: parallelism, distribution- Web interfaces: thin web clients access- Modeling facilities: composition, adaptive, configurable- Efficient execution: design & execution patterns

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

s

Grid

Grid Grid

Grid

BI

SA

AI

…of a Service Grid

BusinessIntelligence

ServiceArchitecture

AdaptiveInfrastructure

BPELETL

SCA / GCMSOA / WS

VM / XENP2P / GTK

Standards Functional

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

sMain building blocks

Systems Virtualization Grid Utility Interface Resource Manager Tasks & Services

Scheduler Enterprise Service Bus Distributed Shared

Memory Database Interface Application Server

Interface Web Service Interface

SOA Registry / Repository SOA Governance Component Architecture Load Balancer Fault Tolerance Reconfiguration Process Business Process Execution Real-time Monitoring Tools Service Level Management Business Intelligence Reporting Tools

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

sDefine the business processes

Processes defined in terms of business items Cope with business processes workflow Focus on business interactions between human and applications

- How are used the applications- How are they chained in any given scenario- What is expected as an output

Not a programmatic task flow- Do not cope with service implementation details- Far from resource allocation concerns

Handle specific concepts:- Long transactions (suspended time, recurring patterns…)- Requests approval (decision making, validation…)- Human interactions (subjective perception, biased usage…)- Business ecosystem (management, billing, visualization…)

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

s

Business Intelligence

Business Process

Service Level Management

ReportingBusiness

Intelligence

ServiceArchitecture


- Delivered results analysis

- Scenario validation

- Service & Service Level monitoring

- Reporting on key aspects by category

- Validation of performance gain, result precision, time to deliver…

- Comparison with reference benchmarks

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

sCalling value-added services

Services stored in a registry with governance applied Manipulating proxies to available web services Services as composite components With predefined characteristics and features

- Interface contract- Policy enforcement- Delivered quality of service

Mapped onto:- Existing web services (black-box)- Composite web services (functional composition)- Skeleton web services (parallel pattern)- Subordinate processes (time composition)- Simple sequential services

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

sImplemented as grid services

Components executed on a set of resources Skeletons providing parallel patterns for efficient

execution on the grid Transformed as jobs delegated to the grid

scheduler Optimized active objects exposed as web

services on the grid Jobs scheduled on the grid with direct access to

the grid scheduler Specialized grid schedulers dedicated for specific

usage

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

s

Service Architecture

Service CompositionService Component Architecture

Parallel ProgrammingDistributed shared memory

SOA EnvironmentRepository, Registry, Orchestration


ServiceArchitecture


- Distributed parallel services accessing data in distributed shared memory

- Functional composition of service components with dynamic reconfiguration

- Exposed as web services in registry with a policy enforcement contract

- Executed through proxies & orchestrated by a workflow engine

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

sExecuting on heterogeneous resources

Code execution on well-known resources or over a set of dynamically discovered resources- As tasks through a scheduler managing load-balancing, fault

tolerance, high availability…- As active objects with associated features such as migration, file

transfer…- As uncontrolled legacy application- As tasks delegated to another grid utility environment

Evolving set of resources (join / leave support) With specific or adaptive characteristics (virtualized or

clustered) On a set of various operating systems Protected or not, intrusive or not

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

s

Resource Virtualization

Grid Utility interface

Resource Manager

Task Scheduling


ServiceArchitecture


Adaptive Infrastructure

- Virtual resources managed dynamically- Tasks executed on the set of evolving resources- Requests of tasks execution received via the bus

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

s

Quality Testing Real-Time Monitoring

Parallel Computing


Grid Middleware

Business Services


Service Level Agreement

Management

Business Processes Workflow

Data Warehouse Reporting

Service Components

Parallel ServicesEnd-to-end Service Testing

End-to-end Monitoring

Resource Manager

Tasks Scheduler

Distributed Shared Memory

SOA Environment

GRID Environment

AI Environment

Policy Enforcement

Registry

Grid Components

Skeletons

Native Runtime Virtual Machine Hypervisor Grid Utility

Proposed architecture

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

sProducts integration mapping

Global contribution- ActiveEon products from the ProActive Suite- Oracle products from the Business Services Suite- HP products from the Virtualization, Quality and SOA

Centers

Alternatives from the open-source among others- ActiveBPEL, Cockpit- Sequoia- Tuscany- Xen VM

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

sProposal

OS, HW

OS Virtualization Grid Utility interface

Enterprise Service Bus

Resource Manager

Task Scheduling



Business Process

Qua

lity

Testi

ng

SLA Mgt

ReportingReal-Tim

e Monitoring


ServiceArchitecture



BPEL EngineBPEL Process Manager

Integrity VM System Insight

BAC Dashboard

BAC

Qua

lity

Cent

er


BIEE

BPA Suite

JDeveloper

DSM Coherence J2EE AS

DB + RAC

Resource Manager

Tasks Scheduler

Grid Utility

IC2D

GCM components

SOA ManagerSOA Systinet

Skeletons

ProActive Hewlett Packard Oracle

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

sOpen Source

OS, HW



Resource Manager

Task Scheduling



Qua

lity

Testi

ngReal-Tim

e Monitoring


ServiceArchitecture



BPEL EngineActive BPEL

Xen VM

IC2D

Apache ServiceMix

Eclipse

Sequoia Jboss / Jonas

MySQL

Resource Manager

Tasks Scheduler

Gtk

Skeletons

Timit

Apache jUDDI

Cockpit

RUDDI

Tuscany

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

s

HP Integration Points

Products & Methodology

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

sHP contribution

Adaptive Infrastructure lab for the project Architecture blueprint draft Consulting & Integration methodology HP Software products from:

- SOA Center (Service Oriented Architecture)- BSM Center (Business Service Management)- VSE Center (Virtual Server Environment)- Quality Center (Quality Test Framework)

Objective: Virtualization of services and resources of the parallel / distributed grid in a business context

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

sAGOS blueprint draft

Parallel Computing

Parallel ServicesDistributed Shared Memory Services

Grid Monitoring

AGOS blueprint


Grid Middleware

Business Services

Business IntelligenceQuality Testing Real-Time Monitoring

Native Operating Systems

Virtualized Operating Systems

Hardware Resource

Virtual Machines

Hypervisor

Resource Monitoring

Grid Utility Interface

Resource Manager

Task / Grid Service

Scheduler

Component Runtime


Parallel ProgrammingDistributed Shared

MemoryPolicy

EnforcementService Registry

Service Component

Business Process Engine

ReportingDataware

HouseService Level Management

Consolidated End-to-End Monitoring

Grid Monitoring

Task / Grid Service

Monitoring

End-to-End Service Testing

Business Workflow

Data Mining

Service Monitoring

9a

8b 8a

11g1

11b

11c

9b

10

11g

MessageMonitoring

5c 5d

5a

16

2b

3a3d

1b

5b

11a

6c

1a

11c1

11c3

4b

11c2

11b3

11b2

11b1

2a

11g2 11g3 11g4

6a

6b

5e

3c3b

1c

4a 11f

7a7b

11c4

12a 12b 12c

13

14a

14b

Benchmarking

14c

15a

4c

15b12d 14d

14d

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

sProducts from HP SOA Center

HP Systinet- UDDI Registry- As the de-facto SOA reference repository

HP Policy Enforcement- Policy Enforcement- As the smart SOA governance runtime

HP BAC for SOA- Services Monitoring- As the integrated SOA services monitoring- Combining business, service and resource monitoring

HP Quality for SOA- Services Testing- As the best-in-class testing environment

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

sIntegrated specifically within AGOS

Provide a library of business processes templates including both functional aspects (gather input data, run simulation, perform pre-processing, analyze output data…) and business aspects (warn user, check execution time, request for more resources…)

Define the library of predefined SOA services Declare all the SOA services to be used by the business

processes in the SOA registry Define and declare all SOA policy enforcements (as proxy

of services) on top of existing SOA services

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

sProducts from HP BSM Center

HP Business Availability Center- Integrated business dashboard- Reporting for business metrics at every layer

• Business processes• Services execution• Resource consumption

HP System Insight Manager- Collect raw metrics from adaptive infrastructure- Feeder of HP BAC systems for real or virtualized

hardware problems

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice


Define or discover the business service models to get monitored

Get in SOA registry all the services to get metrics on the architecture components

Push the indicator values into the BAC monitoring system (source, metrics name + value)

Define the SLA based on customer’s expected results

Define the reports to deliver to the business managers

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

sProducts from HP VSE Center

HP VM- Virtualization of resources and operating systems- Adaptive deployment of resources with hypervisors- Support enterprise hardware and configurations

• Storage Area Network• Blade Servers (Itanium, Pa-Risc, x86)• Hardware & Software High Availability

HP RDP- Resource deployment protocol- Manage VM and installed software transparently

Integration expertise with Xen environment Integration expertise with VMWare environment Thursday 23rd 10:30: HP VSE by Francois Donze

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice


Deploy hardware (cpu, ram, disk) on demand Deploy operating system / virtual machine or

dedicated application through web services Integrate with ProActive resource manager

- Add, get or list available nodes- Define constraints of usage or quality of service

• Deploy within the hour• Schedule the deployment of x VM• Get node with x ram + y cpu

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

s

ProActive used features

Framework in action

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

sProActive Suite

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

sIntegration with ProActive ProActiveAdapterX

OS, HW



Resource Manager

Task Scheduling



Qua

lity

Testi

ngReal-Tim

e Monitoring


Service Architecture



BPEL EngineBPEL Process Manager

Integrity VM System Insight

BAC Dashboard

BAC

Qua

lity

Cent

er


BIEE

BPA Suite

JDeveloper

Coherence J2EE AS

DB + RAC

Resource Manager

Tasks Scheduler

Grid Utility

IC2D

Skeletons

SOA ManagerSOA Systinet

1 42

5

96

3

78

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

sIntegration points with ProActive

1) Generate SCA component as a GCM component2) Expose grid components in SOA Repository3) Provide distributed shared memory’s data to tasks &

components4) Publish metrics from components, skeletons, tasks and

resources5) Make tasks scheduler accessible as a service6) Expose virtual machines as resource nodes7) Expose external grid services (like scheduler as a node

resource hosting a tasks scheduler)8) Provide IC2D aggregated metrics towards monitoring system9) Provide introspection agent to test a particular service,

component, task or object

Arc

hite

ctur

e G

rill

e O

rien

tée

Ser

vice

s

Thank youQuestions / Answers

Architecture Grille Orientée Services ProActive Integration with HP Software in the AGOS project...

Documents

Transcript of Architecture Grille Orientée Services ProActive Integration with HP Software in the AGOS project...