ProActive Parallel Suite

124
Emil Salageanu ProActive Parallel Suite ActiveEon Version February 2008 ActiveEon 1 Overview

description

ProActive Parallel Suite. Overview. ActiveEon Version February 2008. Agenda. ProActive and ProActive Parallel Suite Programming and Composing ProActive Core High Level Programming models ProActive Components Deployment Framework Development Tools. - PowerPoint PPT Presentation

Transcript of ProActive Parallel Suite

Page 1: ProActive Parallel Suite

Emil Salageanu

ProActive Parallel Suite

ActiveEon

Version February 2008

ActiveEon1

Overview

Page 2: ProActive Parallel Suite

Emil Salageanu

Agenda

ProActive and ProActive Parallel Suite Programming and Composing

ProActive Core High Level Programming models ProActive Components

Deployment Framework Development Tools

ActiveEon2

Page 3: ProActive Parallel Suite

Emil Salageanu

Unification of Multi-Threading and Multi-Processing

Multi-Threading

Multi-Core Programming

SMP Symmetric Multi-

Processing Shared-Memory

Parallelism Solutions : OpenMP,

pThreads, Java Threads,…

Multi-Processing

Distributed programming, Grid Computing

MPP Massively Parallel

Programming or Message Passing

Parallelism Solutions: PVM, MPI, RMI,

sockets ,…

3 ActiveEon

Page 4: ProActive Parallel Suite

Emil Salageanu

ProActive

ProActive is a JAVA middleware for parallel, distributed and multi-threaded computing.

ProActive features:programming model a comprehensive framework

ActiveEon4

To simplify the programming and execution of parallel applications within multi-core processors, distributed on Local Area Network (LAN), on clusters and data centers, on intranet and Internet Grids.

Page 5: ProActive Parallel Suite

Emil Salageanu

ProActive Parallel Suite

ProActive Parallel Suite includes: The ProActive middleware featuring services like:

- Fault tolerance, Load balancing, Distributed GC, Security, Web Services

- A set of parallel programming frameworks

- A framework for deploying applications on distributed infrastructures

Software for scheduling applications and resource management Software for monitoring and profiling of distributed applications Online documentation Full set of demos and examples

ActiveEon5

Page 6: ProActive Parallel Suite

Emil Salageanu

ProActive Parallel Suite

6 ActiveEon

ProActive Parallel SuiteProActive Parallel Suite

Physical InfrastructurePhysical Infrastructure

Page 7: ProActive Parallel Suite

Emil Salageanu

ProActive Parallel Suite

7 ActiveEon

Page 8: ProActive Parallel Suite

Emil Salageanu

Ways of using Proactive Parallel Suite?

To easily develop parallel/distributed applications from scratch

Develop applications using well-known programming paradigms thanks to our high-level programming frameworks (master-worker, Branch&Bound, SPMD, Skeletons)

To transform your sequential mono-threaded application into a multi-threaded one (with minimum modification of code) and distribute it over the infrastructure.

ActiveEon8

Page 9: ProActive Parallel Suite

Emil Salageanu

Ways of using Proactive Parallel Suite?

To wrap your native application with ProActive in order to distribute it

Define jobs containing your native-applications and use ProActive to schedule them on the infrastructure

ActiveEon9

Page 10: ProActive Parallel Suite

Emil Salageanu

ProActive Parallel Suite

10 ActiveEon

JVM JVM JVM

Page 11: ProActive Parallel Suite

Emil Salageanu

Distribution model

Open Source Yes, the source code is under GPL version 2 licence.

Close Source No

Commercial support Yes, the ActiveEon company provides commercial support for

the ProActive/GCM middleware.

ActiveEon11

Page 12: ProActive Parallel Suite

Emil Salageanu

Backed up by ActiveEon

12

ActiveEon

Page 13: ProActive Parallel Suite

Emil Salageanu

Agenda

ProActive and ProActive Parallel Suite Programming and Composing

ProActive Core High Level Programming models ProActive Components

Deployment Framework Development Tools

ActiveEon13

Page 14: ProActive Parallel Suite

Emil Salageanu

ProActive Parallel Suite

14 ActiveEon

Page 15: ProActive Parallel Suite

Emil Salageanu

ACTIVE OBJECTS

ProActive Core

ActiveEon15

Page 16: ProActive Parallel Suite

Emil SalageanuActiveEon16

A uniform framework: An Active Object pattern

A programming model: Remote Objects

Asynchronous Communications, Wait-By-Necessity, Groups, Mobility,Components, Security, Fault-Tolerance: Checkpoints

A formal model behind: Determinism (POPL’04)

ProActive A 100% Java API + Tools for

Parallel, Distributed Computing

Page 17: ProActive Parallel Suite

Emil Salageanu

Active Objects

17 ActiveEon

AADeveloper writes

Object AU s e r

With ProActive, he gets …

new A(...) newActive(A,..)

Page 18: ProActive Parallel Suite

Emil Salageanu

ProActive model

18 ActiveEon

Active objects : coarse-grained structuring entities (subsystems) Each active object: - possibly owns many passive objects

- has exactly one thread. Remote Method Invocation (Object RPC) No shared passive objects -- Parameters are passed by deep-

copy Asynchronous Communication between active objects Full control to serve incoming requests (reification)

18

JVM

Page 19: ProActive Parallel Suite

Emil Salageanu

Active objects

19 ActiveEon 19

A

Proxy

Java Object

A ag = newActive (“A”, […], Node)V v1 = ag.foo (param);V v2 = ag.bar (param);...v1.bar(); //Wait-By-Necessity

V

Wait-By-Necessity

provides

Dataflow

Synchronization

JVM

A

JVM

Active Object

Future Object Request

Req. Queue

Thread

v1v2 ag

WBN!

Page 20: ProActive Parallel Suite

Emil Salageanu

Active Object Structure

20 ActiveEon

Page 21: ProActive Parallel Suite

Emil Salageanu

ProActive : Reuse and seamless

Two key features:

Polymorphism between standard and active objects- Type compatibility for classes (and not only interfaces)- Needed and done for the future objects also

Wait-by-necessity: inter-object synchronization- Systematic, implicit and transparent futures- Ease the programming of synchronizations, and the reuse of routines

ActiveEon21 21

Page 22: ProActive Parallel Suite

Emil Salageanu

Unification of Multi-threading and Multi-processing

22 ActiveEon

Most of the time, activities and distribution are not known at the beginning, and change over time

Seamless implies reuse, smooth and incremental transitions

22

Sequential Multithreaded Distributed

Seamless

Page 23: ProActive Parallel Suite

Emil SalageanuActiveEon23

Proofs in GREEK

Page 24: ProActive Parallel Suite

Emil Salageanu

MIGRATION: MOBILE AGENTS

ProActive Core

ActiveEon24

Page 25: ProActive Parallel Suite

Emil Salageanu

Mobile Agents: Migration

Migration is initiated by the active object itself

Can be initiated from outside through any public method

The active object migrates with:• its state

• all pending requests

• all its passive objects

• all its future objects

Automatic update of references:• requests (remote references remain valid)

• replies (its previous queries will be fullfilled)

ActiveEon25 25

Page 26: ProActive Parallel Suite

Emil Salageanu

Migration: Communicating with Mobile Agents

Two strategies

1. Forwarders

2. Location Server

ActiveEon26 26

Page 27: ProActive Parallel Suite

Emil Salageanu

Migration: Forwarders

ActiveEon27 27

Page 28: ProActive Parallel Suite

Emil Salageanu

Migration: Forwarders

ActiveEon28 28

Page 29: ProActive Parallel Suite

Emil Salageanu

Migration: Forwarders

ActiveEon29 29

direct

Page 30: ProActive Parallel Suite

Emil Salageanu

Migration: Forwarders

ActiveEon30 30

direct

direct

Page 31: ProActive Parallel Suite

Emil Salageanu

Migration: Forwarders

ActiveEon31 31

direct

direct

forwarder

Page 32: ProActive Parallel Suite

Emil Salageanu

Migration: Forwarders

ActiveEon32 32

direct

direct

forwarder

Page 33: ProActive Parallel Suite

Emil Salageanu

Migration: Forwarders

ActiveEon33 33

direct

direct

forwarder

Page 34: ProActive Parallel Suite

Emil Salageanu

Migration: Forwarders

ActiveEon34 34

direct

direct

forwarder

Page 35: ProActive Parallel Suite

Emil SalageanuActiveEon35 35

Migration: Location Server

S

Host A

A

Host B Host C Host D

S : SourceA : Agent

referenceServer

Page 36: ProActive Parallel Suite

Emil SalageanuActiveEon36 36

Migration: Location Server

S

Host A

Host B

A

Host C Host D

S : SourceA : Agent

reference

Migration

Server

Server Update

A migrating object updates the server

Page 37: ProActive Parallel Suite

Emil SalageanuActiveEon37 37

Migration: Location Server

S

Host A

Host B Host C Host D

S : SourceA : Agent

reference

Message

MigrationA

Server

UpdateFailed

A migrating object updates the server

Page 38: ProActive Parallel Suite

Emil SalageanuActiveEon38 38

Migration: Location Server

S

Host A

Host B Host C Host D

S : SourceA : Agent

référence

A

ServerAsk for a new

reference

Response

Message

But the AO might have moved again in the meantime … just play again.

!

The source get a new reference from the server

Page 39: ProActive Parallel Suite

Emil Salageanu

PROACTIVE GROUPS

ProActive Core

ActiveEon39

Page 40: ProActive Parallel Suite

Emil Salageanu

ProActive Groups

40 ActiveEon 40

Typed and polymorphic Groups of local and remote objectsDynamic generation of group of resultsLanguage centric, Dot notation

Manipulate groups of Active Objects, in a simple and typed manner:

Be able to express high-level collective communications (like in MPI):• broadcast, • scatter, gather, • all to all

A ag=(A)ProActiveGroup.newGroup(«A»,{{p1},...},{Nodes,..});V v = ag.foo(param); v.bar();

Page 41: ProActive Parallel Suite

Emil Salageanu

ProActive Groups

Group Members Active Objects POJO Group Objects

Hierarchical Groups

Based on the ProActive communication mechanism Replication of N ‘ single ’ communications Parallel calls within a group (latency hiding)

Polymorphism Group typed with member’s type

ActiveEon41 41

Page 42: ProActive Parallel Suite

Emil Salageanu

Two Representations Scheme

42 ActiveEon 42

Group of objects‘Group’

Typed group‘A’

getGroupByTypestatic method of class

ProActive

getGroupmethod of class

Group

Management

of the group

Functional use

of the group

Page 43: ProActive Parallel Suite

Emil Salageanu

Creating AO and Groups

43 ActiveEon 43

A

Typed Group Java or Active Object

A ag = newGroup (“A”, […], Node[])V v = ag.foo(param);...v.bar(); //Wait-by-necessity

V

JVM

Page 44: ProActive Parallel Suite

Emil Salageanu

Typed Group as Result of Group Communication

Ranking Property: Dynamically built and updated

B groupB = groupA.foo();

Ranking property: order of result group members = order of called group members

Explicit Group Synchronization Primitive: Explicit wait

ProActiveGroup.waitOne(groupB);

ProActiveGroup.waitAll(groupB);

PredicatesnoneArrived

kArrived

allArrived, ...

ActiveEon44 44

Page 45: ProActive Parallel Suite

Emil Salageanu

FAULT TOLERANCE SERVICE

ProActive Core

ActiveEon45

Page 46: ProActive Parallel Suite

Emil Salageanu

Fault-tolerance in ProActive

Fault-tolerance is set in a deployment descriptor file Fault-tolerance service attached to resources No source code alteration

Protocol selection

Server(s) location

Checkpoint period

46 ActiveEon

Page 47: ProActive Parallel Suite

Emil Salageanu

Fault-tolerance in ProActive

Rollback-Recovery fault-tolerance After a failure, revert the system state back to some earlier and

correct version Based on periodical checkpoints of the active objects Stored on a stable server

Two protocols are implemented Communication Induced Checkpointing (CIC)

+ Lower failure free overhead– Slower recovery

Pessimistic Message Logging (PML)– Higher failure free overhead+ Faster recovery

Transparent and non intrusive

47 ActiveEon

Page 48: ProActive Parallel Suite

Emil Salageanu

Fault-tolerance Server

Fault-tolerance is based on a global server

This server is provided by the library, with

Checkpoint storage

Failure detectionDetects fail-stop failures

Localization serviceReturns the new location of a failed object

Resource management serviceManages a set of nodes on which restart failed objects

48 ActiveEon

Page 49: ProActive Parallel Suite

Emil Salageanu

SECURITY SERVICE

ProActive Core

ActiveEon49

Page 50: ProActive Parallel Suite

Emil SalageanuActiveEon50

Security

ProActive Security Framework No security related code in the application source code Declarative security language Access control, communication privacy and integrity Security at user- and administrator-level Security context dynamic propagation

Page 51: ProActive Parallel Suite

Emil SalageanuActiveEon51

A ProActive Application

Virtual Node 1

Virtual Node 2

Virtual Node 3

Active object

Page 52: ProActive Parallel Suite

Emil SalageanuActiveEon52

Multiple Deployments

One Host Local Grid Distributed Grids

Page 53: ProActive Parallel Suite

Emil SalageanuActiveEon53

Example: std. code, no security

/…/proActiveDescriptor.activateMappings();vn1 = proActiveDescriptor.getVirtualNode("vm1");vn2 = proActiveDescriptor.getVirtualNode("vm2");/…/Flower rose = (Flower) ProActive.newActive(Flower.class,new Object[]{« Rose »}, vn1.getNode()};Flower daliah = (Flower) ProActive.newActive(Flower.class,new Object[]{« Daliah »}, vn2.getNode()};/* next VN1 node inside the same domain */rose.migrateTo(vn1);/* communication inside the same domain */rose.sayHelloTo(daliah);/* other virtual node, forbidden */rose.migrateTo(vn2);/* next VN1 Node, other domain */rose.migrateTo(vn1);/* communication with another domain */rose.sayHelloTo(daliah);

Page 54: ProActive Parallel Suite

Emil SalageanuActiveEon54

Example

Domain GridA Domain GridB

VN1

VN2

Policy rules databases

JVM

Page 55: ProActive Parallel Suite

Emil SalageanuActiveEon55

Example

Domain GridA Domain GridB

Rose

Daliah

VN1

VN2

Policy rules databases

JVM

Page 56: ProActive Parallel Suite

Emil SalageanuActiveEon56

Example

Domain GridA Domain GridBMigration : - same VN - same domain

Can I migrate to the next VN1 node ?

VN1

VN2

Policy rules databases

JVM

Rose

Daliah

Page 57: ProActive Parallel Suite

Emil SalageanuActiveEon57

Example

Domain GridA Domain GridB

1 - retrieve VN policy2 - migration allowed

Rose

Daliah

VN1

VN2

Policy rules databases

JVM

Migration : - same VN - same domain

Page 58: ProActive Parallel Suite

Emil SalageanuActiveEon58

Example

Domain GridA Domain GridB

Rose

Daliah

VN1

VN2

Policy rules databases

JVM

Migration : - same VN - same domain

Page 59: ProActive Parallel Suite

Emil SalageanuActiveEon59

Example

Domain GridA Domain GridB

Can I migrate to the next VN1 node on

GridB domain?

Rose

Daliah

VN1

VN2

Policy rules databases

JVM

Migration : - same VN - other domain

Page 60: ProActive Parallel Suite

Emil SalageanuActiveEon60

Example

Domain GridA Domain GridB

1- VN1 policy -> none2- GridA -> GridB : [+A,+I,+C] 3- migration with [+A,+I,+C] VN1

VN2

Policy rules databases

JVM

Migration : - same VN - other domain Rose

Daliah

Page 61: ProActive Parallel Suite

Emil SalageanuActiveEon61

Example

Domain GridA Domain GridB

Rose

Daliah

VN1

VN2

Policy rules databases

JVM

Page 62: ProActive Parallel Suite

Emil SalageanuActiveEon62

VN1

VN2

Policy rules databases

JVM

Example

Domain GridA Domain GridBMethod call : - other VN - other domainFrom Rose --> Daliah

Daliah -> Rose : [+A,+I,?C]

Negotiated Policy:Rose -> Daliah : [+A,+I,+C]

Daliah

Rose

Rose -> Daliah : [+A,?I,+C]

Page 63: ProActive Parallel Suite

Emil SalageanuActiveEon63

SSH Tunneling

A fact : overprotected clustersFirewalls prevent incoming connections

Use of private addresses

NAT, IP Address filtering, …

A consequence :

Multi clustering is a NIGHTMARE

Context :SSH protocol : encrypt network traffic

Administrators accept to open SSH port

SSH provides encryption

Page 64: ProActive Parallel Suite

Emil SalageanuActiveEon64

Without SSH Tunneling

Denied

Runtime Active object Communications

Page 65: ProActive Parallel Suite

Emil SalageanuActiveEon65

With SSH Tunneling

SS

H

For

wa

rder

SS

H

serv

er

SS

H

For

wa

rder

Runtime Active object Communications Secure channel

Page 66: ProActive Parallel Suite

Emil Salageanu

WEB SERVICES

ProActive Core

ActiveEon66

Page 67: ProActive Parallel Suite

Emil Salageanu

Web Service Integration

Aim... Turn active objects and components interfaces into Web Services

interoperability with any foreign language or any foreign technology.

API Expose an active object as a web Service on a web server (the

user can choose the methods he wants to expose)

exposeAsWebService(Object o, String url, String urn, String [] methods );

Expose the interfaces of a component as web services :

exposeComponentAsWebService(Component component, String url, String componentName );

67 ActiveEon

Page 68: ProActive Parallel Suite

Emil Salageanu

ProviderProvider

Web applicationserver

ProActive

ProActive.exposeAsWebServic

e (………)

.NETC#

-WSDL fileUrn=

‘piComputation’

1. ProActive.exposeAsWebService ()1. ProActive.exposeAsWebService ()2. Deployment2. Deployment

ProActive Comm.

3. Client Call3. Client Call68 ActiveEon

Page 69: ProActive Parallel Suite

Emil Salageanu

Agenda

ProActive and ProActive Parallel Suite Programming and Composing

ProActive Core High Level Programming models ProActive Components

Deployment Framework Development Tools

ActiveEon69

Page 70: ProActive Parallel Suite

Emil Salageanu

ProActive Parallel Suite

70 ActiveEon

Page 71: ProActive Parallel Suite

Emil Salageanu

High Level Programming models

Master-Worker Framework

ActiveEon71

Page 72: ProActive Parallel Suite

Emil Salageanu

Motivations

Embarrassingly parallel problems : simple and frequent model

Write embarrassingly parallel applications with ProActive :

May require a sensible amount of code (fault-tolerance, load-balancing, …).

Requires understanding of ProActive concepts ( Futures, Stubs, Group Communication )

ActiveEon72

Page 73: ProActive Parallel Suite

Emil Salageanu

Goals of the M/W API

Provide a easy-to use framework for solving embarrassingly parallel problems: Simple Task definition Simple API interface (few methods) Simple & efficient solution gathering mechanism

Provide automatic fault-tolerance and load-balancing mechanism

Hide ProActive concepts from the user

ActiveEon73

Page 74: ProActive Parallel Suite

Emil SalageanuActiveEon74

C re a teS la ve 1

C re a teS la ve 2

C re a teS la ve 3

C re a teS la ve n

How does it work?

U s e r

D e p lo y m e nt D e s c rip to rM a s te r

S o lv eT a s k 1 ... T a s k M

p u b lic c la s s M yT a s k im p le m e n ts T a s k <S trin g >{

p ub l ic S trin g run () {re tu rn "H e llo W o rld !" ;

}}

T a s k D e fin itio n

S c he d uleT a s k 1

S c he d uleT a s k 2

S c he d uleT a s k 3

S c he d uleT a s k n

S e ndR e s ult1

S e ndR e s ult2

S e ndR e s ult3

S e ndR e s ultn

R e su lts

1 Res ult1

2 Res ult2

3 Res ult3

4 Res ult4

...

n Res ultn

...

MS c he d uleT a s k n+1

S la ve 1 S la ve 2 S la ve 3 S la ve n

R e s ult1 ... R e s ultM

Page 75: ProActive Parallel Suite

Emil Salageanu

M/W Success Story: Artificial Life Generation

75 ActiveEon

Initial Application (C++)

1 PC 56h52 => Crashed

ProActive Version 300 CPUs 19 minutes

Sylvain Cussat-Blanc, Yves Duthen – IRIT TOULOUSE

Application D0+1 D0+5 D0+6 D0+7

ProActive Version

251300 CPUs

Developpement of artificial creatures

Page 76: ProActive Parallel Suite

Emil Salageanu

Comparison between specific implementation and M/W

Experiments with nQueens problem Runs up to 25 nodes

ActiveEon76

NQueensOpt vs MasterWorker

00:00:00

00:14:24

00:28:48

00:43:12

00:57:36

01:12:00

01:26:24

20 40 60 80 100

number of nodes used

co

mp

uta

tio

n t

ime

Nqueens Opt

MasterWorker

Page 77: ProActive Parallel Suite

Emil Salageanu

High Level Programming models

Skeletons Framework

ActiveEon77

Page 78: ProActive Parallel Suite

Emil Salageanu

Algorithmic Skeletons

High Level Programming Model Hides the complexity of parallel/distributed

programming. Exploits nestable parallelism patterns

ActiveEon78

Page 79: ProActive Parallel Suite

Emil Salageanu

Skeletons Big Picture

Parameters/Results are passed through streamsStreams are used to connect skeletons (CODE)

ActiveEon79

Input Stream

Parameter (Data)

Skeleton Code

Output Stream

Solved: Results

Page 80: ProActive Parallel Suite

Emil Salageanu

Pipe Skeleton

Represents computation by stages. Stages are computed in parallel for different

parameters.

ActiveEon80

R1P

2

R1P

2P

3

P3P

4P

5

P4

P5

Tim

e

Skeleton 1 Skeleton 2

Input Stream Output StreamExecute Skeleton

Page 81: ProActive Parallel Suite

Emil Salageanu

Simple use of Pipe skeleton

ActiveEon81

Page 82: ProActive Parallel Suite

Emil Salageanu

High Level Programming models

Branch-and-Bound Framework

ActiveEon82

Page 83: ProActive Parallel Suite

Emil Salageanu

Branch & Bound API (BnB)

Provide a high level programming model for solving BnB problems: manages task distribution and provides task communications

Features: Dynamic task split Automatic result gather Broadcasting best current result Automatic backup (configurable)

83ActiveEon

Page 84: ProActive Parallel Suite

Emil Salageanu84

Global Architecture : M/W + Full connectivity

JVM

JVMTask

Worker

JVMTask

Worker

JVMTask

Worker

JVMTask

Worker

Task Queue

Manager

ActiveEon

Page 85: ProActive Parallel Suite

Emil Salageanu

High Level Programming models

OO-SPMD

ActiveEon85

Page 86: ProActive Parallel Suite

Emil SalageanuActiveEon86

Object-Oriented Single Program Multiple Data

Motivation Cluster / GRID computing SPMD programming for many numerical simulations Use enterprise technology (Java, Eclipse, etc.) for Parallel

Computing

Able to express most of MPI’s Collective Communications (broadcast, gathercast, scattercast,..) Barriers Topologies

With a small object-oriented API

Page 87: ProActive Parallel Suite

Emil SalageanuActiveEon87

Execution example A ag = newSPMDGroup (“A”, […], VirtualNode)

// In each member myGroup.barrier (“2D”); // Global Barrier myGroup.barrier (“vertical”); // Any Barrier myGroup.barrier (“north”,”south”,“east”,“west”);

A

Page 88: ProActive Parallel Suite

Emil SalageanuActiveEon88

Topologies

Topologies are typed groups Customizable Define neighborhood

Plan plan = new Plan(groupA, Dimensions);Line line = plan.getLine(0);

Page 89: ProActive Parallel Suite

Emil Salageanu

High Level Programming models

Workflow

ActiveEon89

Page 90: ProActive Parallel Suite

Emil Salageanu

Programming with flows of tasks

Program an application as an ordered tasks set Logical flow : Tasks execution are orchestrated Data flow : Results are forwarded from ancestor tasks to their children as

parameter

The task is the smallest execution unit Two types of tasks:

Standard Java Native, i.e. any third party application

ActiveEon90

Task 1(input 1) Task 2(input 2)

Task 3(res1,res2)

res1 res2

Page 91: ProActive Parallel Suite

Emil Salageanu

Defining and running jobs with ProActive A workflow application is a job

a set of tasks which can be executed according to a

dependency tree

Rely on ProActive Scheduler only

Java or XML interface Dynamic job creation in Java Static description in XML

Task failures are handled by the ProActive Scheduler A task can be automatically re-started or not (with a user-defined

bound) Dependant tasks can be aborted or not The finished job contains the cause exceptions as results if any

ActiveEon91

Page 92: ProActive Parallel Suite

Emil Salageanu

Agenda

ProActive and ProActive Parallel Suite Programming and Composing

ProActive Core High Level Programming models ProActive Components

Deployment Framework Development Tools

ActiveEon92

Page 93: ProActive Parallel Suite

Emil Salageanu

ProActive Parallel Suite

93 ActiveEon

Page 94: ProActive Parallel Suite

Emil Salageanu

A framework for Grid components

Facilitating the design and implementation of complex distributed systems

Leveraging the ProActive libraryProActive components benefit from underlying features

Allowing reuse of legacy components (e.g. MPI)

Providing tools for defining, assembling and monitoring distributed components

94 ActiveEon

Page 95: ProActive Parallel Suite

Emil Salageanu

Component - What is it ?

A component in a given infrastructure is:

a software module,

with a standardized description of what it needs and provides,

to be manipulated by tools for Composition and Deployment

ActiveEon95

C

Page 96: ProActive Parallel Suite

Emil Salageanu

ProActive Component Definition A component is:

Formed from one (or several) Active Object Executing on one (or several) JVM Provides a set of server ports: Java Interfaces Uses a set of client ports: Java Attributes Point-to-point or Group communication between components

Hierarchical: Primitive component: define with Java code and a descriptor Composite component: composition of primitive + composite Parallel component: multicast of calls in composites

Descriptor: XML definition of primitive and composite (ADL) Virtual nodes capture the deployment capacities and needs

Virtual Node: a very important abstraction for GRID components

ActiveEon96

Page 97: ProActive Parallel Suite

Emil Salageanu

ProActive Components for the GRID

97 ActiveEon

3. Parallel and composite component

1. Primitive component

2. Composite component

An activity, a process, …potentially in its own JVM

C D

Composite: Hierarchical, and

Distributed over machines

Parallel: Composite

+ Broadcast (group)

Page 98: ProActive Parallel Suite

Emil Salageanu

Components vs. Activity and JVMs

98 ActiveEon

Components are orthogonal to activities and JVMs They contain activities, span across

several JVMs

Components are a way to globally manipulate distributed, and running activities

Activity JVM Component

A B1

C

B2

B3

Page 99: ProActive Parallel Suite

Emil Salageanu

Agenda

ProActive and ProActive Parallel Suite Programming and Composing

ProActive Core High Level Programming models ProActive Components

Deployment Framework Development Tools

ActiveEon99

Page 100: ProActive Parallel Suite

Emil Salageanu

ProActive Parallel Suite

100 ActiveEon

Page 101: ProActive Parallel Suite

Emil Salageanu

GCM Deployment

101 ActiveEon

Page 102: ProActive Parallel Suite

Emil SalageanuActiveEon102

Abstract Deployment Model

Problem

Difficulties and lack of flexibility in deployment

Avoid scripting for configuration, getting nodes, connecting,…

A key principle: Virtual Node (VN) in XML deployment file

Abstract Away from source code:– Machines names– Creation/Connection Protocols– Lookup and Registry Protocols

Interface with various protocols and infrastructures:– Cluster: LSF, PBS, SGE , OAR and PRUN(custom protocols)

– Intranet P2P, LAN: intranet protocols: rsh, rlogin, ssh

– Grid: Globus, Web services, ssh, gsissh

Page 103: ProActive Parallel Suite

Emil Salageanu

Virtualization of resources

ApplicationApplication

Deployment DescriptorDeployment Descriptor

Mapping

ConnectionsConnections

NodesNodes

AcquisitionAcquisition

CreationCreation

Infrastructure

VN

Runtime structured entities: 1 VN --> n Nodes in m JVMs on k Hosts

103 ActiveEon

Page 104: ProActive Parallel Suite

Emil Salageanu

Virtualization of resources

104 ActiveEon

VN1

VN2

GCM XMLDeployment Descriptor

node

node

HostJVM

node

JVM

Host

node

JVM

Page 105: ProActive Parallel Suite

Emil Salageanu

Virtualization of resources

105 ActiveEon

VN1

VN2

node

node

HostJVM

node

JVM

Host

node

JVM

Page 106: ProActive Parallel Suite

Emil Salageanu

Multiple Deployments

106 ActiveEon

One Host Local Grid Distributed Grids

Internet

Page 107: ProActive Parallel Suite

Emil Salageanu

File Transfer

107 ActiveEon

Page 108: ProActive Parallel Suite

Emil Salageanu

Data management: File transfer

Integrated with Deployment: • Resource acquisition file transfer when available: unicore, nordugrid, Globus• Other non-deployment protocols: scp, rcp, …• ProActive own protocol when other fails: ProActive Failsafe File Transfer

ProActive supports the following transfers:• Push: To a remote node• Pull: From a remote node• Triangle: Triggered from node A, occurring between node B C

It can be used at:• Deployment time (XML)• Execution at any time (API)• Retrieval time (XML)

ActiveEon108

Page 109: ProActive Parallel Suite

Emil Salageanu

Deployment File Transfer

<fileTransferDefinitions> <fileTransfer id="ProActiveLite"> <file src="lib/ProActive.jar" dest="ProActive.jar"/> <file src="lib/log4j.jar" dest="log4j.jar" /> <file src="lib/components/fractal.jar" dest="fractal.jar”/> <file src="lib/xercesImpl.jar" dest="xercesImpl.jar" /> <file src="lib/bouncycastle.jar" dest="bouncycastle.jar"/> <file src="lib/jsch.jar" dest="jsch.jar" /> <file src="lib/javassist.jar" dest="javassist.jar"/> <file src="scripts/proactive.java.policy" dest="proactive.java.policy"/> <file src="scripts/proactive-log4j"dest="proactive-log4j"/> </fileTransfer></fileTransferDefinitions>

109 ActiveEon

Page 110: ProActive Parallel Suite

Emil Salageanu

Deployment File Transfer

<sshProcess class="org.objectweb.proactive.core.process.ssh.SSHProcess" hostname="sea.inria.fr">

<processReference refid="remoteJVM"/>

<fileTransferDeploy refid="ProActiveLite"> <copyProtocol>rcp,scp</copyProtocol> <sourceInfo prefix="/home/user/ProActive/"/> <destinationInfo prefix="/tmp"/> </fileTransferDeploy>

</sshProcess>

110 ActiveEon

Page 111: ProActive Parallel Suite

Emil Salageanu

The ProActive P2P

111 ActiveEon

Page 112: ProActive Parallel Suite

Emil Salageanu

The ProActive P2P

Unstructured P2PEasier to deploy/manageOnly 1 resource : CPU

Java codeEach peer is written in Java and can run any Java

application

Direct communicationsPeers are reachable using their name (URLs)One peer can send/receive a reference on another

peer

112 ActiveEon

Page 113: ProActive Parallel Suite

Emil Salageanu

The ProActive P2P (2)

P2P Infrastructure

Resource Management Direct Access

Applications

113 ActiveEon

Page 114: ProActive Parallel Suite

Emil Salageanu

Infrastructure

A peer is an Active Object in a JVM Each peer knows a limited number of other

peers (bi-directional links) Its acquaintancesThe number is set by a variable (NOA)

Goal of a peerA peer will always try to maintain the number of its

acquaintances equals to its NOA 2 basic operations

Adding an acquaintanceRemoving an acquaintance

114 ActiveEon

Page 115: ProActive Parallel Suite

Emil Salageanu

Requesting Nodes

To request a nodeContact only a Peer (URLs)

The infrastructure will handle the reservation

The application will have to wait until the nodes are available

Using the P2P network JAVA API XML

115 ActiveEon

Page 116: ProActive Parallel Suite

Emil Salageanu

Scheduler and Resource manager

116 ActiveEon

Page 117: ProActive Parallel Suite

Emil Salageanu

Scheduler / Resource Manager Overview

117 ActiveEon

• Multi-platform Graphical Client (RCP)

• File-based or LDAP authentication

• Static Workflow Job Scheduling, Native and Java tasks, Retry on Error, Priority Policy, Configuration Scripts,…

• Dynamic and Static node sources, Resource Selection by script, Monitoring and Control GUI,…

• ProActive Deployment capabilities : Desktops, Clusters, ProActive P2P,…

Page 118: ProActive Parallel Suite

Emil Salageanu

Agenda

ProActive and ProActive Parallel Suite Programming and Composing

ProActive Core High Level Programming models ProActive Components Legacy code wrapping

Deployment Framework Development Tools

ActiveEon118

Page 119: ProActive Parallel Suite

Emil Salageanu

ProActive Parallel Suite

119 ActiveEon

Page 120: ProActive Parallel Suite

Emil Salageanu

IC2D

Features:

Graphical visualization Textual visualization Monitoring and Control

120 ActiveEon

Interactive Control & Debug for Distribution

Page 121: ProActive Parallel Suite

Emil Salageanu

IC2D: Monitor your application in real-time

121 ActiveEon

Page 122: ProActive Parallel Suite

Emil Salageanu

TimIt: Automatic Timers in IC2D

122 ActiveEon

Page 123: ProActive Parallel Suite

Emil Salageanu

Analysis and Optimization

123 ActiveEon

Page 124: ProActive Parallel Suite

Emil Salageanu

Now, let’s play with ProActive…

124 ActiveEon

Start and monitor with IC2D the ProActive examples, and have a look at the source codeorg.objectweb.proactive.examples.*

Features ApplicationsBasics, Synchronization Doctors problem (doctors.bat), Reader/Writer

problem (readers.bat),…

Futures, Automatic Continuation

Binary Search Tree (bintree.bat)

Migration Migrating Agent (/migration/penguin.bat)

Group Chat (/group/chat.bat)

Fault-Tolerance N-body problem (/FT/nbodyFT.bat)

All Distributed 3D renderer (c3d*.bat)