Post on 18-Jan-2016
DISCOGRIDSophia, 13-14 mars 2006
The Padico Environment
Christian Pérez
PARIS Research Team, IRISA/INRIA, Rennes, France
Outline
Motivation A Distributed Parallel Component Model
GridCCM Handing Network Issues
PadicoTM Automatic Deployment of Applications
ADAGE Conclusion
Programming Grid?
Visualization
Homogeneous cluster
SAN
SAN
Homogeneous cluster
LAN
WAN
Supercomputer
Flowcode
Transport code
Controller
velocity
scalarsscalars
concentration
Programming Environment( Virtualization)
Requested Features Support of legacy codes
Multi-languages, multi-OS,multi-architectures
Support of parallel codes MPI, PVM, OpenMP, ...
Support for network transparency Support for deployment
“Single-provider” application “Multi-provider” application
Software component!
Transport
Flow
Distributed components: OMG CCM A distributed component-oriented
model An architecture for defining
components and their interaction Interaction implemented through
input/output interfaces Synchronous and asynchronous
communication A runtime environment based on the
notion of containers (lifecycle, security, transaction, persistent, events)
Multi-languages/OS/ORBs/vendors, … With a packaging/deployment
model A packaging technology for deploying
binary multi-lingual executables Could be deployed and run on several
distributed nodes simultaneously
MyComponent
Component interface
Facets
Eventsources
Eventsinks
Attributes
Receptacles
OF
FE
RE
D
RE
QU
IRE
D
Component-based application
Courtesy of P. Merle, LIFL, France
0
50
100
150
200
250
1 100 10000 1000000 100000000Message size (bytes)
CORBA/Myrinet-2000
MPI/Myrinet-2000
CORBA/SCI
MPI/SCI
Bandwidth (MB/s)
High Performance CORBA Request
Use a zero-copy ORB OmniORB
Use a High Perf network Myrinet-2000
Bandwidth Myrinet-2000 : 240MB/s 96% of the hardware
bandwidth! Latency
CORBA/Ethernet : 150 µs CORBA/Myrinet-2000 : 18
µs MPI/Myrinet-2000 : 11 µs
Bandwidth of several runtimes over Myrinet-2000 and SCI networks
Component and Parallelism
How to embed a parallel code into a component?
Parallel Code A Parallel Code B
MPI PVM
Parallel Component
HPCComponent
A
HPCComponent
B
What the application designer should see…
… and how it must be implemented !
// Comp. A // Comp. B
Distributed Component Model
Caller Receiver
// Emitter Codeo->factorize(m);
Distributed component (a process)
// Receiver Codevoid serv::factorize(const Matrix mat) { ... ;}
SPMD Parallel Component Model (1)
DataRedistribution
Caller Parallel Receiver
Distributed component (a process)
// SPMD Receiver codevoid factorize(const DMatrix mat) { .... MPI_Bcast(...) ... ;}
Parallel Distributed Component
// Emitter Codeo->factorize(m);
SPMD Parallel Component Model (2)
DataRedistribution
Parallel Caller Receiver
// Receiver Codevoid factoriser(const Matrice mat) { ... ;}
Distributed component (a process) Parallel Distributed Component
// SPMD Emitter Codeo->factorize(m);
Parallel ReceiverParallel Caller
// SPMD Emitter Codeo->factorize(m);
// SPMD Receiver Code SPMDvoid factorizer(const Matrix mat) { .... MPI_Bcast(...) ... ;}
DataRedistribution
Distributed component (a process) Parallel Distributed Component
SPMD Parallel Component Model (3)
GridCCM: Parallel CORBA Component
Extends the CORBA Component Model No modification of CCM specification Portable Extension of CCM descriptor (XML)
Parallel component Parallelism is a non-functional
property of a component It is an implementation issue
Collection of sequential component SPMD execution model
Support of distributed arguments API for data redistribution API for communication scheduling w.r.t. network
properties Support of parallel exceptions
Parallel Comp.
SPMDProc.SPMDProc.SPMDProc.SPMDProc.SPMDProc.
GridCCM Architecture Overview
Object Request Broker
CORBA stub/skeleton
CommunicationLibrary ( MPI)
Application
Client view management- Data distribution description
Communication management- Communication matrix computation- Communication scheduling computation- Communication execution
Redistribution Library 1
CommunicationLibrary
GridCCM runtime
SchedulingLibrary
Latency SAN
Myrinet-200/GM Follows MPI barrier
times Pt-2-Pt overhead
CORBA : 25 µs PaCO++ : 35 µs
WAN French VTHD
Network 2 Gbit/s ~ 1000 Km long
Negligible overhead 10 µs << 8 ms
0
20
40
60
80
100
120
1 2 3 4 5 6 7 8
Nombre de nœuds par objet parallèle
PaCO++
MPI
Latence en microsecondes
Nodes milliseconds
1x1 8.01
2x2 8.04
4x4 8.08
8x8 8.22
16x16 8.40
Bandwidth SAN
Myrinet-2000/GM Aggregation
4 nodes :600 MB/s 4x150 MB/s
WAN French VTHD Network 1 Gbit/s switch
Saturate for n > 12 Sustaining bandwidth
Using a Scheduling Algorille
algorithmINRIA/LORIA
But need to knownetwork properties
0
100
200
300
400
500
600
700
1 10 100 1000 10000 100000 1000000 10000000 1E+08
Taille des messages (octet)
1 nœud par objet parallèle
2 nœuds par objet parallèle
4 nœuds par objet parallèle
Bande passante agrégée (Mo/s)
0
20
40
60
80
100
120
6 7 8 9 10 11 12 13 14 15 16
Nombre de nœuds par objet parallèle
Sans ordonnancement
Avec ordonnacement
Bande passante agrégée (Mo/s)
Relationship between Models
GridCCM: a parallel distributed component model Extension of CCM Based on PaCO++
PaCO++: a parallel distributed object model Extension of CCM
CORBA ObjectModel
CORBA ComponentModel (CCM)
Parallel ObjectModel (PaCO++)
Parallel ComponentModel (GridCCM)
Parallel Extension Parallel Extension
ComponentExtension
ComponentExtension
Status PaCO
Obsolete, not maintain any longer PaCO++
Available on the Web (version 0.2) LGPL/GPL licence, cf http://www.irisa.fr/paris/Padico
MPI support Data Redistribution
Identity, 1D Bloc-cyclic, “EPSN” Still going on
GridCCM Not yet available, only prototype version Being developed
Outline
Motivation A Distributed Parallel Component Model
GridCCM Handing Network Issues
PadicoTM Automatic Deployment of Applications
ADAGE Status Conclusion
Deploy the same code using different networking resources
SAN
SAN
LAN
WAN
process
processusprocessus
processusprocessusprocess
processusprocessusprocessus
processusprocess
Configuration 1
process
processusprocessus
processusprocessusprocess
processusprocessusprocessus
processusprocess
Configuration 2
WAN
SAN
SAN
High performance middleware
Middlewareover a WAN
The middleware adapt itself to the available network
Mapping Objects onto Resources
Handling network issues: PadicoTM
Network heterogeneity Several networks, Non IP network, parallel stream,
online compression, firewalls Decouple communication API from actual
network Enables different access to the networks Transparent network access High Performance provided
to middleware systems Based on PM2
Consequences:• Control the mapping of
a middleware toa network
• Add special adapters
TCP/IP Myrinet SCI ...
AdOC
ParallelStream
PadicoTM
MPI CORBA SOAP JXTA
SSL
0
50
100
150
200
250
1 100 10000 1000000 100000000Message si ze (bytes)
CORBA/ Myr i net - 2000
MPI / Myr i net - 2000
J ava/ Myr i net - 2000CORBA/ SCI
MPI / SCI
TCP/ Et her net - 100
Bandwi dth (MB/ s)
Bandwidth Myrinet-2000 : 240MB/s SCI : 86MB/s 96% of the hardware
bandwidth for CORBA,MPI and Java
Latency CORBA/Myrinet-2000: 18 µs MPI/Myrinet-2000: 11 µs Java/Myrinet-2000: 40 µs CORBA/Ethernet: 150 µs
PadicoTM-enabled Middleware bandwidth over Myrinet-2000 and SCI
PadicoTM-enabled Middleware Perfs
WAN PadicoTM Performance Parallel streams
VTHD Network (Rennes-Nice) Without : 8 MB/s (64 Mb/s) With (4 streams): 12 MB/s (96 Mb/s) For all runtimes
Overhead negligible
Adaptive Compression (AdOC) ADSL link 608 kbit/s Without compression : 580 kbit/s With compression : 800 to 1200
kbit/s For all runtimes
Overhead negligible The VTHD network
Middleware and methods supported by PadicoTM : MPI mpich-1.1.2 & mpich-1.2.4 (ch_mad)
CORBA omniORB omniORB-3.0.2 through omniORB-4.0.6 MicoCCM mico-2.3.4 through mico-2.3.12 others Iona ORBacus-4 and Orbix/E
SOAP gSOAP-2.1 through 2.6 JVM kaffe-1.0.6 HLA Certi-3.0 (Onera) DSM Mome (Y. Jégou) ICE ICE-1.0 & 1.1 (ZeroC Inc.) JXTA JXTA-C (CVS cersion)
PadicoTM 0.1.5 available 0.3 release coming very soon
Now, co-developed with RUNTIME/LaBRI http://runtime.futurs.inria.fr/PadicoTM
PadicoTM Status
Outline
Motivation A Distributed Parallel Component Model
GridCCM Handing Network Issues
PadicoTM Automatic Deployment of Applications
ADAGE Status Conclusion
How to Deploy my Applicationon Grid Resources?
Visualization
Homogeneous cluster
SAN
SAN
Homogeneous cluster
LAN
WAN
Supercomputer
Flowcode
Transport code
Controller
velocity
scalarsscalars
concentration
Manual Deployment
Discover available grid resources Select grid resources for execution
OS, architecture compatibility Map the application onto the selected
resources MPI processes Components
Select compatible compiled executables Upload and install executables, stage input
files in Launch processes on remote computers Set configuration parameters of the application
Components' attributes Network topology information
Too complex!
Automatic Deployment Automatic
Resource discovery Execution node selection File installation Process launch Application configuration
Hide application complexity
Hide grid complexity
stop reading your e-mails!
Grid
Identification of the Stepsof Automatic Deployment
MPI Application Description
CCM Application Description
Resource Description
Generic Application Description
Control Parameters
Deployment Planning
Deployment Plan Execution
Application Configuration
Stat
ic
Applic
atio
ns
Deployme
nt Tool
Generic Application Description
Translator From specific to generic application description Straightforward to write
CCM Appl. Descriptio
n
MPI Appl. Descriptio
n
GridCCM Appl.
Description
CCM Appl. Descriptio
n
MPI Appl. Descriptio
n
GridCCM Appl.
Description
CCM Planner
MPI Planner
GridCCM Planner
Plan Execution
Plan Execution
Plan Execution
Generic Application Description
Deployment Planner
Deployment Plan Execution
ADAGE Automatic Deployment of Applications in a Grid
Environment http://www.irisa.fr/paris/ADAGE/
Already implemented MPI, CCM, and JXTA application description Network topology description Generic application description Simple control parameters, simple planner Deployment plan execution (SSH, Globus)
Not fully implemented yet GridCCM application description and configuration Integrate more planners (GRAAL, LIP, Lyon)
How Easy is it, in Practice?
As simple as A-B-C adage_grid_deploy –resource http://host/file.xml
–application my_appl.zip–ctrl_param
my_control_parameters Grid resource description
Written once for all by grid admins Optional control_parameters constraints
Keep control on the deployment process
Conclusion Software component is a promising technology
for handling code & grid complexity Padico = GridCCM + PadicoTM + ADAGE
http://www.irisa.fr/paris/Padico GridCCM : extends CCM
Parallel component, scalable communication achieved Data redistribution / communication scheduling API
Approach applicable to other component/service models PadicoTM: an open integration framework for
communication middleware and runtimes Decouple middleware from actual networking technology Supports several middleware at the same time
Adage: Automatic Deployment of Applications in a Grid Environment Application/resource constraints, user goal, and mapping
algorithm needed