Parallel and Distributed Scilab Simulations with ProActive ... · Parallel and Distributed Scilab...
Transcript of Parallel and Distributed Scilab Simulations with ProActive ... · Parallel and Distributed Scilab...
Agenda
1. Background: INRIA, ActiveEon
2. ProActive Open Source: Programming, Scheduling, Resourcing
3. Parallel Scilab Simulations
4. ANR OMD2: Distributed Multi-Disciplinary Optimizations
5. Conclusion: Cloud Portal ProActive PACA Cloud
D. Caromel, et al.
Parallel and Distributed
Scilab Simulations with
ProActive Parallel Suite
Parallelism is Complex:
Hiding Details and Seamless Integration with Scilab
3 3
OASIS Team & INRIA
A joint team, Now about 35 persons
2004: First ProActive User Group
2009, April: ProActive 4.1, Distributed & Parallel:
From Multi-cores to Enterprise GRIDs
4 4
OASIS Team Composition (35)
Researchers (5): D. Caromel (UNSA, Det. INRIA)
E. Madelaine (INRIA)
F. Baude (UNSA)
F. Huet (UNSA)
L. Henrio (CNRS)
PhDs (11): Antonio Cansado (INRIA, Conicyt)
Brian Amedro (SCS-Agos)
Cristian Ruz (INRIA, Conicyt)
Elton Mathias (INRIA-Cordi)
Imen Filali (SCS-Agos / FP7 SOA4All)
Marcela Rivera (INRIA, Conicyt)
Muhammad Khan (STIC-Asia)
Paul Naoumenko (INRIA/Région PACA)
Viet Dung Doan (FP6 Bionets)
Virginie Contes (SOA4ALL)
Guilherme Pezzi (AGOS, CIFRE SCP)
+ Visitors + Interns
PostDoc (1): Regis Gascon (INRIA)
Engineers (10):
Elaine Isnard (AGOS)
Fabien Viale (ANR OMD2, Renault )
Franca Perrina (AGOS)
Germain Sigety (INRIA)
Yu Feng (ETSI, FP6 EchoGrid)
Bastien Sauvan (ADT Galaxy)
Florin-Alexandru.Bratu (INRIA CPER)
Igor Smirnov (Microsoft)
Fabrice Fontenoy (AGOS)
Open position (Thales)
Trainee (2): Etienne Vallette d’Osia (Master 2 ISI)
Laurent Vanni (Master 2 ISI)
Assistants (2): Patricia Maleyran (INRIA)
Sandra Devauchelle (I3S) Located in Sophia Antipolis, between
Nice and Cannes, Visitors Welcome!
5 5
Co-developing, Support for ProActive Parallel Suite
Worldwide Customers: Fr, UK, Boston USA
Startup Company Born of INRIA
Some Partners:
Some Customers:
7
Product: ProActive Parallel Suite
Java Parallel
Toolkit
Multi-Platform
Job Scheduler
Resource
Manager
Strong Differentiation:
Java Parallel Programming + Integration + Portability: Linux, Windows, Mac +
Versatility: Desktops, Cluster, Grid, Clouds = Perfect Flexibility
Used in Production Today:
50 Cores 300 Cores early 2010
12 12 12
Broadcast and Scatter
JVM
JVM
JVM
JVM
ag cg
ag.bar(cg); // broadcast cg
ProActive.setScatterGroup(cg);
ag.bar(cg); // scatter cg
c1 c2 c3 c1 c2 c3
c1 c2 c3 c1 c2 c3
c1 c2 c3
c1 c2 c3
s
c1 c2 c3
s
Broadcast is the default behavior
Use a group as parameter, Scattered depends on rankings
13 13 13
Dynamic Dispatch Group
JVM
JVM
JVM
JVM
ag cg
c1
c2
c3
c4
c5
c6
c7
c8 c0
c9 c1
c2
c3
c4
c5
c6
c7
c8 c0
c9
c1
c2
c3
c4
c5
c6
c7
c8 c0
c9
Slowest
Fastest
ag.bar(cg);
Key Point
“MPI and programming languages from
the 60’s will not make it”
Jack Dongarra, 2/13/2009,
Wake Forest University talk
“It is time to get ride of MPI”
Alan Edelman, MIT, 06/16/2010,
ScilabTec’10 Users’ Day
18 18 18
OO SPMD: Object-Oriented SPMD A ag = newSPMDGroup (“A”, […], VirtualNode)
// In each member
myGroup.barrier (“2D”); // Global Barrier
myGroup.barrier (“vertical”); // Any Barrier
myGroup.barrier (“north”,”south”,“east”,“west”);
A
27
ProActive Scheduling Big Picture
RESOURCES
Multi-platform Graphical Client (RCP)
File-based or LDAP authentication
Static Workflow Job Scheduling, Native and
Java tasks, Retry on Error, Priority Policy, Configuration Scripts,…
Dynamic and Static node sources, Resource
Selection by script, Monitoring and Control GUI,…
ProActive Deployment capabilities:
Desktops, Clusters, Clouds,…
ProActive
Scheduler
ProActive
Resource Manager
28
Job
Workflow Example : Picture Denoising
Split
Denoise Denoise Denoise Denoise
Merge
•with selection on native executable availability (ImageMagik, GREYstoration)
• Multi-platform selection and command generation •with file transfer in pre/post scripts
32
Node source Usecase :
Configuration for external cloud with EC2
ProActive
Scheduler
ProActive
Resource Manager
Dedicated resources
LSF
Static Policy
Amazon EC2
EC2
Dynamic
Workload Policy
Desktops
Desktops
Timing Policy
12/24
35
Seamless Parallel & Distributed Scilab
Dedicated resources
LSF
Static Policy
Amazon EC2
EC2
Dynamic
Workload Policy
Desktops
Desktops
Timing Policy
12/24
38
Interface ProActive Scilab / Matlab
Primitives de haut niveau Scilab/Matlab pour distribuer les calculs :
f = PAeval(@foo,p1) // evalue f(p1) en asynchrone // f est un futur
L = PAsolve(@foo, {p1,p2, … pk}) // evalue foo(p1), foo(p2) .. foo(pk) en asynchrone // L = f1,…fk est une liste de futurs
39
Synchronization des calculs
Primitives d’attente :
r = PAwaitForAny(L)
// bloque et attend jusqu’a l’obtention du
// premier des fk et retourne le resultat r
lr = PAgetAny(L)
// non-blocant, retourne une liste des
resultats // deja obtenus
43
Plateforme OMD2, vue générale :
Optimizer
(surrogate based, CMA ES ...)
Cluster 1 Cluster 2 Cluster N
Middleware Proactive
f(X1) Output
(Analysis)
adapco UTC INRIA
...
Sirehna +
CD-adapco
f(XN)
f(X2)
ENSM-SE, ECP TAO, UTC, ENS , INRIA
DIG
ITE
O /
SC
ILA
B
User /
Renault
test case 1
local
test cases 2,3,4
webservice
Input
(Xi,obj fct, constr)
end
44
Distribution des calculs
Scheduler
WS
BD
PO
Acquisition des maillages, etc
Stratégie de résolution
Boucle optimisation
Calculs f(x)
Mailleur Solveur
Soumission du PO
Catia
Visualisation stats, etc
N1
N2
Ni1
Nik
Ressources
45
Les cas tests
Conduit de climatisation 3D Conduit de climatisation 2D
Culasse multi-disciplinaire Aéro externe
<1min
CPU
10min
CPU
100h
CPU 1000h
CPU
50
The ProActive PACA Grid Platform (4)
Total:
816 Cores
480 CUDA Cores
14.8TB Storage
Publically Available Today
53
Conclusion
Java Parallel
Toolkit
Multi-Platform
Job Scheduler
Resource
Manager
Portability:
Windows, Linux, Mac
Versatility:
Desktops, Grids, Clouds
Free Professional
Open Source Software
Seamless Scilab Interface (& Matlab)
Multi-Core: Ready for a strong evolution
Cloud: Smooth transition available (Desktop , Server, Cluster)