Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, …...
Transcript of Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, …...
![Page 1: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/1.jpg)
Parallel Metaheuristics: from Design to Implementation
Prof. E-G. Talbi
Parallel Cooperative Optimization Research Group
Laboratoire d’InformatiqueFondamentale de Lille
![Page 2: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/2.jpg)
MotivationsHigh-dimensional and complex optimizationproblems in many areas of industrial concern→ Telecommunication, genomics, transportation, engineering design, …
Metaheuristics reduce the computational complexity of search BUT Parallel issue remains important:
More and more complex metaheuristicsHuge search spaceCPU or memory intensive of the objective function
![Page 3: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/3.jpg)
Motivations
Rapid development of technology
Processors: Multi-coreprocessors, …Networks (LAN & WAN): Myrinet, Infiniband, Optical networks, …Data storage
Increasing ratio performance / cost
![Page 4: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/4.jpg)
Goals
Speedup the search real-time and interactive optimization methods, dynamic optimisation, …
Improve quality of solutions cooperation, better evenon a single processor
Improve robustness (different instances, design space)
Solving large problems instances, accuracy of mathematics models
![Page 5: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/5.jpg)
Taxonomy of metaheuristics
Metaheuristics
Singlesolution Population
HillClimbing
Simulatedannealing
Tabusearch
Evol.algorithms
Scattersearch
Solution-based metaheuristics: Hill Climbing, Simulated Annealing, Tabu Search, VNS, …
Population-based metaheuristics: Evolutionary Algorithms, Scatter Search, Swarm optimization, …
![Page 6: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/6.jpg)
Outline
Parallel Metaheuristics: Design issuesParallel Metaheuristics: Implementation issues
Hardware platforms, Programming modelsPerformance evaluation
Adaptation to Multi-objective OptimizationSoftware frameworksIIlustration : Network design problem
![Page 7: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/7.jpg)
Parallel Models for Metaheuristics
An unified view for single-based metaheuristics and population based metaheuristics
Three major hierarchical models:
Algorithm-Level: Independent/Cooperative self-contained metaheuristics
Iteration-Level: parallelization of a single step of the metaheuristic(based on distribution of the handled solutions)
Solution-Level: parallelization of the processing of a single solution
![Page 8: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/8.jpg)
Parallel Models for Metaheuristics
SPH ××Scalability
Solution-level:Processing of a single solution(Objective / Data partitioning)
Algorithm-level:Independent walks,Multi-start model,
Hybridization/Cooperationof metaheuristics
Iteration-level:Parallel evaluation of
the neighborhood/population
Heuristic Population / Neighborhood Solution
![Page 9: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/9.jpg)
Algorithm-level Parallel ModelIndependent on the problemAlter the behavior of the metaheuristicDesign questions:
When ? Migration decision:Blind: Periodic or Probabilistic« Intelligent »: state dependent
Where ? Exchange topology:complete graph, ring, torus, random, …Which ? Information : elite solutions, search memory, …How ? Integration
Meta.
Meta
Meta.
Cooperation
Meta
Simulated Annealing Genetic ProgrammingEvolution Strategy Tabu searchAnt ColoniesScatter search, …
![Page 10: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/10.jpg)
Ex : Evolutionary Algorithms (Island ModelDistribution of the population in a set of islands in which semi-isolated EAsare executed
Sparse individual exchanges are performed among these islands with the goal of introducing more diversity into the target populations
Improvement of robustness and quality
GA
GA
GAGA
GA
GA GA
GA
![Page 11: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/11.jpg)
Ex : The multi-start model in LS
Independent set of Local SearchesImprove robustness and quality of solutionsMay start from the same or different initial solution, populationDifferent parameters
encoding, operators, memory sizes (tabu list, …), etc.
![Page 12: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/12.jpg)
Flat classification
Homogeneous Heterogenenous
Global
GeneralSpecialist
Partial
Algorithm-Level Parallel Model
E-G. Talbi, «A taxonomy of hybrid metaheuristics », Journal of Heuristics, Vol.8, No2, 2002.
![Page 13: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/13.jpg)
Homogeneous / HeterogeneousHomogeneous hybrids = Same MetaheuristicHeterogeneous = Different Metaheuristics
ExampleEvolutionary
Algorithm SimulatedAnnealing
Tabu Search
Communicationmedium
Several different metaheuristics cooperate andco-evolve some solutions
![Page 14: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/14.jpg)
Global / PartialPartial : Problem is decomposed in sub-problems. Each algorithm is dedicated to solve one sub-problemCombinatorial / Continuous : Decomposition of the problem, decision space, …Problem dependant: Vehicle routing, Scheduling, …
Problem
Sub-problem Sub-problem Sub-problem
Meta Meta Meta EAs, Tabu searchSimulated annealing
Synchronization : Build a global viable solution
![Page 15: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/15.jpg)
Specialist / GeneralSpecialist: Algorithms which solve different problems (ex: Co-evolutionary)
Diversifying Agent Intensifying Agent
Local Search Agent
Explored regions Promising regionsAdaptative Memory
Good solutions
exploredspace
Promisingsolutions
Initial solutions
Ref
erst
o
Ref
erst
oSolutionsfrom
unexplored regions
S. Cahon, N. Melab and E-G. Talbi. COSEARCH: A parallel cooperativ metaheuristic. Journal ofMathematical Modeling and algorithms JMMA, Vol.5(2), 2006.
![Page 16: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/16.jpg)
Iteration-level Parallel ModelIndependent on the problemComplex objective functions (non linear, simulations, …)Don’t alter the behavior of the metaheuristic Speedup the search
Design questions:Single-solution based algorithms: decomposition of the neighborhoodPopulation based algorithms: decomposition of the population
Meta
Solution(s)
Fullfitness
![Page 17: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/17.jpg)
Ex : Single-Solution Based Metaheuristic
Evaluation of the neighborhood: computationally intensive step Decomposition of the neighborhood :
Asynchronous for SA (# behavior)Synchronous for deterministic algorithms
LS
Solution
Partitionof the
neighborhood
![Page 18: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/18.jpg)
Ex : Population Based Metaheuristic
Decomposition of the population:
Individuals (EAs), Ants (AS), Particles (PSO), etc.Sharing Information: Pheromone matrix, etc.Asynchronous for steady state GAs (# behavior)Synchronous for generational GAs
E.A.
Evaluatedoffsprings
Couple ofindividuals
Recombination,Mutation,evaluation
SelectionReplacement
E-G. Talbi, O. Roux, C. Fonlupt, D. Robillard, «Parallel ant colonies for the quadratic assignment problem», Future Generation Computer Systems, 17(4), 2001.
![Page 19: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/19.jpg)
Solution-level Parallel Model
Dependent on the problemDon’t alter the behavior of the metaheuristic Speedup the search (CPU or I/O intensive)SynchronousDesign questions:
Data / Task DecompositionData : database, geographical area, structure, …Task : sub-functions, solvers, …
Solution
Partialfitness
Aggregation ofpartial fitnesses
CPU #1 CPU #3
CPU #4CPU #2
![Page 20: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/20.jpg)
Outline
Parallel Metaheuristics: Design issuesParallel Metaheuristics: Implementation issues
Hardware platforms, Programming modelsPerformance evaluation
Adaptation to Multi-objective OptimizationSoftware frameworksIIlustration : Network design problem
![Page 21: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/21.jpg)
Parallel Metaheuristics: Implementation issues
P PP P P P..
Parallel Programming Environments
Parallel Programming Environments
Parallel Architecture Hardware
Execution Support
ProcessProcessor ThreadPP
Design of Parallel
Metaheuristics
Programming Paradigms
Main criteria : Memory sharing, Homogeneity,Dedicated, Scalability, Volatility
![Page 22: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/22.jpg)
Shared Memory Machine
Easy to program: conventional OS and programming paradigms
Poor Scalability: Increase in processors leads memory contention
Ex. : Dual-core (Intel, AMD), Origin (Silicon Graphics), ....
CPU CPU CPU
Memory
network
interconnection network: bus, crossbar, multistage
crossbar
![Page 23: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/23.jpg)
Distributed Memory Architectures
Good Scalability : Several hundred nodes with a high speed interconnection network/switch
Harder to program
Communication cost
Ex. : Clusters
CPU
CPU
CPUMemory
network
Memory
Memory
Interconnection schema : hypercube, (2D or 3D) torus, fat-tree, multistage crossbars
![Page 24: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/24.jpg)
Clusters & NOWsClusters: A
collections of PCs interconnected through high speed network,
Low costStandard components
NOWs: Network Of Workstations
take advantage of unused computing power
![Page 25: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/25.jpg)
ccNUMA architectures
Mixing SMP and DSMSmall number of processors (up to 16) clustered in SMP nodes (fast connection, crossbar for instance)SMPs are connected through a less costly network with poorer performance
CPUCPU
Memorynetwork
CPU
CPU CPUCPU
Memorynetwork
CPU
CPU CPUCPU
Memorynetwork
CPU
CPU
Interconnection network Periph.
![Page 26: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/26.jpg)
Grid Computing
Billions of idle PCs …Stealing unused CPU cycles of processors (a mean of 47%)
High-Performance Computing GRID
High-ThroughputComputing GRID
• Offer a virtual supercomputer
Inexpensive, potentially very powerful but more difficult to program than traditional parallel computers
N. Melab, S. Cahon, E-G. Talbi, « Grid computing for parallel bioinspired algorithms», Journal of Parallel and Distributed Computing (JDPC), 66(8), 2006.
““Coordinated resource sharing and problem solving in Coordinated resource sharing and problem solving in dynamic, multidynamic, multi--institutional virtual organizationsinstitutional virtual organizations””
![Page 27: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/27.jpg)
GRID PlatformsLille
Orsay NancyRennes
Bordeaux
Toulouse
Lyon
Grenoble
Sophia
HPC Grid: GRID’5000: 9 sites distributed in France and
inter-connected by Renater5000 proc: between 500 and
1000 CPUs on each site
HTC Grid: PlanetLab: 711 nodes on 338 sites over 25
countries
R. Bolze, …, E-G. Talbi, …, «GRID’5000: A large scale and highly reconfigurable Grid», International Journal of High Performance Computing Applications (IJHPCA), Vol.20(4), 2006.
![Page 28: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/28.jpg)
Main criteria for an efficient implementation
Criteria
Architecture
SharedMemory /Distributed
Homogenous/Heterogenous
Dedicated/Non Dedi.
Local network /Large network
Volatility& FaultTolerance
SMP SM Hom Dedi Local No
COW DM Hom Dedi Local No
NOW DM Het Non Local Yes
HP Grid DM Het Dedi Large No
HT Grid DM Het Non Large Yes
![Page 29: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/29.jpg)
Parallel Programming EnvironmentsShared-memory:
Distributed-memory:
Hybrid model
System Category Language bindingPThreads Operating system CJava threads Programming language Java
OpenMP Compiler directives Fortran, C, C++
Message passing
RPC or Object-based systems
Gridcomputing
Sockets Java RMI Globus
PVM CORBA Condor
MPI
![Page 30: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/30.jpg)
Algorithm-Level Parallel Model
Granularity = Execution cost betweenexchanges / Information exchanged
Large Granularity Well suited to large scalearchitecturesFrequency of migration, Size of information exchanged
Asynchronous exchange more efficient thanSynchronous exchange for Heterogeneous or Non-dedicated Architectures Fault tolerance for Volatile architectures Checkpointing (reduced cost)
![Page 31: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/31.jpg)
Iteration-Level Parallel Model
Granularity = Evaluation of a partition / Partition communication cost
Adapt granularity to target architecture (size of partitions)
Asynchronous evaluation is more efficient:Heterogeneous or Non Dedicated or Volatile platformsHeterogeneous computation of the objective function(#solutions #costs)
Scale: Limited by Size of populationSize of Neighborhood.
![Page 32: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/32.jpg)
Solution-Level Parallel Model
Granularity = Evaluation of sub-function / Cost of solution communication Not well suited to Large scale and distributedmemory architectures
Limited scalability (number of sub-functions or data partitions)
Synchronous
![Page 33: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/33.jpg)
Fault-tolerance issues
Important in volatile, non-dedicated, large-scaleplatformsCheckpointing – Recovery mechanism
Application-oriented and NOT System-oriented(reduced complexity in time and space)Algorithmic-level: Current solution or population, iteration number, …Iteration-level: Which partition of population or neighborhood + fitnessesSolution-level: Solution + partial fitnesses
E-G. Talbi, Z. Hafidi, D. Kebbal, J-M. Geib, "A fault-tolerant parallel heuristic for assignment problems", Future Generation Computer Systems, 14(5), 1998.
![Page 34: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/34.jpg)
Load Balancing, Security, …
Static load balancing important for heterogeneous architecturesDynamic load balancing important for non dedicated, volatile architectures.Security issues important for large-scalearchitectures (multi-domain administration, firewalls, …) and some applications (medical and bio research, industrial, …)
![Page 35: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/35.jpg)
Performance evaluationSpeedup ?
SN = T1 / TNTi : Execution time using i processors (wall clock time)N : number of used processors
If S > N Sublinear speedupIf S = N Linear speedupIf S > N Superlinear speedup
Absolute speedupStrong: T1 = Best known sequential algorithm ??Weak : - Population of N individuals compared to K islands of
N/K individuals- Single-solution metaheuristic with N iterations WITH K SSMs with N/K iterations
E-G. Talbi, P.Bessière, "Superlinear speedup of a parallel genetic algorithm on the SuperNode", SIAM News, Vol.24, No.4, pp.12-27, Juil 1991.
![Page 36: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/36.jpg)
Performance evaluationRelative speedup
Fixed number of generationsInteresting for evaluating the efficiency of the implementationSuperlinear speedup possible: architecture source (memory, cache, …)
Convergence to a solution with a given qualityInteresting for evaluation the efficiency of the parallel design (Algorithm-level parallel model)Superlinear speedup possible: search source (such as in branch and bound)
Heterogenous or Non Dedicated architectures: Efficiency: EN = SN * 100% / N (fraction of time processors are conducting work)100% efficiency means linear speedupt(j): time for task i, U(i): availaibility of worker i
Stochastic : Mean, … ∑∑
∈
∈=Ii
Jj
iU
jtE
)(
)(
![Page 37: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/37.jpg)
Outline
Parallel Metaheuristics: Design issuesParallel Metaheuristics: Implementation issues
Hardware platforms, Programming modelsAdaptation to Multi-objective OptimizationSoftware frameworksIIlustration : Network design problem
![Page 38: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/38.jpg)
Multi-Objective Optimization
(MOP))(),...,(),()(min
21xxxxf fff
n=
s.c.
2≥n
Sx∈( )
Dominancey dominates z if and only if ∀i∈[1, …, n], yi ≤ zi
and ∃i∈[1, …, n], yi < zi
Pareto solutionA solution x is Pareto if a solution which dominates x does not exist
Goal: Find a good quality and welldiversified set of Pareto solutions f 1
f 2
Feasible solutions
Solution foundPareto front
E-G. Talbi, et al., “Parallel multi-objective optimization”. In Multi-criteria Optimization …, Springer, 2007.
![Page 39: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/39.jpg)
Algorithm-Level Parallel Model
General and Global Cooperative Models:Which Information to exchange ? % Pareto archive or current populations (random, elite, uniform, …), …Any replacement strategy for the current population
Partial:Decompose the Pareto front (objective space)# dominance criteria
Specialist :Solve # problems (# objectives subsets)
![Page 40: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/40.jpg)
Iteration-Level Parallel Model
Many MOP applications with complexobjectives (CFD – computational fluid dynamics, CEM – computations electromagnetics, FEM - finiteelement method, …)
Pareto ranking : complex procedure to parallelizePareto archiving : complex procedure to parallelize
![Page 41: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/41.jpg)
Solution-Level Parallel Model
Decomposition of the n objectivesMulti-disciplinary Design Optimization
Many engineering domains with differentmodels (# disciplines, #solvers)Ex: Car designOptimize the air flow around a car computational fluid dynamics (CFD) solverOptimize the toughness of materials finiteelement method (FEM) solver
![Page 42: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/42.jpg)
Outline
Parallel Metaheuristics: Design issuesParallel Metaheuristics: Implementation issues
Hardware platforms, Programming modelsAdaptation to Multi-objective OptimizationSoftware Frameworks for Parallel MetaheuristicsIIlustration : Network design problem
![Page 43: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/43.jpg)
Why ?From scratch: high development cost, error prone, difficult to maintain, …Code reuse: difficult to reuse, adaptation cost, …Design and code reuse – software components: Hollywood principle « Don’t call us, we call you »
Combinatorial Optimization Problems (COPs) in practice: DiversityContinual evolution of the modeling (regards needs, objectives, constraints, …)Need to experiment many solving methods, techniques of parallelization, hybridization, parameters, …
![Page 44: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/44.jpg)
Design ObjectivesMaximal Reuse of code and design
Separation between resolution methods and target problemsInvariant part givenProblem specific part specified but to implement
Flexibility et AdaptabilityAdding and updating other optimization methods, search mechanisms, operators, encoding, ...… to solve new problems
UtilityLarge panel of methods, hybrids, parallel strategies, …
PortabilityDeployment on different platforms (Standard library)
Transparent access to performance and robustnessParallel implementation is tranparent to the target hardware platform
Open source, Efficiency, Easy to use, …
![Page 45: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/45.jpg)
Examples
Metaheuristics Parallelism support at design
Parall. and dist. at
implementation
ECJ E.A. Island cooperation Threads / Sockets
D. BEAGLE E.A. Centralized model /Island cooperation.
Sockets
J-DEAL E.A. Centralized model Sockets
DREAM E.A. Island cooperation Sockets / P2P
MALLBA L.S. / E.A. All MPI, Netstream
PARADISEO LS / EA All PVM, MPI, Condor,
PThreads, Globus
N. Melab, E-G. Talbi, S. Cahon, E. Alba, G. Luque, “Parallel metaheuristics: algorithms and frameworks”. In Parallel Combinatorial Optimization, Wiley, 2006.
![Page 46: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/46.jpg)
PARADISEO (PARAllel and DIStributed Evolving Objects)
PARADISEO in some words …http://paradiseo.gforge.inria.fr
An Open Source C++ framework (STL-Template)Paradigm-free, unifying metaheuristicsFlexible regards the tackled problemGeneric and reusable components (operators of variation, selection, replacement, criterion of termination, …)Many services (visualization, management of command line parameters, check-pointing, …)
![Page 47: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/47.jpg)
PARADISEO :
EO
PEO
MO MOEO
Evolving Objects (EO) for the design of population-based metaheuristics,
Moving Objects (MO) for the design of solution-based metaheuristics,
Multi-Objective EO (MOEO) embedding features and techniques related to multi-objective optimization,
PEO for the parallelization and hybridization of metaheuristics
S. Cahon, N. Melab and E-G. Talbi. ParadisEO: A framework for the reusable design of parallel and distributed metaheuristics. Journal of Heuristics, Vol.10(3), 2004.
http://paradiseo.gforge.inria.fr
![Page 48: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/48.jpg)
Architecture (level of execution)Parallel and distributed platforms
Parallelism and distributionCommunication libraries (MPI LAM, PVM)→ Deployment on networks/clusters of stations (COWs, NOWs)
Multi-threading layer (Posix threads)→ multi-core, multi-processors with shared memory (SMPs)
Support of parallel and distributed environments → Clusters of SMPs (CLUMPS)
Transparent to the user
EO
PEO
MO MOEO LAM-MPI PVM PThreads
S. Cahon, N. Melab and E-G. Talbi. Building with ParadisEO reusible parallel and distributed evolutionary Algorithms, Journal of Parallel Computing, , Vol.30(5), 2004.
![Page 49: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/49.jpg)
Architecture (level of execution)Grid computing
Master/Worker
Condor (HTC) Globus (HPC)EO
PEO
MO MOEO
GridificationRe-visit parallel models taken into account the characterisitics of GridsCoupling of ParadisEO with a Grid middleware (Condor-MW and Globus)
Transparent volatility & checkpointingEx : Definition in ParadisEO-CMW of the memory of each metaheuristic and the associated parallel models
N. Melab, S. Cahon and E-G. Talbi. Grid Computing for Parallel Bioinspired Algorithms. Journal of Paralleland Distributed Computing (JPDC), 2006.
![Page 50: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/50.jpg)
Illustration: Core classes of the Hill Climbing
UML notation(Unified Modeling Language)
![Page 51: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/51.jpg)
Illustration: Core classes of the Evolutionary Algorithm
![Page 52: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/52.jpg)
Illustration: The cooperative island model of E.A.s
E.A.
E.A.Migration
/* To enable migrations (i.e. exchanges ofindividuals) */
eoPopChan <Route> pop_chan ; /* Migrations will occur periodically */eoFreqContinue <Route> mig_cont (FREQ_MIG) ;/* Migrations are composed of random individuals */eoRandomSelect <Route> mig_select_one ; /* Selector of NUM_EMIG emigrants*/eoSelectNumber <Route> mig_select (mig_select_one,
NUM_EMIG) ;/* Emigrants replace the worst individuals*/eoPlusReplacement <Route> mig_replace ;/* The ring topology */eoRingTopology topo (naming_chan) ; /* Building a manager of migrations */eoDistAsyncIslandMig <Route> island_mig
(naming_chan,pop_chan,mig_cont,mig_select,mig_replace,pop,pop,topo) ;
![Page 53: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/53.jpg)
Illustration: The parallelization of the evaluation step
E.A.
Solution
Fullfitness
![Page 54: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/54.jpg)
Illustration: The parallelization of the objective function
Solution
Partialfitness
Aggregation ofpartial fitnesses
![Page 55: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/55.jpg)
Outline
Parallel Metaheuristics: Design issuesParallel Metaheuristics: Implementation issues
Hardware platforms, Programming modelsAdaptation to Multi-objective OptimizationSoftware frameworksIIlustration : Network design problem
Combined use of the 3 parallel modelsMulti-objective problemImplemented on COWs, NOWs, HPC Grids and HTC GridsUsing of PARADISEO – EO & MO & MOEO & PEO
![Page 56: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/56.jpg)
Design of radio networks in mobile telecommunication
Financial context(cost of the network)
Number of sitesQuality of Service
Network designPositioning sitesFixing a set of parameters for each antenna
NP-hard problem with:Huge search space
High cost (CPU and memory )objective functions, constraints.
![Page 57: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/57.jpg)
A brief description
Données bornesPs [26, 55] dBmDiagram 3 typesHauteur [30, 50] mAzimut [0, 359] °Inclinaison [-15, 0] °TRX [1, 7]
Working area
Used sites
Useless sites
Handover area
Cover area
Parameters of antennas
A set of base stations that satisfy the following constraints …
CoverHandover
… and optimizes the following criterion
Min. the number of sitesMin. interferencesMax. yield traffic
Propagation model(Free spaceOkumura-Hata)
![Page 58: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/58.jpg)
A multi-layer hierarchical parallel/hybrid metaheuristic
2. Distributionof networks
3. Parallel evaluation(data partitioning)
1. Deployment ofincremental
Local Searches
1. Cooperativeisland EAs
E-G. Talbi, H. Meunier, « Hierarchical parallel approach for GSM mobile network design», Journal of Parallel and Distributed Computing (JDPC), 66(2), 2006
![Page 59: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/59.jpg)
Iteration-level Parallel Model (Homogeneous and Dedicated Cluster)
Synchronous/AsynchronousDeploying irregular tasks→ The computation time is
dependent of the number of activated sites of the network
Limited scalability of the synchronous model (size of the pop., e.g. 100)
A.E.
Network
Fullfitness
![Page 60: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/60.jpg)
Solution-level Parallel ModelSynchronous, fine-grained !More larger instancePoor scalability
SolutionPartialfitness
Agregation
Partitioningof the geographical
data
(f1.1,f2.1, f3.1)(f1.2,f2.2, f3.2)(f1.3,f2.3, f3.3)
Partitioneddata
![Page 61: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/61.jpg)
Parallel evaluation of the population (model 2)
Synchronous vs. Asynchronous
Parallel Evaluation of a solution (model 3)
Influence of the granularity on the efficiency (synchronous)
models 2&3
model 2 alone
Experimentation under PARADISEO(non-dedicated cluster of PCs)
Superlinear speedup
![Page 62: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/62.jpg)
High-Throughput Computing Grid: Campus of Lille (3 # administrative domains)
Platform HTC Grid (Polytech, IUT, LIFL)Prog. Environment Condor
Number of proc. 100 (heterog. and non dedicated)
Cumulative wall clock time 30681 h.Wall clock time Almost 15 daysParallel efficiency 0.98
E-G. Talbi, S. Cahon and N. Melab. Designing cellular networks using a parallel hybrid metaheuristicon the Grid. Computer Communications Journal, 30(2), 2007.
Highway Urban
![Page 63: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/63.jpg)
High-Performance Computing Grid: GRID’5000 under Globus
400 CPUs on 6 sites: Lille, Nice-Sophia Antipolis, Lyon, Nancy, Rennes
GRID’5000: A fully reconfigurable grid! : Linux « images » having Globus and MPICH-G2
already installed.
Parallel efficiency = 0.92Best results obtainedMore than 22 years of
cumulative wall clock time (other benchmark on 2465 processors)
![Page 64: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/64.jpg)
ConclusionsUnifying Parallel Modelsfor Metaheuristics
Clear separation betweenparallel design and parallel implementation
Encourage the use of software framework for [parallel] metaheuristics
[Parallel] Metaheuristics
Different Architectures:Sequential, Cluster, NOW,
Grid, SMP, …
Software framework for[Parallel] metaheuristics
![Page 65: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/65.jpg)
Perspectives
Parallel models combining Metaheuristics & Exact methods (Algorithms, Coupling of Software, …)Parallel models for dynamic and robustoptimization problemsParallel models for optimization problems withuncertainty
Need a multiple evaluation of a solutionSolving challenging problems on Grids (Ex: Molecular biology, Engineering design, …)
![Page 66: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/66.jpg)
Perpectives: Towards an international Grid
DAS
Grid’5000NAREGI
NAtional REsearch Grid Intiative (NAREGI) – Japan (11 sites)Distributed ASCII Supercomputer 2 (DAS2) – Netherlands (5 sites)
March 23th, 2006
![Page 67: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/67.jpg)
• E-G. Talbi, « Parallel Combinatorial Optimization », Wiley, USA, 2006
• E-G. Talbi, A. Zomaya, « Grid computing for Bioinformatics and Computational Biology », Wiley, USA, 2007
![Page 68: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/68.jpg)
Iteration-Level Parallel Model
Large-Scale to improve robustness : multiple evaluation
Fitn
ess
xx+dxx−dx
xx−dx x+dx
Design
![Page 69: Parallel Metaheuristics - LIFLtalbi/metaheuristic/Chap-6.pdf · Load Balancing, Security, … Static load balancing important for heterogeneous architectures Dynamic load balancing](https://reader034.fdocuments.in/reader034/viewer/2022050106/5f44cc5a018cce784b146da6/html5/thumbnails/69.jpg)
Illustration with a non-dedicated NOWs
Condor : • Management and exploitation of distributed computational resources that are dynamic in space and time• Main features: Check-pointing, Security
Number ofused processors
(full day,2 # administrative
domains )