EasyPDP Introduction and Conclusion Shanjiang Tang 2009-11-13.

EasyPDP Introduction and Conclusion

Shanjiang Tang 2009-11-13

Talk OutlineTalk Outline

Name origin for EasyPDPName origin for EasyPDPBackground & MotivationBackground & MotivationIntroduction to EasyPDPIntroduction to EasyPDPConclusion and Future WorkConclusion and Future Work

Name Origin for EasyPDPName Origin for EasyPDP

Paper’s titlePaper’s title

Easy*: Simplification of Parallel Easy*: Simplification of Parallel Programming Programming

EasyPABEasyPAB vs vs EasyPDPEasyPDP

Background & Background & MotivationMotivation

Background & MotivationBackground & Motivation

MapReduce MapReduce VSVS Phoenix Phoenix (Reported on (Reported on 2009.04.02)2009.04.02)

• Automatic parallelization and distributionAutomatic parallelization and distribution• Fault toleranceFault tolerance• Programming model (API) and runtime systemProgramming model (API) and runtime system

Try to make some

Try to make some

contributions on it

contributions on it !!


The FJRR Design PatternThe FJRR Design PatternRR(Round Robin) VS BC(Block-Cycle)RR(Round Robin) VS BC(Block-Cycle)

Fork

Partition Portions

Data Entity

Join


Parallel Design Pattern (Reference Parallel Design Pattern (Reference papers for FJRR paper writing)papers for FJRR paper writing)

• Parallel Design Pattern for Computational Biology and Parallel Design Pattern for Computational Biology and Scientific Computing ApplicationsScientific Computing Applications

• From Design Patterns to Parallel Architectural SkeletonsFrom Design Patterns to Parallel Architectural Skeletons• Design Patterns for Parallel Computing Using a Network of Design Patterns for Parallel Computing Using a Network of

ProcessorsProcessors• ……

Serial-Like Parallel Programming Serial-Like Parallel Programming (idea from FJRR paper)(idea from FJRR paper)

• Purpose:Purpose: makes the parallel program serial-like from user’s makes the parallel program serial-like from user’s perspective. That is, user could write parallel programs as if perspective. That is, user could write parallel programs as if he or she wrote serial ones.he or she wrote serial ones.

• Implementation:Implementation: hides the parallel parts from the user, hides the parallel parts from the user, thus the user needn’t care much about the parallel parts, all thus the user needn’t care much about the parallel parts, all of work are done within the code-skeletonof work are done within the code-skeleton


Parallelism for Dynamic Parallelism for Dynamic Programming (DP)Programming (DP)

• Learn it from book : Introduction to Parallel Computing. Learn it from book : Introduction to Parallel Computing. Addison WesleyAddison Wesley

• Related Papers : Related Papers : > Parallel Design Pattern for Computational Biology and > Parallel Design Pattern for Computational Biology and

ScientificScientific Computing Applications. Weiguo Liu. 2003.Computing Applications. Weiguo Liu. 2003. > A Tunable Coarse-Grained Parallel Algorithm for > A Tunable Coarse-Grained Parallel Algorithm for

Irregular Dynamic Irregular Dynamic Programming Applications. Weiguo Liu. 2004.Programming Applications. Weiguo Liu. 2004. > A Generic Parallel Pattern-Based System for > A Generic Parallel Pattern-Based System for

Bioinformatics. Weiguo Bioinformatics. Weiguo Liu. 2004.Liu. 2004. > Parallel Pattern-Based Systems for Computational > Parallel Pattern-Based Systems for Computational

Biology: A Case Biology: A Case Study. Weiguo Liu. 2006.Study. Weiguo Liu. 2006. > A Parallel BSP Algorithm for Irregular Dynamic > A Parallel BSP Algorithm for Irregular Dynamic

Programming. Programming. Weiguo Liu. 2007.Weiguo Liu. 2007.



• Various classifications exist for DP algorithms.Various classifications exist for DP algorithms.• In Liu’s papers, it focused on irregular DP algorithms where he In Liu’s papers, it focused on irregular DP algorithms where he

classified classified the DP algorithms intothe DP algorithms into regular regular and and irregular irregular DP algorithms. DP algorithms. • Only static data allocation and scheduling methods brought Only static data allocation and scheduling methods brought

out here.out here. > Column-based wavefront method. (parallel book’s > Column-based wavefront method. (parallel book’s

method)method) > Block-Cycle wavefront methods. (Liu’s method)> Block-Cycle wavefront methods. (Liu’s method)• Why not dynamic data allocation and scheduling method??Why not dynamic data allocation and scheduling method?? > With my first insight and theory, the load balance of > With my first insight and theory, the load balance of

dynamic data dynamic data allocation and scheduling methods should be better allocation and scheduling methods should be better

than Block-than Block- Cycle method. Cycle method. > Search for lots of related papers, and no paper > Search for lots of related papers, and no paper

mentioned mentioned dynamic data allocation and scheduling method found dynamic data allocation and scheduling method found

for DP. for DP.



• Plan 1# : Focus on the irregular DP algorithms Liu Plan 1# : Focus on the irregular DP algorithms Liu mentioned, try to implement and experiment them with the mentioned, try to implement and experiment them with the dynamic methods. dynamic methods.

• Plan 2#: Read and research the source code and paper of Plan 2#: Read and research the source code and paper of phoenix carefully, making phoenix as a reference and phoenix carefully, making phoenix as a reference and exploring more features for EasyPDP. exploring more features for EasyPDP.

• Plan 3#: Continue to collect DP related papers, hoping to Plan 3#: Continue to collect DP related papers, hoping to see whether dynamic method had been mentioned anywhere see whether dynamic method had been mentioned anywhere for DP applications.for DP applications.

Introduction to Introduction to EasyPDPEasyPDP

Dynamic Programming Dynamic Programming AlgorithmsAlgorithms

What is Dynamic Programming (DP) ?What is Dynamic Programming (DP) ? DP is a popular algorithm design technique for the solution DP is a popular algorithm design technique for the solution

of many decision and optimization problems by decomposing the of many decision and optimization problems by decomposing the problem at hand into a sequence of interrelated decision or problem at hand into a sequence of interrelated decision or optimization steps that are solved one after the others. optimization steps that are solved one after the others.

If r represents the cost of a solution composed of sub-If r represents the cost of a solution composed of sub-problems x1, x2…xl, then r can be written as:problems x1, x2…xl, then r can be written as:

r = g(f(x1), f(x2),f(x3), … , f(xl)).r = g(f(x1), f(x2),f(x3), … , f(xl)). The equation is said to be DP formulation when the optimal The equation is said to be DP formulation when the optimal

solution to each problem is determined by composing optimal solution to each problem is determined by composing optimal solutions to the sub-problems and selecting the minimum (or solutions to the sub-problems and selecting the minimum (or maximum).maximum).

For example:For example:

))(),(max()( 11 iiiii mMfbMfMf

ijji zjifyjifxjifjif )1,1(,)1,(,),1(min),(


The classification for DP algorithms:The classification for DP algorithms: The DP algorithms can be classified in terms of the The DP algorithms can be classified in terms of the matrix size and the dependency relationship of each cell matrix size and the dependency relationship of each cell on the matrix: A DP algorithm is called a on the matrix: A DP algorithm is called a tD/eDtD/eD algorithm algorithm if its matrix size is if its matrix size is tt and each matrix cell depends on and each matrix cell depends on OO((nn^̂ee) other cells. it takes time ) other cells. it takes time O(n ^(t+e))O(n ^(t+e)) provided that provided that the computation of each term takes constant time.the computation of each term takes constant time.


Examples:Examples: Algorithm 1. (2D/0D): Given F[i,0] and F[0,j] for 0 <= I,j <=n,Algorithm 1. (2D/0D): Given F[i,0] and F[0,j] for 0 <= I,j <=n,

where xi, yj and zij are computed in constant time.where xi, yj and zij are computed in constant time.

Algorithm 2. (2D/1D): Given c(i,j) for 1<=i<j<=n, F[I,j] = 0 Algorithm 2. (2D/1D): Given c(i,j) for 1<=i<j<=n, F[I,j] = 0 for 1<=i <=n,for 1<=i <=n,

where 1<k<=j and c(i,j) is computed in constant time.where 1<k<=j and c(i,j) is computed in constant time.

Algorithm 3. (2D/2D): Given c(i,j) for 1<=i<j<2n, F[i,0] and Algorithm 3. (2D/2D): Given c(i,j) for 1<=i<j<2n, F[i,0] and F[0,j] for F[0,j] for

0 <= i,j <=n,0 <= i,j <=n,

where 0<=i’<i, 0<=j’<j, and c(i,j) is computed in where 0<=i’<i, 0<=j’<j, and c(i,j) is computed in constant time.constant time.

ijji zjiFyjiFxjiFjiF ]1,1[,]1,[,],1[min],[

],[]1,[min),(],[ jkFkiFjicjiF

),(],[min],[ '''' jijicjiFjiF


DP is widely applied in DP is widely applied in Computational Biology.

DAG Data Driven ModelDAG Data Driven Model

Figure. The DAG Data Driven Model Diagram Figure. The DAG Data Driven Model Diagram


The User Application ModuleThe User Application Module• It is a trigger module which presents the It is a trigger module which presents the

basic steps that should be concerned andbasic steps that should be concerned and done by the programmer in terms of the done by the programmer in terms of the specific applications.specific applications.

• In User Application Model, the programm-In User Application Model, the programmer just needs to do some simple initializa-er just needs to do some simple initialization work and focuses all of his or her att-tion work and focuses all of his or her att- ention on algorithm or functionality of theention on algorithm or functionality of the application, not on parallelization.application, not on parallelization.


The DAG Pattern ModuleThe DAG Pattern Module• A DAG pattern is a regular DAG that defin-A DAG pattern is a regular DAG that defin- es the basic dependence relationships a-es the basic dependence relationships a- mong data nodes, while its size (width,he-mong data nodes, while its size (width,he- ight,etc) is non-determined, set by argum-ight,etc) is non-determined, set by arguments.ents.

• Each DAG node represents a block of data, Each DAG node represents a block of data, we can get the nodes dependency relatio-we can get the nodes dependency relationships from the DAG pattern, while the nships from the DAG pattern, while the workload for each DAG node couldn’t be workload for each DAG node couldn’t be obtained, which is excluded from the DAG obtained, which is excluded from the DAG pattern.pattern.


Five system provided DAG patterns for Five system provided DAG patterns for DP algorithms from EasyPDP DAG DP algorithms from EasyPDP DAG pattern library.pattern library.


The DAG Runtime System ModuleThe DAG Runtime System Module• responsible for DAG operations,responsible for DAG operations, parallelization and concurrency control. parallelization and concurrency control.

• Take master-slave model Take master-slave model

• For slave part, it takes workers pool. For slave part, it takes workers pool. Static Workers PoolStatic Workers Pool Dynamic Workers PoolDynamic Workers Pool

• Support fault toleranceSupport fault tolerance

EasyPDP SystemEasyPDP System

A shared-memory system A shared-memory system implementation of DAG Data Driven implementation of DAG Data Driven Model for DP applicationsModel for DP applications

Adopt the dynamic workers poolAdopt the dynamic workers pool

Current version works with C/C++ Current version works with C/C++ and use P-threadsand use P-threads

Primary Functions for Primary Functions for EasyPDPEasyPDP

Functions can be divided into four Functions can be divided into four categories:categories:

The user programming APIThe user programming API Visible to the programmerVisible to the programmer Two types of functions: system-provided VS user-Two types of functions: system-provided VS user-

defineddefined

DAG operation related functionsDAG operation related functions Only Used by master and invisible to the programmerOnly Used by master and invisible to the programmer DAG initialization, DAG mapping, DAG parsing and DAG initialization, DAG mapping, DAG parsing and

updatingupdating Workers pool related functionsWorkers pool related functions

Workers pool initialization and destroy, pool queue Workers pool initialization and destroy, pool queue task addingtask adding

Thread routine function to implement dynamic workers Thread routine function to implement dynamic workers pool. pool.

Fault tolerance related functionsFault tolerance related functions timeoutQueue related function: tasking adding and timeoutQueue related function: tasking adding and

removing, timeout checking functions. removing, timeout checking functions.

Primary Functions for Primary Functions for EasyPDPEasyPDP

The EasyPDP Runtime The EasyPDP Runtime

A Implementation of DAG runtime A Implementation of DAG runtime system module of DAG Data Driven system module of DAG Data Driven ModelModel

Adopt Dynamic Workers pool Adopt Dynamic Workers pool

Support for fault toleranceSupport for fault tolerance

Details of parallel execution are Details of parallel execution are hidden from the programmerhidden from the programmer


Basic data flow for EasyPDP runtimeBasic data flow for EasyPDP runtime


The user controlling arguments for The user controlling arguments for EasyPDP runtime EasyPDP runtime


The EasyPDP Fault tolerance The EasyPDP Fault tolerance mechanism mechanism Fault tolerance is critical and Fault tolerance is critical and necessary for DP applicationsnecessary for DP applications

detect faults through timeoutdetect faults through timeout

At present, only slave workers At present, only slave workers failure is considered, while notfailure is considered, while not for master scheduler. for master scheduler.


The EasyPDP Fault tolerance The EasyPDP Fault tolerance mechanism mechanism

1. The master distributes the 1. The master distributes the computable DAG nodes dis-computable DAG nodes dis-covered by parsing the DAG covered by parsing the DAG to both the to both the timeoutQueuetimeoutQueue and pool queue simultaneo-and pool queue simultaneo-usly. For theusly. For the timeoutQueue timeoutQueue, , it has a time start field that it has a time start field that records the current time for records the current time for each DAG node when it was each DAG node when it was put into the put into the timeoutQueuetimeoutQueue



2. The workers pool gets the 2. The workers pool gets the computable DAG data tasks computable DAG data tasks from the pool buffer and dis-from the pool buffer and dis-tributes them to its idle wor-tributes them to its idle wor-ker threads dynamically.ker threads dynamically.



3. When a idle worker thread 3. When a idle worker thread obtains a DAG node task, itobtains a DAG node task, itwill register its thread id and will register its thread id and DAG node id in the worker_DAG node id in the worker_DAGPatternNode_register DAGPatternNode_register before do DP algorithm before do DP algorithm computation.computation.



4. Once the worker completes 4. Once the worker completes a DAG node task, it will send a DAG node task, it will send the DAG node id to the the DAG node id to the DAG-DAG-PatternNodeFinishedStackPatternNodeFinishedStack for notifying the master.for notifying the master.



5. The master fetches the 5. The master fetches the finished DAG node id from the finished DAG node id from the DAGPatternNodeFinished-DAGPatternNodeFinished-StackStack in a small regular time. in a small regular time. Then it goes to step 7Then it goes to step 7



6. The master checks the6. The master checks thetimeoutQueue to see whether timeoutQueue to see whether there are timeout DAG nodes. there are timeout DAG nodes. It check nodes from right to It check nodes from right to left until a non-timeout node left until a non-timeout node found. If a timeout DAG node found. If a timeout DAG node discovered, the master firstly discovered, the master firstly finds out the corresponding finds out the corresponding timeout thread id through the timeout thread id through the Worker_DAGPatternNode_regi-Worker_DAGPatternNode_regi-ster, then makes the workers ster, then makes the workers pool kills that thread and in-pool kills that thread and in-stead creates a new thread, stead creates a new thread,



(continue 6) and goes to step (continue 6) and goes to step 7 to remove the timeout DAG 7 to remove the timeout DAG node from the timeoutQueue. node from the timeoutQueue. After that, the master goes to After that, the master goes to step 1 to redistribute the step 1 to redistribute the timeout DAG node.timeout DAG node.



7. The master removes the 7. The master removes the DAG node from the timeout-DAG node from the timeout-Queue. Then it goes to step 1.Queue. Then it goes to step 1.


Refinements for EasyPDP Refinements for EasyPDP performanceperformance

BlockSizeBlockSize Too big or small value for blockSize is not proper.Too big or small value for blockSize is not proper. For irregular DP algorithm, uniform blockSize is not good.For irregular DP algorithm, uniform blockSize is not good. BlockSize self-adjustment mechanism is designing on the BlockSize self-adjustment mechanism is designing on the

wayway for irregular DP algorithms. for irregular DP algorithms.

ThreadNumThreadNum set the value as the number of cores by default.set the value as the number of cores by default.

TimeoutTimeout Too big or small value for timeout is not proper. Too big or small value for timeout is not proper. Timeout self-adjustment mechanism is designed Timeout self-adjustment mechanism is designed

Performance Evaluation Performance Evaluation

Experiment environment Experiment environment

• Dell PowerEdge 2950 Dual Quard Core Server with Xeon E5310 Dell PowerEdge 2950 Dual Quard Core Server with Xeon E5310 processorsprocessors

• Linux 2.6.9-5.ELsmpLinux 2.6.9-5.ELsmp

DP algorithms from computational DP algorithms from computational BiologyBiology

• Smith-Waterman algorithm with linear and affine gap penalty Smith-Waterman algorithm with linear and affine gap penalty (SWLAG)(SWLAG)

• Syntenic alignment (SA)Syntenic alignment (SA)

• Smith-Waterman algorithm with general gap penalty (SWGG)Smith-Waterman algorithm with general gap penalty (SWGG)

• Viterbi Algorithm (VA)Viterbi Algorithm (VA)


Dependency to data sizeDependency to data size


Dependency to block sizeDependency to block size


Dependency to number of threadsDependency to number of threads


Comparison to Block-Cycle Comparison to Block-Cycle Wavefront(BCW)Wavefront(BCW)


Fault Tolerance Fault Tolerance

• occur at arbitrary point of programoccur at arbitrary point of program

• transparent to the programmertransparent to the programmer

• suitable timeout value is criticalsuitable timeout value is critical

• timeout = 10 sec, suit for irregular DPtimeout = 10 sec, suit for irregular DP

algorithms, while too big for regular DPalgorithms, while too big for regular DP algorithms. algorithms.

Conclusion and Conclusion and Future WorkFuture Work

Conclusion and Future Work Conclusion and Future Work

Conclusion Conclusion DAG Data DAG Data

Driven ModelDriven Model

EasyPDP EasyPDP SystemSystem

Concrete DP Concrete DP AlgorithmsAlgorithms

A. The process from model to B. EasyPDP A. The process from model to B. EasyPDP designdesign

concrete application code and implement concrete application code and implement processprocess

Conclusion and Future Work Conclusion and Future Work

Future Work Future Work • Implement EasyPDP with MPI for distributed multi-processors Implement EasyPDP with MPI for distributed multi-processors

systemsystem

• Implement EasyPDP in GPU environment and Cell environment.Implement EasyPDP in GPU environment and Cell environment.

• Exploring more common used DAG patterns for DP applicationsExploring more common used DAG patterns for DP applications

• Applying EasyPDP in other applications, such as astronomy Applying EasyPDP in other applications, such as astronomy

• Research certain performance related topics based on Research certain performance related topics based on EasyPDP, such as EasyPDP, such as

BlockSize. BlockSize. • Visualize EasyPDP in our EasyPAB platformVisualize EasyPDP in our EasyPAB platform

EasyPDP Introduction and Conclusion Shanjiang Tang 2009-11-13.

Documents

Transcript of EasyPDP Introduction and Conclusion Shanjiang Tang 2009-11-13.