Comparative Study of One-Sided Factorizations with Multiple

Comparative Study of One-Sided Factorizations withMultiple Software Packages on Multi-Core Hardware ∗

Emmanuel AgulloUniversity of Tennessee

1122 Volunteer BlvdKnoxville, TN

[email protected]

Bilel HadriUniversity of Tennessee


[email protected]

Hatem LtaiefUniversity of Tennessee


[email protected]

Jack Dongarrra† ‡

University of Tennessee, 1122 Volunteer Blvd, Knoxville, [email protected]

ABSTRACTThe emergence and continuing use of multi-core architec-tures require changes in the existing software and some-times even a redesign of the established algorithms in orderto take advantage of now prevailing parallelism. The Par-allel Linear Algebra for Scalable Multi-core Architectures(PLASMA) is a project that aims to achieve both high per-formance and portability across a wide range of multi-corearchitectures. We present in this paper a comparative studyof PLASMA’s performance against established linear alge-bra packages (LAPACK and ScaLAPACK), against new ap-proaches at parallel execution (Task Based Linear AlgebraSubroutines – TBLAS), and against equivalent commercialsoftware offerings (MKL, ESSL and PESSL). Our experi-ments were conducted on one-sided linear algebra factoriza-tions (LU, QR and Cholesky) and used multi-core architec-tures (based on Intel Xeon EMT64 and IBM Power6). Aperformance improvement of 67% was for instance obtainedon the Cholesky factorization of a matrix of order 4000, us-ing 32 cores.

1. INTRODUCTION AND MOTIVATIONSThe emergence of multi-core microprocessor designs marked

the beginning of a forced march toward an era of comput-ing in which research applications must be able to exploitparallelism at a continuing pace and unprecedented scale [1].This confronts the scientific software community with both a

∗Research reported here was partially supported by the Na-tional Science Foundation and Microsoft Research†Computer Science and Mathematics Division, Oak RidgeNational Laboratory, USA‡School of Mathematics & School of Computer Science, Uni-versity of Manchester, United Kingdom

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.SuperComputing ’09 Portland, Orgeon USACopyright 200X ACM X-XXXXX-XX-X/XX/XX ...$10.00.

daunting challenge and a unique opportunity. The challengearises from the disturbing mismatch between the design ofsystems based on this new chip architecture – hundreds ofthousands of nodes, a million or more cores, reduced band-width and memory available to cores – and the componentsof the traditional software stack, such as numerical libraries,on which scientific applications have relied for their accu-racy and performance. So long as library developers coulddepend on ever increasing clock speeds and instruction levelparallelism, they could also settle for incremental improve-ments in the scalability of their algorithms. But to deliveron the promise of tomorrow’s petascale systems, library de-signers must find methods and algorithms that can effec-tively exploit levels of parallelism that are orders of magni-tude greater than most of today’s systems offer. This is anunsolved problem.

To answer this challenge, we have developed a projectcalled Parallel Linear Algebra Software for Multi-core Ar-chitectures (PLASMA) [2, 3]. PLASMA is a redesign ofLAPACK [4] and ScaLAPACK [5] for shared memory com-puters based on multi-core processor architectures. It ad-dresses the critical and highly disruptive situation that isfacing the linear algebra and high performance computing(HPC) community due to the introduction of multi-core ar-chitectures. To achieve high performance on this type ofarchitecture, PLASMA relies on tile algorithms, which pro-vide fine granularity parallelism. The standard linear alge-bra algorithms can then be represented as a Directed AcyclicGraph (DAG) [6] where nodes represent tasks, either panelfactorization or update of a block-column, and edges repre-sent dependencies among them. Moreover, the developmentof programming models that enforce asynchronous, out oforder scheduling of operations is the concept used as the ba-sis for the definition of a scalable yet highly efficient softwareframework for computational linear algebra applications. InPLASMA, parallelism is no longer hidden inside Basic Lin-ear Algebra Subprograms (BLAS) [7] but is brought to thefore to yield much better performance. Each of the one-sidedtile factorizations presents unique challenges to parallel pro-gramming. Cholesky factorization is represented by a DAGwith relatively little work required on the critical path. LUand QR factorizations have exactly the same dependencypattern between the nodes of the DAG. These two factor-izations exhibit much more severe scheduling and numerical

(only for LU) constraints than the Cholesky factorization.PLASMA is currently scheduled statically with a trade offbetween load balancing and data reuse.

As multicore systems require finer granularity and higherasynchronicity, considerable advantages may be obtained byreformulating old algorithms or developing new algorithmsin a way that their implementation can be easily mapped onthese new architectures. For instance, block operations inthe standard LAPACK algorithms for some common factor-izations need to be broken into sequences of smaller tasksor tiles in order to achieve finer granularity and higher flex-ibility in the scheduling of tasks to cores. Besides providingfine granularity, the use of tile algorithms also makes it pos-sible to use more efficient storage format for the data such asBlock Data Layout (BDL). Matrix elements are now storedcontiguously in memory within one tile which engender asmall stride access of memory and therefore, a low cachemiss rate.

The development of these new algorithms for multi-coresystems on one-sided factorizations motivated us to showimprovements made by the tile algorithms in performing acomparative study of PLASMA against established linearalgebra packages (LAPACK and ScaLAPACK) as well asequivalent commercial software offerings (MKL, ESSL andPESSL). Moreover, we also compared a new approach atparallel execution called TBLAS [8] (Task Based Linear Al-gebra Subroutines) in the Cholesky and QR cases – theLU factorization not being implemented yet. The experi-ments were conducted on two different multi-core architec-tures based on Intel Xeon EMT64 and IBM Power6, respec-tively.

The paper is organized as follows. Section 2 provides anoverview of the libraries and the hardware used in the ex-periments, as well as a guideline to interpret the results.Section 3 describes the tuning process to achieve the bestperformance of each software. In section 4 we present acomparative performance evaluation of the different librariesbefore concluding and presenting future work directions inSection 5.

2. EXPERIMENTAL ENVIRONMENTWe provide below an overview of the software used for

the comparison against PLASMA as well as a description ofthe hardware platforms, i.e., Intel Xeon EMT64 and IBMPower6 machines. We also introduce some performance met-rics that will be useful to the interpretation of the experi-mental results.

2.1 Libraries

2.1.1 LAPACK, MKL and ESSLLAPACK 3.2 [4], an acronym for Linear Algebra PACK-

age, is a library of Fortran subroutines for solving the mostcommonly occurring problems in numerical linear algebrafor high-performance computers. It has been designed toachieve high efficiency on a wide range of modern high-performance computers such as vector processors and sharedmemory multiprocessors. LAPACK supersedes LINPACK [9]and EISPACK [10]. The major improvement was to rely onthe use of a standard set of Basic Linear Algebra Subpro-grams (BLAS) [7], which can be optimized for each comput-ing environment. LAPACK is designed on top of the level1, 2, and 3 BLAS, and nearly all of the parallelism in the

LAPACK routines is contained in the BLAS.MKL 10.1 (Math Kernel Library) [11], a commercial offer-

ing from Intel, is a library of highly optimized, extensivelythreaded math routines that require maximum performancesuch as BLAS. We consider in this paper the 10.1 version –the latest available version when the paper was submitted.

Library) [12] is a collection of subroutines providing a widerange of highly tuned mathematical functions specifically de-signed to improve the performance of scientific applicationson the IBM Power processor-based servers (and other IBMmachines).

2.1.2 ScaLAPACK and PESSLScaLAPACK 1.8.0 (Scalable Linear Algebra PACKage) [5]

is a parallel implementation of LAPACK for parallel ar-chitectures such as massively parallel SIMD machines, ordistributed memory machines. It is based on the MessagePassing Interface (MPI) [13]. As in LAPACK, ScaLAPACKroutines rely on block-partitioned algorithms in order tominimize the frequency of data movement between differ-ent levels of the memory hierarchy. The fundamental build-ing blocks of the ScaLAPACK library are distributed mem-ory versions of the Level 1, Level 2, Level 3 BLAS, calledthe Parallel BLAS or PBLAS [14], and a set of Basic Lin-ear Algebra Communication Subprograms (BLACS) [15] forcommunication tasks that arise frequently in parallel lin-ear algebra computations. In the ScaLAPACK routines, themajority of interprocessor communication occurs within thePBLAS and the interface is as similar to the BLAS as pos-sible. The source code of the top software layer of ScaLA-PACK looks very similar to that of LAPACK.

PESSL 3.3 (Parallel Engineering Scientific Subroutine Li-brary) [12] is a scalable mathematical library that supportsparallel processing applications. It is an IBM commercialoffering for Power processor-based servers (and other IBMmachines).

Although ScaLAPACK and PESSL are primarily designedfor distributed memory architectures, a driver optimized forshared-memory machines allows memory copies instead ofactual communications.

2.1.3 TBLASThe TBLAS [8] project is an on-going effort to create a

scalable, high-performance task-based linear algebra librarythat works in both shared memory and distributed memoryenvironments. TBLAS takes inspiration from SMP Super-scalar (SMPSs) [16], PLASMA and ScaLAPACK. It adoptsa dynamic task scheduling approach to execute dense linearalgebra algorithms on multi-core systems. A task-based li-brary replaces the existing linear algebra subroutines suchas PBLAS. TBLAS transparently provides the same inter-face and functionality as ScaLAPACK. Its runtime systemextracts a DAG of tasks from the sequence of function callsof a task-based linear algebra program. Data dependenciesbetween tasks are identified dynamically. Their execution isdriven by data availability. To handle large DAGs, a fixed-size task window is enforced to unroll a portion of the activeDAG on demand. Hints regarding critical paths (e.g., paneltasks for factorizations) may also be provided to increase thepriority of the corresponding tasks. Although the TBLASruntime system transparently handles any data movementrequired for distributed memory executions, we will not usethis feature since we focus on shared-memory multi-core ar-

chitectures. So far the TBLAS project is in an experimentalstage and the LU factorization (DGETRF routine) is notcompleted yet.

PLASMA, TBLAS, LAPACK and ScaLAPACK are alllinked with the optimized vendor BLAS available on thesystem, which are MKL 10.1 and ESSL 4.3 on the Intel64and Power6 architectures, respectively.

2.2 Hardware

2.2.1 Intel64-based 16 cores machineOur first architecture is a quad-socket quad-core machine

based on an Intel Xeon EMT64 E7340 processor operatingat 2.39 GHz. The theoretical peak is equal to 9.6 Gflop/s/per core or 153.2 Gflop/s for the whole node, composed of 16cores. There are two levels of cache. The level-1 cache, localto the core, is divided into 32 kB of instruction cache and 32kB of data cache. Each quad-core processor being actuallycomposed of two dual-core Core2 architectures, the level-2cache has 2×4 MB per socket (each dual-core shares 4 MB).The effective bus speed is 1066 MHz per socket leading to abandwidth of 8.5 GB/s (per socket). The machine is runningLinux 2.6.25 and provides Intel Compilers 11.0 together withthe MKL 10.1 vendor library.

2.2.2 Power6-based 32 cores machineThe second architecture is a SMP node composed of 16

dual-core Power6 processors. Each dual-core Power6 pro-cessor runs at 4.7 GHz, leading to a theoretical peak of 18.8Gflop/s per core and 601.6 Gflop/s per node. There arethree levels of cache. The level-1 cache, local to the core, cancontain 64 kB of data and 64 kB of instructions; the level-2cache is composed of 4 MB per core, accessible by the othercore; and the level-3 cache is composed of 32 MB commonto both cores of a processor with one controller per core (80GB/s). An amount of 256 GB of memory is available andthe memory bus (75 GB/s) is shared by the 32 cores of thenode. The machine runs AIX 5.3 and provides the xlf 12.1and xlc 10.1 compilers together with the ESSL 4.3 vendorlibrary. Actually, the whole machine is the IBM Vargas1 –ranked 50th on the TOP500 of November 2008 [17] – fromIDRIS2 and is composed of 112 SMP nodes for a total 67.38Tflops/s theoretical peak. However, we use only a singlenode at a time since we focus on shared-memory, multi-corearchitectures.

2.3 Performance metrics (How to interpret thegraphs)

In this section, we present some performance referencesand definitions that we will consistently report in the paperto evaluate the efficiency of the libraries presented above.

We report three performance references. The first one isthe theoretical peak of the hardware used. All the graphsin the paper are scaled to that value, which depends on theprocessor used (Intel64 or Power6) and the number of coresinvolved (or the maximum number of cores if results on dif-ferent number of cores are presented in the same graph).For instance, the graphs of Figures 1(a) and 1(b) are scaledto 153.2 Gflop/s and 601.6 Gflop/s since they respectively

1http://www.idris.fr/eng/Resources/index-vargas.html2Institut du Developpement et des Ressources en Informa-tique Scientifique: http://www.idris.fr.

involve a maximum of 16 Intel64 cores and 32 Power6 cores.The second reference is the parallel vendor DGEMM perfor-mance (i.e., the MKL and ESSL ones on Intel64 and Power6,respectively). DGEMM is a commonly used performancereference for parallel linear algebra algorithms; achieving acomparable performance is challenging since it is a parallel-friendly level-3 BLAS operation. Figures 1(a) and 1(b) il-lustrate very good scalability on both platforms.

The last reference we consider is related to the peak per-formance of the serial computational intensive level-3 BLASkernel effectively used by PLASMA. The kernel used de-pends on the factorization; they are respectively dgemm-seq,dssrfb-seq and dssssm-seq for the Cholesky (DPOTRF), QR(DGEQRF) and LU (DGETRF) algorithms. The dssrfb-seqand dssssm-seq routines are new kernels (not part of BLASnor LAPACK standard) that have been specifically designedfor tile algorithms. They involve extra-flops (that can leadto a total 50% overhead if no inner blocking occurs) [2].

Their complete description is out of the scope of this paperand more information can be found in [2]. The performanceof a PLASMA thread (running on a single core) is boundedby the performance – that we note kernelseq – of the cor-responding serial level-3 kernel. Therefore, the performanceof an execution of PLASMA on p cores is bounded by p ×kernelseq . For instance, Figures 2(a), 2(b) and 2(c) respec-tively show the performance upper bound for the DPOTRF,DGEQRF and DGETRF (PLASMA) factorizations; theycorrespond to the horizontal 16×dgemm-seq, 16×dssrfb-seqand 16×dssssm-seq dashed lines.

The speedup of a parallel execution on p cores is definedby the following formula: sp = t1

tp

where p is the number of

cores, t1 the execution time of the sequential algorithm andtp the execution time of the parallel algorithm when p coresare used. A linear speedup is obtained when sp = p. Thiscorresponds to the ideal case where doubling the number ofcores doubles the speed.

Another performance metric commonly used for parallelperformance evaluation is the efficiency. It is usually de-fined as follows: Ep = t1

p×tp. However, this metric does not

provide much information about the relative performanceof a library against another one, which is the focus of thispaper. Therefore we introduce the normalized efficiency as

ep =tkernelseq

p×tp

where tkernelseq is the time that the serial level-

3 kernel (dgemm-seq, dssrfb-seq or dssssm-seq) would spendto perform an equivalent number of effective floating pointoperations. We say it is normalized since the sequentialreference time tkernelseq consistently refers to a common op-eration independently of the considered library (but thatstill depends on the considered factorization). Therefore,that metric directly allows us to compare two libraries toeach other. Furthermore, PLASMA normalized efficiencyis bounded by a value of 1 (ep(PLASMA) ≤ 1) since thelevel-3 serial kernels are the tasks that achieve the high-est performance during the PLASMA factorization process.That normalized efficiency can also be interpreted geomet-rically. It corresponds to the performance ratio of the con-sidered operation to the p × kernelseq reference. For in-stance, in Figure 2(b), the normalized efficiency of PLAS-MA is the ratio between the PLASMA performance graphsto the 16×dgerfb-seq dashed line. The peak performance(obtained when NB = 200, IB = 40 and N = 12000) beingequal to 102.1 Gflop/s and the 16×dgerfb-seq performance

0

20

40

60

80

100

120

140

0 2000 4000 6000 8000 10000 12000

Matrix size

16xdgemm-seq16 cores8 cores4 cores2 cores1 core

(a) Intel64

0

100

200

300

400

500

600

0 2000 4000 6000 8000 10000 12000

Matrix size

32xdgemm-seq

32 cores16 cores

8 cores4 cores2 cores1 core

(b) Power6

Figure 1: DGEMM performance (Gflop/s).

being equal to 129.6 Gflops, the peak normalized efficiencyis thus equal to ep = 0.79.

3. TUNINGMaximizing the performance and minimizing the execu-

tion time of scientific applications is a daily challenge forthe HPC community. The libraries presented in the previ-ous section have tunable execution parameters and a wrongsetting for a single parameter can dramatically slow downthe whole application. This section presents an overview ofthe tuning method applied to each library.

3.1 PLASMAPLASMA performance strongly depends on tunable exe-

cution parameters trading off utilization of different systemresources, the outer and the inner blocking sizes as illus-trated in Figure 2. The outer block size (NB) trades off par-allelization granularity and scheduling flexibility with singlecore utilization, while the inner block size (IB) trades offmemory load with extra-flops due to redundant calculations.Only the QR and LU factorizations use inner blocking. Ifno inner blocking occurs, the resulting extra-flops overheadmay represent 25% and 50% for the QR and LU factoriza-tion, respectively [2]. Tuning PLASMA consists of findingthe (NB,IB) pairs that maximize the performance dependingon the matrix size and on the number of cores. An exhaus-tive search is cumbersome since the search space is huge. Forinstance, in the QR and LU cases, there are 1352 possiblecombinations for (NB,IB) even if we constrain NB to be aneven integer between 40 and 500 and if we constrain IB to di-vide NB. All these combinations cannot be explored for largematrices (N >> 1000) on effective factorizations in a rea-sonable time. Knowing that this process should be repeatedfor each number of cores and each matrix size motivates usto consider a pruned search. The idea is that tuning theserial level-3 kernel (dgemm-seq, dssrfb-seq and dssssm-seq)is not time-consuming since peak performance is reachedon relatively small input matrices (NB < 500) that can beprocessed fast. Therefore, we first tune those serial kernels.As illustrated in Figure 3, not all the (NB,IB) pairs resultin a high performance. For instance, the (480,6) pair leadsto a performance of 6.0 Gflop/s whereas the (480,96) pairachieves 12.2 Gflop/s, for the dssrfb-seq kernel on Power6(Figure 3(e)). We select a limited number of (NB,IB) pairs(pruning step) that achieve a local maximum performanceon the range of NB. We have selected five or six pairs onthe Intel64 machine for each factorization and eight on thePower6 machine (Figure 3). We then benchmark the per-

formance of PLASMA factorizations only with this limitednumber of combinations (as seen in Figure 2). Finally, thebest performance obtained is selected.

The dssssm-seq efficiency depends on the amount of piv-oting performed. The average amount of pivoting effectivelyperformed during a factorization is matrix-dependent. Be-cause the test matrices used for our LU benchmark are ran-domly generated with a uniform distribution, the amountof pivoting is likely to be important. Therefore, we haveselected the (NB,IB) pairs from dssssm-seq executions withfull pivoting (figures 3(c) and 3(f)). The dssssm-seq perfor-mance drop due to pivoting can reach more than 2 Gflop/son Power6 (Figure 3(f)).

We have validated our pruned search methodology for thethree one-sided factorizations on Intel64 16 cores. To doso, we have measured the relative performance overhead(percentage) of the pruned search (PS) over the exhaustivesearch (ES), that is: 100 × (ES

PS− 1). Table 1 shows that

the pruned search performance overhead is bounded by 2%.Because the performance may slightly vary from one runto another on cache-based architectures [18], we could fur-thermore observe in some cases higher performance (up to0.95%) with pruned search (negative overheads in Table 1).However, the (NB,IB) pair that leads to the highest perfor-mance obtained with one method consistently matches thepair leading to the highest performance obtained with theother method. These results validate our approach. There-fore, in the following, all the experimental results presentedfor PLASMA will correspond to experiments conducted witha pruned search methodology.

3.2 TBLASLike PLASMA, TBLAS performance relies on two tunable

parameters, NB and IB. However, preliminary experimentalresults led the authors to systematically set IB equal to 20%of NB (for the QR factorization). In its current developmentstage, TBLAS requires that NB divides the matrix size N;this constraint reduces the search space and makes it pos-sible to perform an exhaustive search. The TBLAS resultsreported in Section 4 will thus correspond to experimentsconducted with an exhaustive search.

3.3 ScaLAPACK and PESSLScaLAPACK and PESSL are based on block-partitioned

algorithms and the matrix is distributed among a processorgrid. The performance thus depends on the the topology ofthe processor grid and on the block size (NB). For instance,when using 32 cores, the grid topology can be 8x4 (8 rows, 4

0

20

40

60

80

100

120

140

0 2000 4000 6000 8000 10000 12000

Matrix size

16xdgemm-seqNB=196NB=168NB=120NB=84NB=60

(a) DPOTRF - Intel64 - 16 cores

0

20

40

60

80

100

120

140

0 2000 4000 6000 8000 10000 12000

Matrix size

16xdssrfb-seqNB=256 IB=64NB=200 IB=40NB=168 IB=56NB=120 IB=40NB=84 IB=28NB=60 IB=20

(b) DGEQRF - Intel64 - 16 cores

0

20

40

60

80

100

120

140

0 2000 4000 6000 8000 10000 12000

Matrix size

16xdssssm-seq

NB=252 IB=28NB=196 IB=28NB=168 IB=28NB=120 IB=24

NB=84 IB=28NB=60 IB=20

(c) DGETRF - Intel64 - 16 cores

0

100

200

300

400

500

600

0 2000 4000 6000 8000 10000 12000

Matrix size

32xdgemm-seq

NB=288NB=224NB=220NB=192NB=160NB=120NB=80NB=60

(d) DPOTRF - Power6 - 32 cores

0

100

200

300

400

500

600

0 2000 4000 6000 8000 10000 12000

Matrix size

32xdssrfb-seq

NB=480 IB=96NB=340 IB=68NB=300 IB=60NB=256 IB=64NB=168 IB=56NB=160 IB=40NB=120 IB=40NB=80 IB=40

(e) DGEQRF - Power6 - 32 cores

0

100

200

300

400

500

600

0 2000 4000 6000 8000 10000 12000

Matrix size

32xdssssm-seq

NB=480 IB=60NB=340 IB=68NB=280 IB=56NB=240 IB=60NB=196 IB=28NB=168 IB=28NB=120 IB=40

NB=80 IB=20

(f) DGETRF - Power6 - 32 coresFigure 2: Effect of (NB,IB) on PLASMA performance (Gflop/s).

0 50 100 150 200 2500

1

2

3

4

5

6

7

8

9

NB

dgemm performance for NB peak performanceselected NB

6084

120168

196

(a) dgemm-seq - Intel64

0 50 100 150 200 2500

1

2

3

4

5

6

7

8

9

NB

dsssrf performance for (NB,IB) peak performanceselected (NB,IB) pairs

(256,64)

(60,20)

(84,28)(120,40)

(168,56) (200,40)

(b) dssrfb-seq - Intel64

0 50 100 150 200 2500

1

2

3

4

5

6

7

8

9

NB

peak performance without pivoting peak performance with full pivotingselected (NB,IB) pairs performance for (NB,IB) with full pivoting

(252,28)

(120,24)(168,28)

(196,28)

(60,20)(84,28)

(c) dssssm-seq - Intel64

0 50 100 150 200 250 300 350 400 450 5000

2

4

6

8

10

12

14

16

18

NB

dgemm performance for NB peak performanceselected NB

288192160120

80

60

220 224

(d) dgemm-seq - Power6

0 50 100 150 200 250 300 350 400 450 5000

2

4

6

8

10

12

14

16

18

NB

dssrf performance for (NB,IB) peak performanceselected (NB,IB) pairs

(480,96)

(480,6)

(340,68)(300,60)(256,64)

(120,40)

(80,40)

(160,40) (168,56)

(e) dssrfb-seq - Power6

0 50 100 150 200 250 300 350 400 450 5000

2

4

6

8

10

12

14

16

18

NB

peak performance without pivoting peak performance with full pivotingselected (NB,IB) pairs performance for (NB,IB) with full pivoting

(340,68)(80,56)(240,60)(196,28)

(80,20)

(120,40)

(168,28) (480,60)

(f) dssssm-seq - Power6

Figure 3: Effect of (NB,IB) on the performance of the serial PLASMA computational intensive level-3 BLASkernels (Gflop/s).

Table 1: Overhead (in %) of Pruned search (Gflop/s) over Exhaustive search (Gflop/s) on Intel64 16 coresDPOTRF DGEQRF DGETRF

Matrix Pruned Exhaustive Over- Pruned Exhaustive Over- Pruned Exhaustive Over-Size Search Search head Search Search head Search Search head1000 53.44 52.93 -0.95 46.35 46.91 1.20 36.85 36.54 -0.842000 79.71 81.08 1.72 74.45 74.95 0.67 61.57 62.17 0.974000 101.34 101.09 -0.25 93.72 93.82 0.11 81.17 80.91 -0.326000 108.78 109.21 0.39 100.42 100.79 0.37 86.95 88.23 1.478000 112.62 112.58 -0.03 102.81 102.95 0.14 89.43 89.47 0.04

columns), 4x8, 2x16, 16x2, 1x32 or 32x1. Four possible blocksizes are commonly used: 32, 64, 128 and 256. Figures 4show the effect of the block size NB on the performanceof ScaLAPACK and PESSL (we report the maximum per-formance obtained for the different possible processor gridconfigurations).

Because there is a limited number of possible combina-tions, in the following we will report performance resultsobtained with an exhaustive search: for each matrix sizeand number of cores, all the possible processor grid distri-butions are tried in combination with the four block sizes(NB ∈ {32, 64, 128, 256}). And the highest performance isfinally reported.

3.4 LAPACK,MKL andESSLIn LAPACK, the blocking factor (NB) is hard coded in

the auxiliary routine ILAENV [19]. This parameter has beenset to 64 for both LU and Cholesky factorizations and to 32for QR factorization3 .

MKL andESSL have been highly optimized by their re-spective vendors.

4. COMPARISON TO OTHER LIBRARIES

4.1 MethodologyFactorizations are performed in double precision. We briefly

recall the tuning techniques used in the experiments dis-cussed below according to Section 3. We employ a prunedsearch for PLASMA; an exhaustive search for TBLAS, ScaLA-PACK and PESSL; and we consider that LAPACK, MKLand ESSL have been tuned by the vendor.

Furthermore, to capture the best possible behavior of eachlibrary, we repeat the number of executions (up to 10 times)and we report the highest performance obtained. We do notflush the caches [20] before timing a factorization4. However,the TLB (Translation Lookaside Buffer) is flushed betweentwo executions: the loop over the different executions is per-formed in a script (rather than within the executable) andcalls several times the same executable.

3These values may not be optimum for all the test cases.Another approach might have consisted in tuning NB foreach number of cores and each matrix size. However, sincewe did not expect this library to be competitive, we did notinvestigate that possibility further.4It is kernel usage, not problem size, that dictates whetherone wish to flush the cache [20]. Warm (or partially warm)cache executions are plausible for dense linear factorizations.For instance, sparse linear solvers, which rely on dense ker-nels, intend to maximize data reuse between successive callsto dense operations.

ScaLAPACK, PESSL and PLASMA interfaces allow theuser to provide data distributed on the cores. In our shared-memory multi-core environment, because we do not flush thecaches, these libraries have thus the advantage to start thefactorization with part of the data distributed on the caches.This is not negligible. For instance, a 8000 × 8000 doubleprecision matrix can be held distributed on the L3 caches ofthe 32 cores of a Power6 node.

4.2 Experiments on few coresWe present in this section results of experiments con-

ducted on a single socket (4 cores on Intel64 and 2 coreson Power6). In these conditions, data reuse between coresis likely to be efficient since the cores share different levelsof cache (see Section 2.2). Therefore, we may expect a highspeedup (see definition in Section 2.3) for the different al-gorithms. Figures 5(a) and 5(d) indeed show that parallelDGEMM – given as a reference – almost reaches a linearspeedup on both platforms (s4 ≈ 4 on Intel64 and s2 ≈ 2 onPower6).

Even LAPACK has a linear speedup. In the DPOTRFcase, most of the computation is spent by the dgemm-seqkernel. Since LAPACK exploits parallelism at the BLASlevel, the parallel DGEMM version is actually executed. Fig-ures 5(a) and 5(d) show that LAPACK DPOTRF efficiencyfollows the one of the parallel DGEMM at a lower rate. OnIntel64, its performance stabilizes at 20.0 Gflop/s, whichcorresponds to a normalized efficiency e4 = 0.56 (accord-ing to the definition provided in Section 2.3). Parallelism isobtained by processing each BLAS task with all the cores.However, since those tasks are processed in sequence, thisapproach is very synchronous and does not scale with thenumber of cores. Indeed, Figure 9(a) shows that LAPACKDPOTRF does not benefit from any significant performanceimprovement anymore when more than 4 cores (i.e. morethan one socket) are (is) involved. On Power6, the LA-PACK DPOTRF performance also stabilizes at 20.0 Gflop/s(on 2 cores this time, see Figure 5(d)) which correspondsto an even higher normalized efficiency e2 = 0.69. Thevery high bandwidth (80 MB/s per core) of the L3 sharedcache strongly limits the overhead due to this synchronousapproach. Furthermore, the high memory bandwidth (75GB/s shared bus) allows a non negligible improvement ofthe LAPACK DPOTRF when going to more than 2 cores –and up to 16 – as illustrated by Figure 9(b). However, theperformance hardly exceeds 100 Gflop/s, which is only onesixth of the theoretical peak of the 32 cores node. Similarobservations can be reported from Figures 5 and 9 for theLAPACK DGEQRF and DGETRF routines (except that anon negligible performance improvement can be obtained up

0

20

40

60

80

100

120

140

0 2000 4000 6000 8000 10000 12000

Matrix size

16xdgemm-seqNB=500NB=400NB=250NB=200NB=125


0

20

40

60

80

100

120

140

0 2000 4000 6000 8000 10000 12000

Matrix size

16xdssrfb-seqNB=500NB=400NB=250NB=200NB=125


0

100

200

300

400

500

600

0 2000 4000 6000 8000 10000 12000

Matrix size

32xdgemm-seq

NB=500NB=400NB=250NB=200NB=125

(c) DPOTRF - Power6 - 32 cores

0

50

100

150

200

250

300

0 2000 4000 6000 8000 10000 12000

Matrix size

16xdgemm-seq

NB=500NB=400NB=250NB=200NB=125


0

50

100

150

200

250

300

0 2000 4000 6000 8000 10000 12000

Matrix size

16xdssrfb-seq

NB=500NB=400NB=250NB=200NB=125


0

100

200

300

400

500

600

0 2000 4000 6000 8000 10000 12000

Matrix size

32xdssrfb-seq

NB=500NB=400NB=250NB=200NB=125

(f) DGEQRF - Power6 - 32 cores

0

20

40

60

80

100

120

140

0 2000 4000 6000 8000 10000 12000

Matrix size

16xdgemm-seqNB=256NB=128NB=64NB=32

(g) DPOTRF - Intel64 - 16 cores

0

20

40

60

80

100

120

140

0 2000 4000 6000 8000 10000 12000

Matrix size

16xdssrfb-seqNB=256NB=128NB=64NB=32

(h) DGEQRF - Intel64 - 16 cores

0

20

40

60

80

100

120

140

0 2000 4000 6000 8000 10000 12000

Matrix size

16xdssssm-seq

NB=256NB=128NB=64NB=32

(i) DGETRF - Intel64 - 16 cores

0

100

200

300

400

500

600

0 2000 4000 6000 8000 10000 12000

Matrix size

32xdgemm-seq

NB=256NB=128NB=64NB=32

(j) DPOTRF - Power6 - 32 cores

0

100

200

300

400

500

600

0 2000 4000 6000 8000 10000 12000

Matrix size

32xdssrfb-seq

NB=256NB=128NB=64NB=32

(k) DGEQRF - Power6 - 32 cores

0

100

200

300

400

500

600

0 2000 4000 6000 8000 10000 12000

Matrix size

32xdssssm-seq

NB=256NB=128NB=64NB=32

(l) DGETRF - Power6 - 32 coresFigure 4: Effect of NB on TBLAS (a,b,c,e,f,g) ScaLAPACK (i,j,k) and PESSL (l,m,n) performance (Gflop/s).

to 8 cores in the LAPACK DGETRF case on Intel – see Fig-ure 9(a)). In a nutshell, parallelism at the BLAS level canachieve a decent performance when the cores involved arelimited to a single socket.

Figures 5(c) and 5(f) show that the vendor libraries (MKLand ESSL) outperform the other libraries – including PLAS-MA– when the LU factorization (DGETRF routine) is exe-cuted on a single socket. They even outperform the upperbound of the tile algorithms. Indeed, geometrically, theirgraphs are above the 4×dssssm-seq (resp. 2×dssssm-seq)dashed lines and, numerically, their peak normalized effi-ciency e4 = 1.13 (resp. e2 = 1.29) is higher than 1. Thisresult shows that when the hardware allows a very efficientdata reuse across the cores – thanks to the high bandwidthof shared level of caches – the lower performance of the ker-nels used in the tile algorithms cannot be balanced by thebetter scheduling opportunities that these algorithms pro-vide. The lower performance of tile algorithms kernels isboth due to a current non optimized implementation (thatwe plan to improve) and to the extra-flops performed (thatmay be decreased at the price of a lower data reuse). Thedssssm-seq kernel is the one involving the largest amount ofextra-flops for going to tile algorithms (that can lead to atotal 50% overhead if no inner blocking occurs). When theextra-flops count is lower, tile algorithms remain competi-tive. Figures 5(b) and 5(e) illustrate this phenomenon forthe QR factorization (DGEQRF routine) whose extra-flopscount is bounded by 25% (and less if inner blocking occurs).In this case, tile algorithms (PLASMA and TBLAS) can al-most compete against vendor libraries; their better schedul-ing opportunities enable balancing the extra-flops count (aswell as the current non optimized implementation of theirserial kernels) although the hardware efficiency for paralleldata reuse limits the impact of a non optimum scheduling.The Cholesky factorization (DPOTRF routine) does not in-volve extra-flops for going to tile algorithms and directlyrelies on a kernel optimized by the vendor (dgemm-seq).As illustrated in figures 5(a) and 5(d), in this case, tile al-gorithms (and in particular PLASMA) outperform the al-gorithms implemented in the other libraries, including thevendor libraries.

In a nutshell, the overhead of imperfect scheduling strate-gies is limited when the algorithms are executed on few coresthat share levels of cache they can access with a very highbandwidth. In this case, tile algorithms have a higher perfor-mance than established linear algebra packages dependingon whether the extra-flops count for going to tile and theoverhead of a non optimized serial kernel is not too largecompared to the limited practical improvement gained bytheir scheduling opportunities. The advantage of tile al-gorithms relies on their very good scalability achieved bytheir scheduling opportunities; Figure 7 shows their excellentpractical scalability on both machines in the PLASMA casefor instance. In the next section, we discuss their relativeperformance against other libraries when a larger number ofcores are used.

4.3 Large number of coresWe present in this section results of experiments con-

ducted on a larger number of cores (16 cores on Intel64;16 and 32 cores on Power6). In these conditions, the costof data reuse between cores is higher when the cores do notshare a common level of cache. For instance, on the Power6

architecture, if a core accesses data that have been processedby another processor, it will access that data through thememory bus, which is shared by all the 32 cores. Even if thatbus has a high bandwidth (75 GB/s), the actual through-put is limited when many cores simultaneously access thememory. Therefore, we expect a higher sensitivity of thealgorithms to the scheduling strategy than we indicated inSection 4.2.

Figures 6(a), 6(d) and 6(g) present the Cholesky (DPO-TRF routine) performance of the different libraries. PLAS-MA consistently outperforms the other libraries, followed byTBLAS. These results illustrate the performance improve-ment brought by tile algorithms. For instance, a speedupof 67% was obtained for a Cholesky factorization on a ma-trix of order 4000 on 32 cores against vendor libraries andScaLAPACK. The higher efficiency of PLASMA comparedto TBLAS is essentially due to a better data reuse. In-deed, PLASMA scheduling strategy maximizes data reuseand thus benefits from a better cache effect than TBLASwhose scheduler does not take into account date reuse. PLAS-MA is even faster than the parallel DGEMM reference upto a matrix size N = 2000 on the Intel64 machine when16 cores are used. This is not contradictory since PLAS-MA does not rely on the parallel version of DGEMM. EachPLASMA thread indeed uses the serial dgemm-seq. Betterperformance can then be achieved thanks to better schedul-ing. However, the DPOTRF factorization cannot only beperformed with efficient level-3 BLAS operations (it also in-volves level-2 BLAS operations). Therefore, as expected, theparallel DGEMM dominates all the other operations whenthe matrix size becomes larger. This illustrates the factthat using a fine enough granularity (as in tile algorithms)is more critical when processing small or moderate size ma-trices. Indeed, the better load balancing allowed by a finegranularity does not impact the steady state of the DAG pro-cessing. This also explains the major improvement broughtby PLASMA compared to the other libraries on matrices ofsmall and moderate size.

Still considering the DPOTRF routine, the vendor libraries(MKL and ESSL) remain the most competitive solution ver-sus tile algorithms (but they have a lower efficiency thantile algorithms) compared to ScaLAPACK and LAPACKapproaches, up to 16 cores. However, ESSL does not scalewell to 32 cores (figures 8(j), 8(k), 8(l)). ScaLAPACK andits vendor equivalent, PESSL, both outperform ESSL on 32cores (Figure 6(g)). This result generalizes, to some extent,to all the factorization routines. ScaLAPACK (figures 8(a),8(b), 8(c)) and PESSL (figures 8(d), 8(e), 8(f)) still scaleup to 32 cores – as opposed to ESSL – and they both (with aslight advantage for PESSL) achieve a performance compa-rable to ESSL for the DGEQRF (QR) and DGETRF (LU)routines on Power6 when 32 cores are used (Figures 6). Sim-ilarly, figures 8(g), 8(h) and 8(i) illustrate the not so goodscalability of the vendor library when all the 16 cores of theIntel64 platform are used.

Figures 6(b) and 6(h) illustrate the performance of theQR factorization (DGEQRF routine) when all the availablecores are used (16 on Intel64 or 32 on Power6). PLASMAoutperforms the other libraries and TBLAS is also very com-petitive. These results demonstrate the excellent scalabilityof tile algorithms and show that it is worth performing alittle amount of extra-flops to obtain tasks more convenientto schedule. On the Intel64 machine, TBLAS actually has a

0

5

10

15

20

25

30

35

0 2000 4000 6000 8000 10000 12000

Matrix size

4xdgemm-seqDGEMMPLASMA

TBLASMKL

SCALAPACKLAPACK


0

5

10

15

20

25

30

35

0 2000 4000 6000 8000 10000 12000

Matrix size

4xdssrfb-seqDGEMMPLASMA

TBLASMKL

SCALAPACKLAPACK


0

5

10

15

20

25

30

35

0 2000 4000 6000 8000 10000 12000

Matrix size

4xdssssm-seq

DGEMMPLASMA

MKLSCALAPACK

LAPACK


0

5

10

15

20

25

30

35

0 2000 4000 6000 8000 10000 12000

Matrix size

2xdgemm-seq

DGEMMPLASMA

TBLASESSL

PESSLSCALAPACK

LAPACK


0

5

10

15

20

25

30

35

0 2000 4000 6000 8000 10000 12000

Matrix size

2xdssrfb-seq

DGEMMPLASMA

TBLASESSL

PESSLSCALAPACK

LAPACK


0

5

10

15

20

25

30

35

0 2000 4000 6000 8000 10000 12000

Matrix size

2xdssssm-seq

DGEMMPLASMA

ESSLPESSL

SCALAPACKLAPACK

(f) DGETRF - Power6 - 2 coresFigure 5: Performance comparison when a single socket is used (Gflop/s).

better performance than PLASMA when 16 cores are usedand when the matrix size is larger than or equal to 10,000(N ≥ 10, 000). Indeed, when the matrices processed arelarge, the critical issue of scheduling corresponds to maxi-mizing a steady state throughput. The main disadvantageof a static schedule is that cores may be stalling in situa-tions where work is available. This throughput is easier tomaximize with a dynamic scheduling strategy. Approachessuch as TBLAS, which do implement a dynamic scheduling,are thus likely to achieve a higher performance than ap-proaches that implement a static scheduling (such as PLAS-MA currently does). All in all, these results are motivationto move towards an hybrid scheduling strategy that wouldassign priorities according to a trade off between data reuseand critical path progress and would process available tasksdynamically. Figure 6(e) illustrates the performance of theQR factorization (DGEQRF routine) when only half of theavailable cores are used on Power6. In this case, PLASMAstill achieves the highest performance but TBLAS and ESSLalmost reach a similar performance.

Finally, figures 6(c), 6(f) and 6(i) show that the LU fac-torization (DGETRF routine) has a performance behaviorsimilar to the QR factorization and PLASMA again out-performs the other libraries. However, the lower efficiencyof dssssm-seq compared to dssrfb-seq (dssssm-seq performsmore extra-flops) induces a lower performance of the PLAS-MA LU factorization compared to the PLASMA QR one.On Intel64, this leads MKL to be slightly better than PLAS-MA when the matrix size is larger than or equal to 10,000(N ≥ 10, 000). But, similarly to the QR case, moving to-wards a hybrid scheduling should remove the penalty due tothe static scheduling strategy used in PLASMA and strongly

improve the performance on large matrices. Indeed, on In-tel64 when 16 cores are used, the PLASMA LU peak nor-malized efficiency (see definition in Section 2.3) is equal to78.6%; so there is still room for improving the performanceby enabling dynamic scheduling. Furthermore, as alreadymentioned, an optimized implementation of the dssssm-seqkernel will also improve the performance of tile algorithms.

5. CONCLUSION AND PERSPECTIVESThis paper has analyzed and discussed the behavior of

several software approaches for achieving high performanceand portability of one-sided factorizations on multi-core ar-chitectures. In particular, we have shown the performanceimprovements brought by tile algorithms on up to 32 cores– the largest shared memory multi-core system we could ac-cess. We may expect that these results generalize somewhatto other linear algebra algorithms and even any algorithmthat can be expressed by a DAG of fine-grain tasks. Prelim-inary experiments using tile algorithms for two-sided trans-formations, i.e., the Hessenberg reduction [21] (first step forthe standard eigenvalue problem) and the bidiagonal reduc-tion [22] (first step for the singular value decomposition),show promising results.

We have shown the efficiency of our pruned-search method-ology for tuning PLASMA. However, we currently need tomanually pick up the (NB,IB) samples (from the serial level-3 kernel benchmarking) that are going to be tested in theparallel factorizations. We are currently working to auto-mate this pruning process. Furthermore, not all the matrixsizes and number of cores can be tested. We are also workingon the interpolation of the optimum tuning parameters froma limited number of parallel executions among the range of

0

20

40

60

80

100

120

140

0 2000 4000 6000 8000 10000 12000

Matrix size

16xdgemm-seqDGEMMPLASMA

TBLASMKL

SCALAPACKLAPACK


0

20

40

60

80

100

120

140

0 2000 4000 6000 8000 10000 12000

Matrix size

16xdssrfb-seqDGEMMPLASMA

TBLASMKL

SCALAPACKLAPACK


0

20

40

60

80

100

120

140

0 2000 4000 6000 8000 10000 12000

Matrix size

16xdssssm-seq

DGEMMPLASMA

MKLSCALAPACK

LAPACK


0

50

100

150

200

250

300

0 2000 4000 6000 8000 10000 12000

Matrix size

16xdgemm-seq

DGEMMPLASMA

TBLASESSL

PESSLSCALAPACK

LAPACK


0

50

100

150

200

250

300

0 2000 4000 6000 8000 10000 12000

Matrix size

16xdssrfb-seq

DGEMMPLASMA

TBLASESSL

PESSLSCALAPACK

LAPACK


0

50

100

150

200

250

300

0 2000 4000 6000 8000 10000 12000

Matrix size

16xdssssm-seq

DGEMMPLASMA

ESSLPESSL

SCALAPACKLAPACK

(f) DGETRF - Power6 - 16 cores

0

100

200

300

400

500

600

0 2000 4000 6000 8000 10000 12000

Matrix size

32xdgemm-seq

DGEMMPLASMA

TBLASESSL

PESSLSCALAPACK

LAPACK

(g) DPOTRF - Power6 - 32 cores

0

100

200

300

400

500

600

0 2000 4000 6000 8000 10000 12000

Matrix size

32xdssrfb-seq

DGEMMPLASMA

TBLASESSL

PESSLSCALAPACK

LAPACK

(h) DGEQRF - Power6 - 32 cores

0

100

200

300

400

500

600

0 2000 4000 6000 8000 10000 12000

Matrix size

32xdssssm-seq

DGEMMPLASMA

ESSLPESSL

SCALAPACKLAPACK

(i) DGETRF - Power6 - 32 coresFigure 6: Performance comparison on a large number of cores (Gflop/s).

cores and matrix sizes to the full set of possibilities. This on-going auto-tuning work should eventually be incorporatedwithin the PLASMA distribution.

Furthermore, because the factorization performance stronglydepends on the computational intensive serial level-3 ker-nels, their optimization is paramount. Unlike DGEMM,the dssrfb-seq and dssssm-seq kernels are not a single callto level-3 BLAS operations, but are composed of successivecalls, since the inefficiency. dssrfb-seq and dssssm-seq couldachieve similar performance if implemented as a monolithiccode and heavily optimized.

The experiments have also shown the limits of static schedul-ing for the factorization of large matrices. We are currentlyworking on the implementation of a hybrid scheduling forPLASMA. Even if they are not on the critical path, taskswill be dynamically scheduled on idle cores so as to maximizedata reuse.

6. REFERENCES[1] H. Sutter. A fundamental turn toward concurrency in

software. Dr. Dobb’s Journal, 30(3), 2005.

[2] A. Buttari, J. Langou, J.Kurzak, and J.Dongarra. Aclass of parallel tiled linear algebra algorithms formulticore architectures. Parallel Computing,35(1):38–53, 2009.

[3] A. Buttari, J. Dongarra, J. Kurzak, J. Langou,P. Luszczek, and S. Tomov. The impact of multicoreon math software. PARA 2006, Umea, Sweden, June2006.

[4] E. Anderson, Z. Bai, C. Bischof, Demmel J., andJ. Dongarra. LAPACK Users’ Guide/LAPACK QuickReference Guide to the Driver Routines: Release 2.0.SIAM Press, 1995.

[5] L. S. Blackford, J. Choi, A. Cleary, E. D’Azevedo,J. Demmel, I. Dhillon, J. Dongarra, S. Hammarling,G. Henry, A. Petitet, K. Stanley, D. Walker, and R. C.

0

20

40

60

80

100

120

140

0 2000 4000 6000 8000 10000 12000

Matrix size

16xdgemm-seq16 cores14 cores12 cores10 cores8 cores6 cores4 cores2 cores1 core

(a) DPOTRF - Intel64

0

20

40

60

80

100

120

140

0 2000 4000 6000 8000 10000 12000

Matrix size

16xdssrfb-seq16 cores14 cores12 cores10 cores8 cores6 cores4 cores2 cores1 core

(b) DGEQRF - Intel64

0

20

40

60

80

100

120

140

0 2000 4000 6000 8000 10000 12000

Matrix size

16xdssssm-seq

16 cores14 cores12 cores10 cores

8 cores6 cores4 cores2 cores1 core

(c) DGETRF - Intel64

0

100

200

300

400

500

600

0 2000 4000 6000 8000 10000 12000

Matrix size

32xdgemm-seq

32 cores16 cores8 cores4 cores2 cores1 core

(d) DPOTRF - Power6

0

100

200

300

400

500

600

0 2000 4000 6000 8000 10000 12000

Matrix size

32xdssrfb-seq


(e) DGEQRF - Power6

0

100

200

300

400

500

600

0 2000 4000 6000 8000 10000 12000

Matrix size

32xdssssm-seq

32 cores16 cores


(f) DGETRF - Power6Figure 7: PLASMA performance (Gflop/s).

Whaley. ScaLAPACK Users’ Guide. SIAM, 1997.

[6] N. Christofides. Graph Theory: An algorithmicApproach. 1975.

[7] BLAS: Basic linear algebra subprograms.http://www.netlib.org/blas/.

[8] F. Song, A. YarKhan, and J. Dongarra. Dynamic taskscheduling for linear algebra algorithms ondistributed-memory multicore systems. Technicalreport, UTK CS Technical Report 638, 2009.

[9] J. Dongarra, J. Bunch, C. Moler, and G.W. Stewart.LINPACK User’s Guide. SIAM, 1979.

[10] B. T. Smith, J. M. Boyle, J. J. Dongarra, B. S.Garbow, Y. Ikebe, V. C. Klema, and C. B. Moler.Matrix Eigensystem Routines-EISPACK Guide,volume 6. Springer-Verlag, 1976.

[11] Intel, Math Kernel Library (MKL).http://www.intel.com/software/products/mkl/.

[12] IBM, Engineering and Scientific Subroutine Library(ESSL) and Parallel ESSL.http://www-03.ibm.com/systems/p/software/essl/.

[13] W. Gropp, E. Lusk, and A. Skjellum. Using MPI:portable parallel programming with the message-passinginterface. MIT Press, Cambridge, MA, USA, 1994.

[14] PBLAS: Parallel basic linear algebra subprograms.http://www.netlib.org/scalapack/pblasqref.pdf.

[15] BLACS: Basic linear algebra communicationsubprograms. http://www.netlib.org/blacs/.

[16] SMP Superscalar. http://www.bsc.es/ → ComputerSciences → Programming Models → SMP Superscalar.

[17] H. Meuer, E. Strohmaier, J. Dongarra, and H. Simon.TOP500 Supercomputer Sites, 32nd edition, November

2008. (The report can be downloaded fromhttp://www.netlib.org/benchmark/top500.html).

[18] A. Sloss, D. Symes, and C. Wright. ARM SystemDeveloper’s Guide: Designing and Optimizing SystemSoftware. Morgan Kaufmann Publishers Inc., SanFrancisco, CA, USA, 2004.

[19] R. C. Whaley. Empirically tuning lapackaAZsblocking factor for increased performance. ComputerAspects of Numerical Algorithms, Wisla, Poland,October 20-22, 2008.

[20] C. Whaley and M. Castaldo. Achieving accurate andcontext-sensitive timing for code optimization.Software: Practice and Experience, 38(15):1621–1642,2008.

[21] H. Ltaief, J. Kurzak, and J. Dongarra. Parallel blockhessenberg reduction using algorithms-by-tiles formulticore architectures revisited. University ofTennessee CS Tech. Report, UT-CS-08-624 (alsoLAWN #208), 2008.

[22] H. Ltaief, J. Kurzak, and J. Dongarra. Parallel bandtwo-sided matrix bidiagonalization for multicorearchitectures. University of Tennessee CS Tech.Report, UT-CS-08-631 (also LAWN #209), 2008.

0

20

40

60

80

100

120

140

0 2000 4000 6000 8000 10000 12000

Matrix size



0

20

40

60

80

100

120

140

0 2000 4000 6000 8000 10000 12000

Matrix size

16xdssrfb-seq16 cores8 cores4 cores2 cores1 core

(b) DGEQRF - Intel64

0

20

40

60

80

100

120

140

0 2000 4000 6000 8000 10000 12000

Matrix size

16xdssssm-seq


(c) DGETRF - Intel64

0

100

200

300

400

500

600

0 2000 4000 6000 8000 10000 12000

Matrix size

32xdgemm-seq


(d) DPOTRF - Power6

0

100

200

300

400

500

600

0 2000 4000 6000 8000 10000 12000

Matrix size

32xdssrfb-seq


(e) DGEQRF - Power6

0

100

200

300

400

500

600

0 2000 4000 6000 8000 10000 12000

Matrix size

32xdssssm-seq

32 cores16 cores


(f) DGETRF - Power6

0

20

40

60

80

100

120

140

0 2000 4000 6000 8000 10000 12000

Matrix size


(g) DPOTRF - Intel64

0

20

40

60

80

100

120

140

0 2000 4000 6000 8000 10000 12000

Matrix size

16xdssrfb-seq16 cores8 cores4 cores2 cores1 core

(h) DGEQRF - Intel64

0

20

40

60

80

100

120

140

0 2000 4000 6000 8000 10000 12000

Matrix size

16xdssssm-seq


(i) DGETRF - Intel64

0

100

200

300

400

500

600

0 2000 4000 6000 8000 10000 12000

Matrix size

32xdgemm-seq


(j) DPOTRF - Power6

0

100

200

300

400

500

600

0 2000 4000 6000 8000 10000 12000

Matrix size

32xdssrfb-seq


(k) DGEQRF - Power6

0

100

200

300

400

500

600

0 2000 4000 6000 8000 10000 12000

Matrix size

32xdssssm-seq

32 cores16 cores


(l) DGETRF - Power6Figure 8: ScaLAPACK (a,b,c) and PESSL (d,e,f) MKL (g,h,i) and ESSL (j,k,l) performance (Gflop/s).

0

20

40

60

80

100

120

140

0 2000 4000 6000 8000 10000 12000

Matrix size



0

100

200

300

400

500

600

0 2000 4000 6000 8000 10000 12000

Matrix size

32xdgemm-seq

32 cores16 cores


(b) DPOTRF - Power6Figure 9: LAPACK performance (Gflop/s).

Comparative Study of One-Sided Factorizations with Multiple

Documents

Transcript of Comparative Study of One-Sided Factorizations with Multiple