Multi-Pivot Quicksort: Theory and Experimentsskushagr/multipivotQuicksort.pdf · the algorithm is...

Multi-Pivot Quicksort: Theory and Experiments

Shrinu Kushagra

[email protected]

University of Waterloo

Alejandro Lopez-Ortiz

[email protected]


J. Ian Munro

[email protected]


Aurick Qiao

[email protected]


November 7, 2013

Abstract

The idea of multi-pivot quicksort has recently receivedthe attention of researchers after Vladimir Yaroslavskiyproposed a dual pivot quicksort algorithm that, con-trary to prior intuition, outperforms standard quicksortby a a significant margin under the Java JVM [10]. Morerecently, this algorithm has been analysed in terms ofcomparisons and swaps by Wild and Nebel [9]. Our con-tributions to the topic are as follows. First, we performthe previous experiments using a native C implementa-tion thus removing potential extraneous effects of theJVM. Second, we provide analyses on cache behaviorof these algorithms. We then provide strong evidencethat cache behavior is causing most of the performancedifferences in these algorithms. Additionally, we buildupon prior work in multi-pivot quicksort and proposea 3-pivot variant that performs very well in theory andpractice. We show that it makes fewer comparisons andhas better cache behavior than the dual pivot quicksortin the expected case. We validate this with experimen-tal results, showing a 7-8% performance improvementin our tests.

1 Introduction

1.1 Background Up until about a decade ago it wasthought that the classic quicksort algorithm [3] usingone pivot is superior to any multi-pivot scheme. It waspreviously believed that using more pivots introducestoo much overhead for the advantages gained. In 2002,Sedgewick and Bentley [7] recognised and outlined someof the advantages to a dual-pivot quicksort. However,the implementation did not perform as well as the classicquicksort algorithm [9] and this path was not exploredagain until recent years.

In 2009, Vladimir Yaroslavskiy introduced a noveldual-pivot partitioning algorithm. When run on a bat-tery of tests under the JVM, it outperformed the stan-dard quicksort algorithm [10]. In the subsequent re-lease of Java 7, the internal sorting algorithm was re-placed by Yaroslavskiy’s variant. Three years later,Wild and Nebel [9] published a rigorous average-caseanalysis of the algorithm. They stated that the previ-ous lower bound relied on assumptions that no longerhold in Yaroslavskiy’s implementation. The dual pivotapproach actually uses less comparisons (1.9n lnn vs2.0n lnn) on average. However, the difference in run-time is much greater than the difference in number ofcomparisons. We address this issue and provide an ex-planation in §5.

Aumuller and Dietzfelbinger [1] (ICALP2013) haverecently addressed the following question: If the previ-ous lower bound does not hold, what is really the bestwe can do with two pivots? They prove a 1.8n lnn lowerbound on the number of comparisons for all dual-pivotquicksort algorithms and introduced an algorithm thatactually achieves that bound. In their experimentation,the algorithm is outperformed by Yaroslavskiy’s quick-sort when sorting integer data. However, their algo-rithm does perform better with large data (eg. strings)since comparisons incur high cost.

1.2 The Processor-Memory Performance GapBoth presently and historically, the performance of CPUregisters have far outpaced that of main memory. Ad-ditionally, this performance gap between the processorand memory has been increasing since their introduc-tion. Every year, the performance of memory improvesby about 10% while that of the processor improves by60% [5]. The performance difference grows so quickly

47 Copyright © 2014.by the Society for Industrial and Applied Mathematics.

Dow

nloa

ded

09/2

9/15

to 1

29.9

7.16

8.13

4. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p

that increasingly more levels of cache (L1, L2 and L3)have been introduced to bridge the gap. This resultsin an ever-changing computer architecture where cacheeffects in programs gradually grow more significant.

1.3 Our Work We provide evidence that the re-cent discovery of fast multi-pivot quicksort algorithms isdriven by the aforementioned cache effects. Generally,these algorithms perform more steps of computation butalso incur fewer cache faults in order to break down theproblem into smaller subproblems. With computationperformance improving much more quickly, it is intu-itive that these multi-pivot schemes would, over time,gain an advantage over the classic one pivot algorithm.Thus, we believe that if the trend continues, it will be-come advantageous to perform more computation to usemore pivots.

We present a multi-pivot quicksort variant thatmakes use of three pivots. We prove that our approachmakes, on average, fewer comparisons (1.84n ln(n) vs1.9n ln(n)) and more swaps than the dual pivot ap-proach. However, in our experiments, the 3-pivot algo-rithm is about 7-8% faster than Yaroslavskiy’s 2-pivotalgorithm. Similar to Yaroslavskiy’s quicksort, our al-gorithm performs much better in practice than the dif-ferences in comparisons and moves would predict. Wepresent analyses of the cache behaviors of the variousquicksort schemes. The results of our analyses givestrong evidence that caching is in fact causing the per-formance differences observed.

With the increasing processor-memory performancegap in mind, we consider the technique of presamplingpivots. This technique performs a significant amount ofcomputation to precompute many pivots, with the goalof reducing cache faults. Our experiments show that,on modern architectures, this idea achieves a 1.5-2%improvement in performance.

2 Multi-Pivot Quicksort: 3-pivot

We introduce a variant of quicksort that makes use ofthree pivots p < q < r. At each iteration, the algorithmpartitions the array around the three pivots into foursubarrays and recursively sorts each of them. At firstglance, this algorithm seems to be performing the samework as two levels of regular 1-pivot quicksort in onepartition step. However, note that the middle pivot qis of higher quality since it is a median of three pivots.This is the same as a regular quicksort that picks amedian-of-3 pivot for every recursive call at alternatingdepths. Thus, we expect the performance of the 3-pivot variant to be between classic quicksort and classicquicksort using a median-of-3 pivot. Later, we shall seethat it actually outperforms median-of-3 quicksort in

Figure 1: Invariant kept by the partition algorithm. Allelements before pointer b are less than q, all elementsbefore pointer a are less than p, all elements afterpointer c are greater than q, and all elements afterpointer d are greater than r. All other elements(between pointers b and c inclusive) have not yet beencompared.

practice by a significant margin.

2.1 Partition Algorithm The partition algorithmuses four pointers: a, b, c, and d, which keep theinvariant shown in Figure 1. Pointers a and b initiallypoint to the first element of the array while c and dinitially point to the last element of the array. Thealgorithm works by advancing b and c toward each otherand moving each element they pass through into thecorrect subarray, terminating when b and c pass eachother (b > c).

When A[b] < q, if A[b] < p, it swaps A[b] andA[a] and increments a and b, or else does nothing andincrements b. This case is symmetric to the case whenA[c] > q. When A[b] > q and A[c] < q then thealgorithm swaps both elements into place using oneof the four cases (A[b] < r and A[c] > p, etc.), thenincrements/decrements a, b, c, and d accordingly. Referto algorithm A.1.1 for pseudocode.

3 Analysis

In the next few subsections, we give analyses for the3-pivot quicksort algorithm, as well as cache behavioranalyses for 1- and 2-pivot quicksorts. We show that the3-pivot algorithm makes, on average, fewer comparisonsand cache misses than the 1- or 2-pivot algorithms.

Assumptions for 3-pivot quicksort Throughout thenext few sections we make the following assumptions:

1. The input array is a random permutation of1, . . . , n

2. The elements indexed at the first quartile, themedian and the third quartile are chosen as thethree pivots. On random permutations this is thesame as choosing them at random. Hence each

triplet appears with probablity(n3

)−1.

Given these assumptions, the expected value (or cost)of each of the 3-pivot quantities being analysed can be


Dow

nloa

ded

09/2

9/15

to 1

29.9

7.16

8.13

4. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p

represented by the following recursive formula:

fn = pn +6

n(n− 1)(n− 2)

n−3∑i=0

n−2∑j=i+1

n−1∑k=j+1

((3.1)

fi + fj−i−1 + fk−j−1 + fn−k−1

)= pn +

12

n(n− 1)(n− 2)

n−3∑i=0

(n− i− 1)(n− i− 2)fi

where fn denotes the expected cost (or number of com-parisons) and pn represents the expected partitioningcost of the property being analysed. The solutions tothese recurrences can be found in Appendix A.2.

Notation In our analyses we shall use the followingnotation:

1. Cp(n) – expected number of comparisons of thep-pivot quicksort algorithm sorting an array of nelements

2. Sp(n) – expected number of swaps of the p-pivotquicksort algorithm sorting an array of n elements

3. CMp(n) – expected number of cache misses of thep-pivot quicksort algorithm sorting an array of nelements

4. SPp(n) – expected number of recursive calls to asubproblem greater in size than a block in cacheinvoked by the p-pivot quicksort algorithm sortingan array of n elements

3.1 Number of Comparisons

Theorem 3.1. C3(n) = 2413 n lnn + O(n) ≈

1.846n lnn+O(n)

Proof. The algorithm chooses three pivots and sortsthem. This costs 8

3 comparisons on average. Let thethree pivots chosen be p, q and r with p < q < r. Itis easy to see that each element is compared exactlytwice to determine its correct location. First with qand depending upon the result of this comparison eitherwith p (if less) or r (if greater). Thus the expectednumber of comparisons in a single partition step is givenby pn = 2(n− 3) + 8

3 . Using the above value of pn andplugging it in equation (3.1) gives,

C3(n) = fn =24

13n lnn+O(n)

The mathematical details are omitted here for brevity.Full details can be found in Appendix A.2. The same

result can be derived using the ideas presented in thePhD thesis of Hennequin [2], who took a more generalapproach and showed that if the partitioning costs areof the form pn = αn + O(1) then a 3-pivot quicksortwould have a total cost of 12

13αn lnn+O(n).

This is a lower number of comparisons than both the1-pivot algorithm (2.0n lnn) and the 2-pivot algorithm(1.9n lnn). This theoretical result is validated by ourexperiments as well. Figure 4 in §4 clearly shows thatthe 3-pivot variant makes much fewer comparisons thanits 1- and 2-pivot counterparts. One more point tonote here is that in Yaroslavskiy’s 2-pivot partitioningalgorithm, pn depends upon whether the algorithmcompares with p or q first [9]. This is not the case in3-pivot algorithm because of its symmetric nature.

Tan in his PhD thesis [8] had also analysed thenumber of comparisons for a 3-pivot quicksort variant.He had also obtained an expected count of 1.846n lnn+O(n) for the number of comparisons. However, hisalgorithm made 3 passes of the input. First pass topartition about the middle pivot, then for the left pivotand finally for the right pivot. However, our algorithmsaves on these multiple passes and hence makes fewercache faults. This behavior is rigorously analysed in§3.3.

3.2 Number of Swaps

Theorem 3.2. S3(n) = 813n lnn+O(n) ≈ 0.615n lnn+

O(n)

Proof. The 3-pivot algorithm makes two kinds of swaps.Thus the partitioning process can be viewed as beingcomposed of two parts. The first part partitions theelements about q (the middle pivot). This step is thesame as a 1-pivot partition. In the second part, thetwo parts obtained are further subdivided into two morepartitions leading to a total of four partitions. However,the second part is different from the normal 1-pivotpartitioning process. Here the partition is achieved onlyby the way of swaps. This process is detailed in Figure2.

The algorithm maintains four pointers a, b, c and das shown in Figure 1. The left pointer a is incrementedwhen an element is found to be less than p in which caseit is swapped to the location pointed to by a. Similaranalysis holds for the rightmost pointer d. The swapsmade in the second part can be given by i+n−k wherei and k are the final positions of the pivots p and r.Hence, the total number of swaps in given by:

S3(n) = i+ n− k + swaps made partitioning about q

The swaps made during partitioning using single pivotwas analysed by Sedgewick in 1977 [6] and their number


Dow

nloa

ded

09/2

9/15

to 1

29.9

7.16

8.13

4. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p

i k

1-pivot

Figure 2: Swaps made in the partitioning process. Twotypes of swaps are made. The ones shown in biggerarrows are similar to the swaps made in the 1-pivot case.The ones shown in smaller arrows are made everytimean element is placed in the leftmost or rightmost bucketsrespectively.

is given by n−26 . Hence the expected number of swaps

in the partitioning process is given by:

pn =n− 2

6+

6

n(n− 1)(n− 2)

n−3∑i=0

n−2∑j=i+1

n−1∑k=j+1

i+ n− k

=4n+ 1

6

Plugging the value of pn in Equation (3.1) andsolving the recurrence gives the expected number ofswaps for the 3-pivot quicksort as:

S3(n) =8

13n ln(n) +O(n)

This is greater than the number of swaps by the1-pivot algorithm ( 1

3n ln(n) [6]) and the 2-pivot algo-rithm (0.6n ln(n) [9]) whereas 3-pivot makes 0.62n ln(n)swaps.

3.3 Cache Performance We claimed in §1 thatour 3-pivot algorithm has better cache performancethan previous variants. First we provide an intuitiveargument comparing with the 1-pivot algorithm. Inone partition step of the 3-pivot algorithm, the arrayis split into four subarrays. Two pointers start at eitherend and stop when they meet each other. Thus thesetwo pointers touch every page once. Assuming a perfectsplit, the other two pointers start at either end and scanone quarter of the array. They touch half of the pagesin the array. Thus, assuming a perfect split, the 3-pivot algorithm incurs page faults equal to 1.5 timesthe number of pages. The 1-pivot partition algorithmtouches every page in the subarray being sorted. Inorder for the 1-pivot algorithm to split the array intofour subarrays, it must partition the array once, and thetwo subarrays each once. Thus it touches every pagetwice and incurs twice as many page faults as pagesin the array. However, this performance is the worst

case for the 3-pivot partition scheme. Thus, a 3-pivotalgorithm intuitively incurs less cache faults.

Let M denote the size of the cache, B denote thesize of a cache line. In this section, for simplicity, we willobtain upper bounds on CMp(n), cache misses of the p-pivot quicksort on n elements, and SPp(n), number ofrecursive calls to a subproblem of size greater than blocksize by a p-pivot quicksort.

1-pivot Quicksort The upper bound for the 1-pivotcase was obtained by LaMarca and Ladner [4]. Theyshowed the following:

CM1(n) ≤ 2(n+ 1)

Bln

(n+ 1

M + 2

)+O

( nB

)SP1(n) ≤ 2(n+ 1)

M + 2− 1

where CM(1/3) and SP(1/3) denote the same quantitiesfor median-of-3 1-pivot quicksort.

Theorem 3.3. CM(1/3)(n) ≤ 127

(n+1B

)ln(n+1M+2

)+

O(nB

)and SP(1/3)(n) ≤ 12

7

(n+1M+2

)− 2 +O

(1n

)Proof. Refer to appendix A.3.

2-pivot Quicksort

Theorem 3.4. CM2(n) ≤ 85

(n+1B

)ln(n+1M+2

)+ O

(nB

)and SP2(n) ≤ 12

10

(n+1M+2

)− 1

2 +O(

1n4

)Proof. This algorithm uses three pointers to traversethrough the array. Hence the total number of cachemisses during partitioning will be at most the totalnumber of elements scanned by these pointers dividedby B. This gives rise to the following recurrencerelations:

CM2(n) ≤ 4n+ 1

3B+

6

n(n− 1)

n−2∑i=0

(n− i− 1)CM2(i)

SP2(n) ≤ 1 +6

n(n− 1)

n−2∑i=0

(n− i− 1)NS2(i)

The recurrence for number of subproblems is self-explanatory. A minor point is that the above relationholds for n > M . For n ≤M , CM2(n) = 0. Solving theabove recurrences we get,

CM2(n) ≤ 8

5

(n+ 1

B

)ln

(n+ 1

M + 2

)+O

( nB

)SP2(n) ≤ 12

10

(n+ 1

M + 2

)− 1

2+O

(1

n4

)


Dow

nloa

ded

09/2

9/15

to 1

29.9

7.16

8.13

4. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p

3-pivot Quicksort

Theorem 3.5. CM3(n) ≤ 1813

(n+1B

)ln(n+1M+2

)+O

(nB

)and SP3(n) ≤ 12

13

(n+1M+2

)− 1

3 +O(1n

)Proof. This algorithm uses four pointers to traversethrough the array. Hence the total number of cachemisses during partitioning will be at most the totalnumber of elements scanned by these pointer dividedby B. Hence the partitioning costs for CM3(n) is givenby, 3n+1

2B and for SP3(n) by 1. Solving we get,

CM3(n) ≤ 18

13

(n+ 1

B

)ln

(n+ 1

M + 2

)+O

( nB

)SP3(n) ≤ 12

13

(n+ 1

M + 2

)− 1

3+O

(1

n

)One point to note is that we are overestimating

(upper-bounding) the number of cache misses. This isbecause some of the elements of the left sub-problemmight still be in the cache when the subproblem for thatsubarray is solved. But for the purposes of this analysiswe have ignored these values. Additionally, these cachehits seem to affect only the linear term as was analysedby LaMarca and Ladner in [4]. Hence the asymptoticbehaviour is still accurately approximated by theseexpressions. Note that 3-pivot quicksort algorithm has50% and 25% less cache faults than 1- and 2-pivotalgorithms, respectively.

4 Experiments

The goal for our experiments is to simplify the envi-ronment the code is running in by as much as possibleto remove extraneous effects from the JVM. This way,it is simpler to identify key factors in the experimentalresults. As such, we wrote all tests in C.

We ran rigorous experiments comparing classicquicksort, Yaroslavskiy’s 2-pivot variant, our 3-pivotvariant, as well as optimized versions of them. Opti-mized 1-pivot quicksort picks a pivot as a median ofthree elements. Optimized 2-pivot quicksort picks twopivots as the second and fourth of five elements. Op-timized 3-pivot quicksort picks three pivots as the sec-ond, fourth, and sixth of seven elements. In addition,all three switch to insertion sort at the best subproblemsize determined experimentally for each. The unopti-mized versions do none of these.

For the experiments shown, we ran each algorithmon arrays containing a random permutation of the 32-bitintegers 1 . . . n, where n is the size of the array. Tests onthe smallest array sizes were averaged over thousands oftrials, which is gradually reduced to 2-10 trials for the

Figure 3: Plot of runtime against the size of array forthe various quicksort algorithms. The size, n, is plottedlogarithmically and the runtime is divided by n lnn.

largest array sizes. All experiments were run on themachine specified in Table A.4.1.

4.1 Runtime, Comparisons, and AssignmentsFigure 3 shows the experiment in runtime. The un-optimized 3-pivot variant is faster than both the opti-mized and unoptimized versions of the 1-pivot and 2-pivot quicksort algorithms. Recall that 3-pivot quick-sort is similar to a mix between 1-pivot quicksort andoptimized 1-pivot quicksort, yet it significantly outper-forms both of them. The graph also shows that the per-formance is consistent, doing as just as well for smallnumbers of elements as for large numbers of elements.

Figure 4 shows the experiment in comparisons. Thegraph confirms the results of our analysis. The 3-pivot version uses fewer comparisons than the 2-pivotversion. Note here that the optimized 3-pivot algorithmuses more comparisons on small input sizes but stilloutperforms the others in runtime.

Swaps are implemented with three assignment op-erations using a temporary variable. In our implemen-tations, multiple overlapping swaps are optimized touse fewer assignments. for example, swap(a, b) followedby swap(b, c) can be done with only four assignments.Thus, instead of counting swaps, we count the number


Dow

nloa

ded

09/2

9/15

to 1

29.9

7.16

8.13

4. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p

Figure 4: Plots of number of comparisons and assign-ments against the size of array for the various quick-sort algorithms. The size, n, is plotted logarithmicallyand the comparisons/assignments are divided by n lnn.Note that the graphs for number of comparisons ap-pear to approach the correct coefficients calculated inthe analysis.

of assignment operations done. Figure 4 shows theseresults. The classic 1-pivot algorithm uses far fewer as-signments than the other variants. Our 3-pivot algo-rithm uses slightly fewer assignments than the 2-pivotalgorithm. It is expected that the graphs look slightlydifferent from our swap analysis.

4.2 Comprehensive Tests In addition to the sim-ple tests shown, we also ran two sets of comprehensivetests. These tests were ran on two different platformsin order to highlight artifacts from differing computerarchitectures. The low level details of the platforms aredescribed in Appendix A.4. The compiler used for alltests is:

gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3

The first set of these tests evaluated runtime perfor-mance on different input distributions (see AppendixA.5). The different input types we considered are:

1. Permutation: A random permutation of the inte-gers from 1 to n (see Figure A.5.2 and Figure A.5.7)

2. Random: n random elements were selected from1 . . .√n (see Figure A.5.3 and Figure A.5.8)

3. Decreasing: The integers n to 1, in decreasing order(see Figure A.5.6 and Figure A.5.11)

4. Increasing: The integers 1 to n, in increasing order(see Figure A.5.5 and Figure A.5.10)

5. Same: n equal integers (see Figure A.5.4 andFigure A.5.9)

The 3-pivot algorithm performs well in all the testsexcept for the ”same” distribution on platform 1. Sincethis is not observed in platform 2, we conclude thatartifacts due to architecture play a significant role inperformance.

The second set of tests evaluated runtime perfor-mance under different GCC optimization levels (see Ap-pendix A.6). The graphs show runtimes of the algo-rithms compiled with the -O0, -O1, -O2 and -O3 flags.We see that the results are much less uniform and aredependent on the platform and optimization flag. How-ever, in most cases, the 3-pivot algorithm still outper-forms the others. Using -O0, 3-pivot is faster on bothplatforms. Using -O1, 3-pivot is faster on one platformand only slightly slower than 2-pivot on the other. Using-O2 and -O3, the standard version of 3-pivot is fasterthan the standard version of 2-pivot quicksort, whilethe reverse is true for the optimized versions. Betterunderstanding of the algorithms and fine-tuning themunder compiler optimizations is an area which we markfor future work.


Dow

nloa

ded

09/2

9/15

to 1

29.9

7.16

8.13

4. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p

Variant Cache Misses Comparisons Swaps

1-pivot 2(n+1B

)ln n+1

M+2 2n lnn 0.333n lnn

1-pivot (median of 3) 1.71(n+1B

)ln n+1

M+2 1.71n lnn 0.343n lnn

2-pivot (Yaroslavskiy) 1.6(n+1B

)ln n+1

M+2 1.9n lnn 0.6n lnn

3-pivot 1.38(n+1B

)ln n+1

M+2 1.85n lnn 0.62n lnn

Table 1: Summary of previous results [6, 9] and results of our analysis. Each value has lower order terms thathave been omitted for conciseness.

4.3 Other Experiments Other multi-pivot algo-rithms are also of interest. In particular, we also rantests on a 7-pivot approach. However, these tests con-cluded that the 7-pivot algorithm runs more slowly thanthe 2- and 3-pivot variants.

Another feature of consequence is the behavior ofalgorithms under a multi-core architecture. Thus weperformed a set of tests on these versions of quicksortrunning on four threads on a machine with four cores.The scheme we used to split work is as follows: Use avery large sample to perform a 4-way/3-pivot partitionof the array into four subarrays of (probably) verysimilar sizes. Then run an instance of a quicksortalgorithm on each of the four subarrays. One factto note here is that under this scheme, the runtimeof the entire algorithm is the max of the runtimes ofthe four instances. Thus, a fast algorithm with highvariance in runtime may actually perform worse thana slower algorithm that has a consistent runtime. Ourtests concluded that all three of the 1-, 2-, and 3-pivotapproaches showed comparable speedups (about threetimes faster than single threaded) when run under theseconditions.

5 Theory and Experiments: Explained

The dramatic speed increase of Yaroslavskiy’s algorithmis not fully explained by previous analyses of the numberof comparisons and swaps. We see a 7-9% increase inperformance but the average number of comparisons isonly 5% less, and almost twice as many swaps! This dis-parity between theory and practice is highlighted evenmore clearly with the results of our 3-pivot algorithm(refer Table 1). Our algorithm uses more comparisonsand swaps than the median-of-3 quicksort algorithm yetwe see about a 7% reduction in runtime.

After analysing the cache performance of each ofthe algorithms, we can finally explain the disparitywe see between theory and practice. Even thoughour algorithm uses more swaps and comparisons thanmedian-of-3 quicksort, it make almost 20% fewer cachemisses. This explains why our algorithm performsbetter even though traditional analyses say that it

should do much worse. It also explains why we see sucha speed increase for Yaroslavskiy’s dual-pivot quicksort.

6 Further Cache Improvements

With the insights from caching in modern architec-tures, we design a modification based on the idea ofpresampling pivots. Given an initial unsorted sequencea1, . . . , an of size n, the main ideas of this algorithm canbe summarised as follows:

1. Sample√n elements to be used as pivots for the

partitioning process and sort them. This is donejust once at the start and not for each recursivecall.

2. For every recursive call, instead of choosing a pivotfrom the subarray, choose an appropriate elementfrom the above array as a pivot. Partition the arrayabout the chosen pivot.

3. Once we run out of pivots, fall back to the standardquicksort algorithm (1-pivot, 2-pivot, etc. as thecase may be).

This strategy has some nice properties. By choosingpivots out of a sample of

√n elements, the initial pivots

are extremely good with very high probability. Hence,we expect that using presampling would bring downthe number of subproblems below the cache size morequickly.

We implemented this approach and carried outsome experiments, the details of which have been omit-ted due to space constraints. In practice, it leads toabout a 1.5 − 2% gain in performance when compar-ing the running times of the standard 1-pivot quicksortagainst that of 1-pivot quicksort with presampling. Forlarger array sizes, the presampled version was on aver-age about 2% faster than the standard version. Similarresults were obtained when comparing the presampledand standard versions of 2-pivot quicksort.

We believe that fine tuning this approach further,such as varying the sample size and choosing when tofall back to the standard algorithm, would lead to even


Dow

nloa

ded

09/2

9/15

to 1

29.9

7.16

8.13

4. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p

more gains in performance. Analysing this approachmathematically is another avenue which needs moreinvestigation. We mark these as areas for future work.

7 Conclusions and Future Work

First, we have confirmed previous experimental resultson Yaroslavskiy’s dual-pivot algorithm under a basic en-vironment thus showing that the improvements are notdue to JVM side effects. We designed and analysed a3-pivot approach to quicksort which yielded better re-sults both in theory and in practice. Furthermore, weprovided strong evidence that much of the runtime im-provements are from cache effects in modern architec-ture by analysing cache behavior.

We have learned that due to the rapid developmentof hardware, many of the results from more than adecade ago no longer hold. Further work in the shortterm can be directed at discovering, analysing, andimplementing more interesting multi-pivot quicksortschemes.

References

[1] Martin Aumuller and Martin Dietzfelbinger. Opti-mal partitioning for dual pivot quicksort. CoRR,abs/1303.5217, 2013.

[2] Pascal Hennequin. Combinatorial analysis of quicksortalgorithm. Informatique theorique et applications,23(3):317–333, 1989.

[3] C. A. R. Hoare. Quicksort. Comput. J., 5(1):10–15,1962.

[4] Anthony LaMarca and Richard E. Ladner. The in-fluence of caches on the performance of sorting. J.Algorithms, 31(1):66–104, 1999.

[5] D.A. Patterson and J.L. Hennessy. Computer Archi-tecture: A Quantitative Approach. Morgan Kaufmann,1996.

[6] Robert Sedgewick. The analysis of quicksort programs.Acta Inf., 7:327–355, 1977.

[7] Robert Sedgewick and Jon Bentley. Quicksort isoptimal. http://www.cs.princeton.edu/~rs/talks/

QuicksortIsOptimal.pdf, 2002. [Online; accessed 21-April-2013].

[8] Kok-Hooi Tan. An asymptotic analysis of the numberof comparisons in multipartition quicksort. 1993.

[9] Sebastian Wild and Markus E. Nebel. Average caseanalysis of java 7’s dual pivot quicksort. In LeahEpstein and Paolo Ferragina, editors, ESA, volume7501 of Lecture Notes in Computer Science, pages 825–836. Springer, 2012.

[10] Vladimir Yaroslavskiy. Dual-pivot quicksort.http://iaroslavski.narod.ru/quicksort/

DualPivotQuicksort.pdf, 2009. [Online; accessed21-April-2013].

A Appendix

A.1 Partition Algorithm

Algorithm A.1.1 3-Pivot Partition

Require: A[left] < A[left+1] < A[right] are the threepivots

1: function partition3(A, left, right)2: a← left+ 2, b← left+ 23: c← right− 1, d← right− 14: p← A[left], q ← A[left+ 1], r ← A[right]5: while b ≤ c do6: while A[b] < q and b ≤ c do7: if A[b] < p then8: swap(A[a], A[b])9: a← a+ 1

10: end if11: b← b+ 112: end while13: while A[c] > q and b ≤ c do14: if A[c] > r then15: swap(A[c], A[d])16: d← d− 117: end if18: c← c− 119: end while20: if b ≤ c then21: if A[b] > r then22: if A[c] < p then23: swap(A[b],A[a]), swap(A[a],A[c])24: a← a+ 125: else26: swap(A[b],A[c])27: end if28: swap(A[c],A[d])29: b← b+ 1, c← c− 1, d← d− 130: else31: if A[c] < p then32: swap(A[b],A[a]), swap(A[a],A[c])33: a← a+ 134: else35: swap(A[b],A[c])36: end if37: b← b+ 1, c← c− 138: end if39: end if40: end while41: a← a− 1, b← b− 1, c← c+ 1, d← d+ 142: swap(A[left+ 1],A[a]), swap(A[a],A[b])43: a← a− 144: swap(A[left],A[a]), swap(A[right],A[d])45: end function


Dow

nloa

ded

09/2

9/15

to 1

29.9

7.16

8.13

4. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p

A.2 Solving Recurrences for 3-pivot quicksortAll the quantities analysed in this paper satisfy arecurrence relation of the following form

fn = an+b+12

n(n− 1)(n− 2)

n−3∑i=0

(n− i−1)(n− i−2)fi

Multiplying by n(n− 1)(n− 2) throughout gives:

n(n− 1)(n− 2)fn = an2(n− 1)(n− 2)

+ bn(n− 1)(n− 2)

+ 12

n−3∑i=0

(n− i− 1)(n− i− 2)fi

Substituting n − 1 in the above equation and thensubtracting gives:

(n− 1)(n− 2)(n− 3)fn−1 =

a(n− 1)2(n− 2)(n− 3)

+ b(n− 1)(n− 2)(n− 3)

+ 12

n−4∑i=0

(n− i− 1)(n− i− 2)fi

n(n− 1)(n− 2)fn = (n− 1)(n− 2)(n− 3)fn−1

+ a(n− 1)(n− 2)(4n− 3)

+ 3b(n− 1)(n− 2)

+ 24

n−3∑i=0

(n− i− 2)fi

The idea is to get rid of the summation by subtractingequations. Repeating the process twice on the aboveequation gives the following equation:

n(n− 1)(n− 2)fn = 3(n− 1)(n− 2)(n− 3)fn−1

(A.1)

− 3(n− 2)(n− 3)(n− 4)fn−2

+ (n− 3)(n− 4)(n− 5)fn−3

+ 24fn−3 + 6a(4n− 9) + 6b

We use standard linear algebra software to solve thisrecursive equation giving it the appropriate initial con-ditions. All the equations analysed in this paper havethe above form. Only the value of a and b changes. Forthe case of comparisons a = 2 and b = −10

3 . For theanalysis of swaps a = 2

3 and b = 16 . Similarly for other

cases. We will show the detailed solution for the analy-sis of comparisons. Other analysis are very similar. Thesolution to (A.1) for comparisons is of the form:

C3(n) =24

13(n+ 1)Hn −

311

117+

190

117+G(n)

where Hn is the harmonic function, G(n) is a largeexpression on n output by our recurrence solver whichcontains complex numbers and the gamma function.Hence the analysis of G(n) is very important. We firstprove that the G(n) is indeed real and that it is of O( 1

n ).

Define d = 52 + 1

2 i√

23, z = 10097+i1039√

23. ThenG(n) for the analysis of comparisons is:

G(n) =

− 1

34983πΓ(n+ 1)

{cosh(

1

2π√

23)Γ(n− d)Γ(d)z

+48πΓ(n− d)z

Γ(d)

}− 10097

34983(n+ 1)

Using the properties of gamma function, Γ(n − d) =−d(−d+1)(−d+2)(−d+3) · · · (−d+n−1)Γ(−d). Hence,we get the following equations:

Γ(n− d) = z1Γ(−d)

Γ(n− d) = z1Γ(d)

where z1 = −d(−d + 1) · · · (−d + n − 1). Substitutingthese values in the above equation, we get:

G(n) =

− 1

34983πΓ(n+ 1)

{cosh(

1

2π√

23)Γ(−d)Γ(d)zz1

+48πΓ(d)zz1

Γ(d)

}− 10097

34983(n+ 1)

Now using the properties of gamma function we get:

Γ(−d)Γ(d) =−π

d sin(πd)=

−πd cos(iπ2

√23)

=−π

d cosh(π2√

23)

or

cosh(π

2

√23)Γ(−d)Γ(d) =

−πd

48Γ(d)

Γ(d)=−πd

Substituting these values in the above equation we get:

G(n) =− 1

34983πΓ(n+ 1)

{−πzz1

d− zz1π

d

}− 10097

34983(n+ 1)

=2

34983Γ(n+ 1)Re(zz1d

)− 10097

34983(n+ 1)

=O( 1

n

)− 10097

34983(n+ 1)


Dow

nloa

ded

09/2

9/15

to 1

29.9

7.16

8.13

4. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p

Equation (A.1) hence solves to:

C3(n) =24

13(n+ 1) ln(n)

+

{−311

117+

24

13γ − 10097

34983

}(n+ 1)

+190

117+O

( 1

n

)≈24

13(n+ 1) ln(n)− 1.88(n+ 1) +

190

117+O

( 1

n

)Here, we have shown the exact derivations for thenumber of comparisons. The analysis for number ofswaps and cache misses are very similar to the aboveanalysis and hence they have been omitted.

A.3 Solving recurrences for median-of-3 1-pivot quicksort This algorithm uses two poniters totraverse through the array. Hence the total number ofcache misses during partitioning will be at most the to-tal number of elements scanned by these pointers di-vided by B. This gives rise to the following recurrencerelations:

CM(1/3)(n) ≤

n

B+

12

n(n− 1)(n− 2)

n−1∑i=0

((n− i)(i− 1)

(CM(1/3)(i) + CM(1/3)(n− i)))

≤ n

B+

12

n(n− 1)(n− 2)

n−1∑i=0

i(n− i− 1)CM(1/3)(i)

SP(1/3)(n) ≤

1 +12

n(n− 1)(n− 2)

n−1∑i=0

i(n− i− 1)SP(1/3)(i)

The recurrence for number of subproblems is self-explanatory. A minor point is that the above relationholds for n > M . For n ≤M , CM(1/3)(n) = 0. Both ofthe above recurrence relations can be written in a moregeneral form as,

fn = an+ b+6

n(n− 1)(n− 2)

n−1∑i=1

(i− 1)(n− i)fi

fn = an+ b+12

n(n− 1)(n− 2)

n−1∑i=0

i(n− i− 1)fi

where a = 1B and b = 0 for the 1st recurrence and

a = 0 and b = 1 for the second one. Multiplying by

n(n− 1)(n− 2) throughout gives:

n(n− 1)(n− 2)fn = an2(n− 1)(n− 2)

+ bn(n− 1)(n− 2) + 12

n−1∑i=0

i(n− i− 1)fi

Substituting n − 1 in the above equation and thensubtracting gives:

(n− 1)(n− 2)(n− 3)fn−1 = a(n− 1)2(n− 2)(n− 3)

+ b(n− 1)(n− 2)(n− 3)

+ 12

n−2∑i=0

i(n− i− 2)fi

n(n− 1)(n− 2)fn = (n− 1)(n− 2)(n− 3)fn−1

+ a(n− 1)(n− 2)(4n− 3)

+ 3b(n− 1)(n− 2) + 12

n−1∑i=0

ifi

The idea is to get rid of the summation by subtractingequations. Repeating the process gives the followingequation:

n(n− 1)(n− 2)fn = 3(n− 1)(n− 2)(n− 3)fn−1

− 3(n− 2)(n− 3)(n− 4)fn−2

+ (n− 3)(n− 4)(n− 5)fn−3

+ 24fn−3 + 6a(4n− 9) + 6b

Substituting values for a and b and solving theabove recurrence using standard linear algebra softwarewe get:

CM(1/3)(n) ≤ 12

7

(n+ 1

B

)ln

(n+ 1

M + 2

)+O

( nB

)SP(1/3)(n) ≤ 12

7

(n+ 1

M + 2

)− 2 +O

(1

n

)The analysis for 2-pivot quicksort is also very simi-

lar to the one shown above. We do not show it here.

A.4 Test Machine Specifications

CPU Intel(R) Core(TM)2 Quad CPU

Q9650 @ 3.00 GHz

L1d cache d: 32K i: 32K

L2 cache 6144K

Cache line size 64

Memory 4 × 2048 MB DDR @ 800 MHz

OS Kernel GNU/Linux 3.2.0-53-generic

Table A.4.1: Specifications of the system the tests wererun on (platform 1)


Dow

nloa

ded

09/2

9/15

to 1

29.9

7.16

8.13

4. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p

CPU Intel(R) Core(TM) i5-3570K

CPU @ 3.40GHz

L1d cache d: 32K i: 32K

L2 cache 256K

L3 cache 6144K

Cache line size 64

Memory 2 × 8192 MB DDR3 @ 1333 MHz

OS Kernel GNU/Linux 3.8.0-31-generic

Table A.4.2: Specifications of the system the additionalcomprehensive tests were run on (platform 2)

A.5 Experiments by Distribution Type Thesetests were run on the distribution types outlined in §4.2,on platform 1 and 2 using -O0 optimization flag.

Figure A.5.1: Legend for the following graphs

Figure A.5.2

Figure A.5.3

Figure A.5.4

Figure A.5.5


Dow

nloa

ded

09/2

9/15

to 1

29.9

7.16

8.13

4. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p

Figure A.5.6

Figure A.5.7

Figure A.5.8

Figure A.5.9

Figure A.5.10

Figure A.5.11


Dow

nloa

ded

09/2

9/15

to 1

29.9

7.16

8.13

4. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p

A.6 Experiments by Compiler OptimizationThese tests were run on the permutation distribution,on platform 1 and 2 using -O0, -O1, -O2, and -O3 opti-mization flags.

Figure A.6.1: Legend for the following graphs

Figure A.6.2

Figure A.6.3

Figure A.6.4

Figure A.6.5

Figure A.6.6


Dow

nloa

ded

09/2

9/15

to 1

29.9

7.16

8.13

4. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p

Figure A.6.7

Figure A.6.8

Figure A.6.9


Dow

nloa

ded

09/2

9/15

to 1

29.9

7.16

8.13

4. R

edis

trib

utio

n su

bjec

t to

SIA

M li

cens

e or

cop

yrig

ht; s

ee h

ttp://

ww

w.s

iam

.org

/jour

nals

/ojs

a.ph

p

Multi-Pivot Quicksort: Theory and Experimentsskushagr/multipivotQuicksort.pdf · the algorithm is...

Documents

Transcript of Multi-Pivot Quicksort: Theory and Experimentsskushagr/multipivotQuicksort.pdf · the algorithm is...