1 Divide & Conquer Algorithms Part 4. 2 Recursion Review A function that calls itself either...

1

Divide & Conquer Algorithms

Part 4

2

Recursion Review

A function that calls itself either directly or indirectly through another function

Recursive solutions involve: Base case – the function returns a solution Recursive case

• divide the problem into one or more simpler or smaller parts of the problem

• call the function (recursively) on each part, and• combine the solutions of the parts into a solution for the

problem.

3

Designing Algorithms

Incremental Design Most of the algorithms you have seen and

programmed Iterative

• An algorithmic technique where a function solves a problem by repeatedly working on successive parts of the problem

4

Designing Algorithms (cont)

Divide & Conquer Design Three steps

DIVIDE• Problem is divided into a number of subproblems

CONQUER• Solve the subproblems recursively• Base cases are simple enough to solve directly

COMBINE• The solutions to the subproblems are combined to solve

the original problem

5

Analyzing Divide & Conquer Algorithms Use a recurrence For small subproblem, solution takes constant

time DIVIDE step creates a subproblems, each of

size n/b D(n) time to divide into subproblems C(n) time to combine solutions of subproblems

otherwisenCnDaT

cnifnT

bn )()()(

)1()(

6

MergeSort

Requires additional memory space as a function of n Unlike Insertion Sort which sorts in place

• Requires only a constant amount of additional space

Sorts in (nlgn)

7

MergeSort

DIVIDE the n element sequence to be sorted into two subsequences of n/2 elements each

CONQUER (sort) the two subsequences recursively using merge sort The recursion stops when subproblem contains

only one element

COMBINE (merge) the two sorted subsequences to produce the sorted answer

8

MergeSort (Cont)… 19 9 62 74 94 13 90 48 …

19 9 62 74 94 13 90 48

19 9 62 74 94 13 90 48

19 9 62 74 94 13 90 48

9

MergeSort (Cont)… 9 13 19 48 62 74 90 94 …

9 19 62 74 13 48 90 94

9 19 62 74 13 94 48 90

19 9 62 74 94 13 90 48

10

MergeSort (cont)

To sort entire array: MergeSort( A, 1, length(A) )

MergeSort( A, p, r )1. if p < r2. q (p + r) / 23. MergeSort( A, p, q )4. MergeSort( A, q+1, r )5. Merge( A, p, q, r )

11

MergeSort (Cont)Merge( A, p, q, r )1. n1 q – p + 12. n2 r – q3. create arrays L[1..n1+1] and

R[1..n2+1]4. for i 1 to n15. L[ i ] A[p+i-1]6. for j 1 to n27. R[ i ] A[q+j]8. L[n1+1] 9. R[n2+1] 10. i 111. j 112. for k p to r13. if L[ i ] R[ j ]14. A[ k ] L[ i ]15. i = i + 116. else A[ k ] R[ j ]17. j = j + 1

Sentinel values

12

Analysis of MergeSort

Merge function: Line: 1- 2 (1) Line: 3 (n) Line: 4 – 7 i loop + j loop = (n) Line: 8 – 11 (1) Line: 12 (n) Line: 13 – 17 (1)

Total run time = (n)

13

Analysis of MergeSort (cont)

MergeSort function For simplicity, assume n is a power of 2 If n = 1, takes constant time If n > 1 then

• DIVIDE – Lines 1, 2 (1)– D(n) = (1)

• CONQUER – Lines 3, 4 2T(n/2)– two subproblems, each of size n/2

• COMBINE – Line 5 (n)– C(n) = (n)

14

Analysis of MergeSort (cont)

So the recurrence is:

• Note: D(n) + C(n) = (1) + (n) = (n)

The solution to the recurrence

1)()(2

1)1()(

2

nifnT

nifnTn

)lg()( nnnT

15

QuickSort

Worst time: (n2) Expected time: (nlgn)

Constants in the expected time are small Sorts in place

16

QuickSort (cont)

DIVIDE – Partition A[p..r] into two subarrays A[p..q-1] and A[q+1..r] such that each element of A[p..q-1] is A[q] each element of A[q+1..r]

Conquer – Sort the two subarrays by recursive calls to Quicksort

Combine – Since subarrays are sorted in place, they are already sorted

17

QuickSort (cont)

To sort entire array:

QuickSort( A, 1, length(A) )

QuickSort( A, p, r )

1. if p < r

2. q Partition( A, p, r )

3. QuickSort( A, p, q-1 )

4. QuickSort( A, q+1, r )

18

QuickSort (cont)Partition( A, p, r )

x A[ r ]i p – 1for j p to r-1

if A[ j ] xi i + 1Exchange( A[ i ], A[ j ] )

Exchange( A[ i+1 ], A[ r ] )return i+1

19

QuickSort (cont)

1 2 3 4 5 6 7 8

19 9 62 74 94 13 90 48p,i j r,x

1 2 3 4 5 6 7 8

19 9 62 74 94 13 90 48i p,j r,x

1 2 3 4 5 6 7 8

19 9 62 74 94 13 90 48p i j r,x

1 2 3 4 5 6 7 8

19 9 62 74 94 13 90 48p i j r,x

20

QuickSort (cont)1 2 3 4 5 6 7 8

19 9 62 74 94 13 90 48p i j r,x

1 2 3 4 5 6 7 8

19 9 62 74 94 13 90 48p i j r,x

1 2 3 4 5 6 7 8

19 9 13 74 94 62 90 48p i j r,x

1 2 3 4 5 6 7 8

19 9 13 48 94 62 90 74p i r

Return i+1

which is 4

21

Performance of QuickSort

Partition function’s running time - (n) Running time of QuickSort depends on

the balance of the partitions If balanced, QuickSort is asymptotically as

fast as MergeSort If unbalanced, it is asymptotically as bad

as Insertion Sort

22

Performance of QuickSort (cont)

Worst-case Partitioning Partitions always of size n-1 and 0 Occurs when array is already sorted Recurrence for the running time:

)(

)()1(

)()0()1()(

2n

nnT

nTnTnT

23

Performance of QuickSort (cont)cn

c(n-1)

c(n-2)

...

n

cn

c(n-1)

c(n-2)

c

Total: (n2)

)(2

)1( 2

11

nnn

iccin

i

n

i

24


Best-case Partitioning Partitions always of size n/2 and n/2-1 The recurrence for the running time:

)lg()(

)()(2)( 2

nnnT

nTnT n

Case 2 of Master Method

25

Performance of QuickSort (cont) Balanced Partitioning

Average-case is closer to best-case

Any split of constant proportionality (say 99 to 1) will have a running time of (nlgn)

• The recurrence will be

• Because it yields a recursion tree of depth

(lgn), where cost at each level is (n)

• See page 151 (new book) for picture– Or next slide

cnnTnTnT )()()( 1001

10099

26


c(n/100) c(99n/100)

cn

c(n/10000) c(99n/10000) c(99n/10000) c(9801n/10000)

T(1)

T(1)

T(1)T(1)

Total: (nlgn)

log100ncn

cn

cn

cn

cnlog100/99n

27


Intuition for the average case The behavior depends on the relative

ordering of the values• Not the values themselves

We will assume (for now) that all permutations are equally likely

Some splits will be balanced, and some will be unbalanced

28


In a recursion tree for an average-case, the “good” and “bad” splits are distributed randomly throughout the tree

For our example, suppose• Bad splits and good splits alternate• Good splits are best-case splits• Bad splits are worst-case splits• Boundary case (subarray size 0) has cost of 1

29


The (n-1) cost of the bad split can be absorbed into the (n) cost of the good split, and the resulting split is good

Thus the running time is (nlgn), but with a slightly larger constant

n

0 n - 1

(n-1)/2(n-1)/2-1

(n) n

(n-1)/2(n-1)/2

(n)

30

Randomized QuickSort

How do we increase the chance that all permutations are equally likely? Random Sampling

Don’t always use last element in subarray Swap it with a randomly chosen element from

the subarray Pivot now is equally likely to be any of the r – p + 1

elements We can now expect the split to be reasonably

well-balanced on average

31

Randomized QuickSort (cont)

Randomized-Partition( A, p, r )

1. i Random( p, r )

2. Exchange( A[ r ], A[ i ] )

3. return Partition( A, p, r )

Note that Partition( ) is same as before

32

Randomized QuickSort (cont)

Randomized-QuickSort( A, p, r )

1. if p < r

2. q Randomized-Partition( A, p, r )

3. Randomized-QuickSort( A, p, q-1 )

4. Randomized-QuickSort( A, q+1, r )

33

Analysis of QuickSort

A more rigorous analysis Begin with worst-case

We intuited that worst-case running time is (n2)

Use substitution method to show this is true

)())1()((max)(10

nqnTqTnTnq

34

Analysis of QuickSort (cont)

Guess: (n2) Show: for some c > 0 Substitute:

2)( cnnT

)()(

)()12()(

)())1((max

)())1((max)(

2

2

2

22

10

22

10

nnT

cn

nnccnnT

nqnqc

nqnccqnT

nq

nq

q2+(n-q-1)2 is max

at endpoints of range. Therefore it

is (n-1)2

= n2 – 2n +1

35


Problem 7.4-1 has you show that

Thus the worst-case running time of QuickSort is (n2)

)(

)())1()((max)(

2

10

n

nqnTqTnTnq

36


We will show that the upper-bound on expected running time is (nlgn)

We’ve already shown that the best-case running time is (nlgn)

Combined, these will give an expected running time of (nlgn)

37

Analysis of QuickSort (cont) Expected Running Time

Work done is dominated by Partition Each time a pivot is selected, this element

is never included in subsequent calls to QuickSort

• And the pivot is in its correct place in the array Therefore, at most n calls to Partition will

be made• Each call to Partition involves (1) work plus

the amount of work done in the for loop• Count the total number of times line 4 is

executed, we can bound the amount of time spent in the for loop

Line 4: if A[j] x

38


Lemma 7.1• Let X be the number of comparisons performed

in line 4 of Partition over the entire execution of QuickSort on an n-element array. Then the running time of QuickSort is (n + X)

• Proof: – There are n calls to Partition, each of which does

(1) work then executes the for loop (which includes line 4) some number of times

– Since the for loop executes line 4 during each iteration, X represents the number of iterations of the for loop along with the number of comparisons performed

– Therefore T(n) = (n (1) + X) = (n + X)

39


We need to compute X We do this by computing an overall bound

on the total number of comparisons• NOT by computing the number of comparisons

at each call to Partition Definitions:

• z1, z2, …, zn elements in the array• zi ith smallest element• set Zij = {zi, zi+1, …, zj}

40


When does the algorithm compare zi and zj?

Note: each pair of elements is compared at most once. Why?

Our analysis uses indicator random variables

41

Indicator Random Variables

They provide a convenient method for converting between probabilities and expectations

These are random variables which take on only the value 0 or 1, so they “indicate” whether or not something has happened

Indicator Random Variable I{A} is defined as: Given a sample space S and an event A I{A} = 1 if A occurs

0 if A does not occur

42

Indicator Random Variables (cont) A simple example

Determine the number of heads when flipping a coin

Sample space S = {H, T} Simple random variable Y

• These are random variables whose range contains only a finite number of elements

• In this case, it takes on the values H and T• Each with a probability of ½

XH is associated with the event Y = H• XH = I{Y = H} = 1 if Y = H

0 if Y = T

43

Indicator Random Variables (cont)

The expected number of heads in one flip is the expected value of our indicator variable XH

• Thus the expected number of heads in one flip is ½

21

21

21 )(0)(1

}Pr{0}Pr{1

}]{[][

TYHY

HYIEXE H

44


Lemma 5.1 Given a sample space S and an event A in

the sample space S, let XA = I{A}. Then E[XA] = Pr{A}

• See proof on page 95

To compute the number of heads in n coin flips Method 1 – compute the probability of

getting 0 heads, 1 heads, 2 heads, etc

45


Method 2• Let Xi be the indicator random variable

associated with the event “the ith flip is heads”• Let Yi be the random variable denoting the

outcome of the ith flip– Xi = I{Yi = H}

• Let X be the random variable denoting the total number of heads in the n coin flips

n

iiXX

1

46


• Take the expectation of both sides:

2

][

][

12

1

1

1

n

XE

XEXE

n

i

i

n

i

n

ii

By Lemma

5.1

47

(Back to) Analysis of QuickSort

We will use indicator random variables• Xij = I{zi is compared to zj}• it indicates whether the comparison took place

at any time during execution of QuickSort Since each pair is compared at most once:

1

1 1

n

i

n

ijijXX

48


Take expectation of both sides

}Pr{

][

][

1

1 1

1

1 1

1

1 1

n

i

n

ijji

n

i

n

ijij

n

i

n

ijij

ztocomparedisz

XE

XEXE

49


We still need to compute Pr{zi is compared to zj}

Start by thinking about when two items are not compared

• once a pivot x is chosen with zi < x < zj

– zi and zj will never be compared

• if zi is the first pivot chosen in Zij

– zi will be compared to every other element in Zij

• Similarly for zj

Thus, zi and zj are compared iff the first pivot from Zij is either zi or zj

50


What is the probability that this event occurs?

• Before a pivot has been chosen from Zij, all elements of Zij are in the same partition

• Each element of Zij is equally likely to be chosen as the first pivot

• the probability is

1

1

ij

51


Thus we have

1

2

1

1

1

1

}Pr{

}Pr{

}Pr{}Pr{

ij

ijij

Zfromchosenpivotfirstisz

Zfromchosenpivotfirstisz

Zfromchosenpivotfirstiszorzztocomparedisz

ijj

iji

ijjiji

Because the two events are

mutually exclusive

52

Analysis of QuickSort (cont) Combining the two boxed equations

)lg(

)(lg

2

1

2

1

2][

1

1

1

1 1

1

1 1

1

1 1

nn

n

k

k

ijXE

n

i

n

i

n

k

n

i

in

k

n

i

n

ij

Change of variables:k = j – i

Note the changes in the summation

variables

Bound on Harmonic Series:

)1(ln1

1

nHn

kkn

53


Thus, using Randomized-Partition, the expected running time of QuickSort is (nlgn)

54

Median and Order Statistics

In this section, we will study algorithms for finding the ith smallest element in a set of n elements

We will again use divide-and-conquer algorithms

55

The Selection Problem

Input: A set A of n (distinct) numbers and a number i, with 1 i n

Output: The element x A that is larger than exactly i – 1 other elements of A x is the ith smallest element

i = 1 minimum i = n maximum

56

The Selection Problem (cont)

A simple solution Sort A Return A[ i ] This is (nlgn)

57

Minimum and Maximum

Finding the minimum and maximum

Takes (n-1) comparisons ((n)) This is the best we can do and is optimal with

respect to the number of comparisons

MINIMUM(A)min A[1]for i 2 to length(A)

if min > A[ i ]min A[ i ]

return min

MAXIMUM(A)max A[1]for i 2 to length(A)

if max < A[ i ]max A[ i ]

return max

58

Minimum and Maximum (cont) Simultaneous minimum and maximum

Obvious solution is 2(n-1) comparisons But we can do better – namely The algorithm

• If n is odd, set max and min to first element• If n is even, compare first two elements and set

max, min• Process the remaining elements in pairs• Find the larger and the smaller of the pair• Compare the larger of the pair with the current

max• And the smaller of the pair with the current min

23 n

59

Minimum and Maximum (cont)

Total number of comparisons• If n is odd comparisons • If n is even

– 1 initial comparison– And 3(n – 2)/2 comparisons– For a total of 3n/2 – 2 comparisons

• In either case, total number of comparisons is at most

23 n

23 n

60

Selection in Expected Linear Time Goal: Select ith smallest element from

A[p..r]. Partition into A[p..q-1] and A[q+1..r] if i = q

then return A[q] If ith smallest element is in A[p..q-1]

then recurse on A[p..q-1] else recurse on A[q+1..r]

61

Selection in Expected Linear Time (cont)Randomized-Select(A, p, r, i) 1 if p = r 2 return A[p] 3 q Randomized-Partition(A, p, r) 4 k q - p + 1 //Why this statement?5 if i = k 6 return A[q] 7 else if i < k8 return Randomized-Partition(A, p, q-1, i)9 else10 return Randomized-Partition(A, q+1, r, i-k)

62

i = 6

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

18 45 58 47 6 97 43 10 85 70 21 49 56 51 66

i = 6 q = 12 k = 12

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

18 45 58 47 6 43 10 21 49 56 51 66 70 97 85

i = 6 q = 12 k = 12

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

18 45 47 6 43 10 21 49 51 56 58 66 70 97 85

i = 6 q = 9 k = 9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

18 6 10 21 43 47 49 45 51 56 58 66 70 97 85

i = 6/2 q = 4 k = 4

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

18 6 10 21 43 45 47 49 51 56 58 66 70 97 85

i = 2 q = 7 k = 3

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

18 6 10 21 43 45 47 49 51 56 58 66 70 97 85

i = 2 q = 6 k = 2

63

Analysis of Selection Algorithm

Worst-case running time is (n2) Partition takes (n) If we always partition around the largest

remaining element, we reduce the partition-size by one element each time

What is best-case?

64

Analysis of Selection Algorithm (cont) Average Case

Average-case running time is (n) The time required is the random variable

T(n) We want an upper bound on E[T(n)] In Randomized-Partition, all elements are

equally likely to be the pivot

65

Analysis of Selection Algorithm (cont)

So, for each k such that 1 k n, subarray A[p..q] has k elements

• All the pivot• with probability 1/n

For k = 1, 2, …, n we define indicator random variables Xk where

• Xk = I{the subarray A[p..q] has exactly k elements}

So, E[Xk] = 1/n

66


When we choose the pivot element (which ends up in A[q]) we do not know what will happen next

• Do we return with the ith element (k = i)?• Do we recurse on A[p..q-1]?• Do we recurse on A[q+1..r]?

Decision depends on i in relation to k We will find the upper-bound on the

average case by assuming that the ith element is always in the larger partition

67


Now, Xk = 1 for just one value of k, 0 for all others

When Xk = 1, the two subarrays have sizes k – 1 and n – k

Hence the recurrence:

))()),1(max((

))()),1(max(()(

1

1

nknkTX

nknkTXnT

n

kk

n

kk

68


Taking the expected values:

)()),1(max(

)()),1(max(

)()),1(max(

)()),1(max()]([

1

1

1

1

1

nknkTE

nknkTEXE

nknkTXE

nknkTXEnTE

n

kn

n

kk

n

kk

n

kk

69


Looking at the expression max(k-1, n-k)

• If n is even, each term from appears twice in the summation

• If n is odd, each term from appears twice and appears once in the summation

2

21),1max(n

n

kifkn

kifkknk

)1()( 2 nTtoT n

)1()( 2 nTtoT n

)( 2nT

70


Thus we have

We use substitution to solve the recurrence Note: T(1) = (1) for n less than some

constant Assume that T(n) cn for some constant c

that satisfies the initial conditions of the recurrence

)()]([

2)]([

1

2

nkTEn

nTEn

k n

71


Using this inductive hypothesis

an

nn

n

c

annn

n

c

ankkn

c

anckn

nTE

nn

nn

k

n

n

k

n

n

2

12

2

)1(2

2

1

2

)1(2

2

2)]([

22

22

1

1

1

1

1

2

2

72


anccn

cn

anccn

ann

nc

annn

n

c

annn

n

c nn

24

24

3

2

2

1

4

3

224

3

2

2

2

2

2

23

42 2

73


To complete the proof, we need to show that for sufficiently large n, this last expression is at most cn

• i.e.

ac

c

an

ca

cn

anccn

c

c

4

2

24

024

4

2

As long as we choose the constant c so that c/4 – a > 0 (i.e., c > 4a), we can divide both sides by

c/4 – a

74


Thus, if we assume that T(n) = (1) for , we have T(n) = (n) ac

cn 42

75

Selection in Worst-Case Linear Time “Median of Medians” algorithm It guarantees a good split when array is

partitioned Partition is modified so that the pivot now

becomes an input parameter The algorithm:

If n = 1 return A[n]

76

Selection in Worst-Case Linear Time (cont)

1. Divide the n elements of the input array into n/5 groups of 5 elements each and at most one group of (n mod 5) elements

2. Find the median of each of the n/5 groups by using insertion sort to sort list and then pick the 3rd element of each group

3. Use Select recursively to find the median x of the n/5 medians found in step 2. – If even number of medians, choose lower median

77

Selection in Worst-Case Linear Time (cont)

4. Partition the input array around the “median of medians” x using the modified version of Partition. Let k be one more than the number of elements on the low side of the partition, so that x is the kth smallest element and there are n – k elements on the high side of the partition

5. if i = k, then return x. Otherwise, use Select recursively to find the ith smallest element on the low side if i < k, or the (i – k)th smallest element on the high side if i > k

78

Selection in Worst-Case Linear Time (cont) Example of “Median of Medians”

Input Array A[1..100] Step 1: 25 groups of 5 Step 2: We get 25 medians Step 3: Step 1: Using the 25 medians we get 5

groups of 5Step 2: We get 5 mediansStep 3: Step 1: Using the 5 medians, we

get 1 group of 5 Step 2: We get 1 median

Step 4: Partition A around the median

79

Analyzing “Median of Medians”

Analyzing “median of medians” The following diagram might be helpful:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

10 975 939 701 582 468 766 312 809 673 124 689 597 759 39 612 180 304 380 549 35 858 132 349

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

10 582 701 939 975 312 468 673 766 809 39 124 597 689 759 180 304 380 549 612 35 132 349 858

80

Analyzing “Median of Medians” (cont)

First, we need to put a lower bound on how many elements are greater than x

How many of the medians are greater than x?

• At least half of the medians from the groups – Why “at least half?”

• medians are greater than x

10521 nn

5n

81

The two discarded groups


Each of these medians contribute at least 3 elements greater than x except for two groups

• The group that contains x– contributes only 2 elements greater than x

• The group that has less than 5 elements

So the total number of elements > x is at least:

623 103

221 nn

82


Similarly, there are at least elements smaller than x

Thus, in the worst case, for Step 5 Select is called recursively on the largest partition

• The largest partition has at most elements

6103 n

6107 n

The size of the array minus the number of elements in the smaller partition

6103 nn

83


Developing the recurrence:• Step 1 takes (n) time• Step 2 takes (n) time

(n) calls to Insertion Sort on sets of size (1)

• Step 3 takes • Step 4 takes (n) time• Step 5 takes at most

5nT

6107 nT

84


So the recurrence is

Now use substitution to solve• Assume T(n) cn for some suitable large

constant c and all n ???• Also pick a constant a such that the function

described by the (n) term is bounded above by an for all n > 0

???)(6

???)1()(

107

5

nifnTT

nifnTnn

85

Comes from removing the


70

07

7

7

6

6)(

7010

10

10

109

107

5

107

5

nwhenc

anc

anccn

anc

ancc

anccnT

nan

cn

cn

cn

cncn

nn

Which is at most cn if

If n = 70, then this inequality is undefined

86


We assume that n 71, so Choosing c 710a will satisfy the

inequality on the previous slide You could choose any constant > 70 to be

the base case constant Thus, the selection problem can be

solved in the worst-case in linear time

7170 nn

87

Review of Sorts

Review of sorts seen so far Insertion Sort

• Easy to code• Fast on small inputs (less than ~50)• Fast on nearly sorted inputs• Stable(n) best case (sorted list)(n2) average case(n2) worst case (reverse sorted list)

88

Review of Sorts

• Stable means that numbers with the same value appear in the output array in the same order as they do in the input array.

• That is, ties between two numbers are broken by the rule that whichever number appears first in the input array appears first in the output array.

• Normally, the property of stability is important only when satellite data are carried around with the element being sorted.

89

Review of Sorts (cont)

MergeSort• Divide and Conquer algorithm• Doesn’t sort in place• Requires memory as a function of n• Stable(nlgn) best case(nlgn) average case(nlgn) worst case

90


QuickSort• Divide and Conquer algorithm

– No merge step needed

• Small constants• Fast in practice• Not stable(nlgn) best case(nlgn) average case(n2) worst case

91


Several of these algorithms sort in (nlgn) time MergeSort in worst case QuickSort on average

On some input we can achieve (nlgn) time for each of these algorithms

The sorted order they determine is based only on comparisons between the input elements

They are called comparison sorts

92


Other techniques for sorting exist, such as Linear Sorting which is not based on comparisons

Usually with some restrictions or assumptions on input elements

Linear Sorting techniques include: Counting Sort Radix Sort Bucket Sort

93

Lower Bounds for Sorting

In general, assuming unique inputs, comparison sorts are expressed in terms of

comparisons. are

equivalent in learning about the order of ai and aj

What is the best we can do on the worst case type of input?

What is the best worst-case running time?

ji aa jijijiji aaaaaaaa ,,,

94

The Decision-Tree Model

1:2

2:3

2:3

1:3

1:31,2,3 2,1,3

1,3,2 3,1,2 2,3,1 3,2,1

• n = 3• input: a1,a2,a3• # possible outputs = 3! = 6• Each possible output is a leaf

95

Analysis of Decision-Tree Model

Worst Case Comparisons is equal to height of decision tree

Lower bound on the worst case running time is the lower bound on the height of the decision tree.

Note that the number of leaves in the decision tree n!, where n = number elements in the input sequence

96

Theorem 8.1

Any comparison sort algorithm requires (nlgn) comparisons in the worst case

Proof: Consider a decision tree of height h that

sorts n elements Since there are n! permutations of n

elements, each permutation representing a distinct sorted order, the tree must have at least n! leaves

97

Theorem 8.1 (cont)

A binary tree of height h has at most 2h leaves

The best possible worst case running time for comparison sorts is thus (nlgn)

Mergesort, which is O(nlgn), is asymptotically optimal

)lg(

)!lg(

2!

nn

nh

n h

By equation 3.18

98

Sorting in Linear Time

But the name of this chapter is “Sorting in Linear Time”

How can we do better? CountingSort RadixSort BucketSort

99

Counting Sort No comparisons between elements But depends on assumptions about the

values being sorted Each of the n input elements is an integer

in the range 0 to k When k = (n), the sort runs in (n) time The algorithm:

• Input: A[1..n] where A[ j ] {0, 1, …, k}• Output: B[1..n], sorted

– Notice: elements are not sorted in place

• Also: C[1..k] for auxiliary storage

100

Counting Sort (cont)Counting-Sort(A, B, k)1. for i 0 to k2. C[i] 03. for j 1 to Length(A)4. C[A[j]] C[A[j]] + 15. // C[i] now contains the number of elements = i6. for i 1 to k7. C[i] C[i] + C[i-1]8. // C[i] now contains the number of elements i9. for j Length(A) downto 110. B[C[A[j]]] A[j]11. C[A[j]] C[A[j]] – 1

k+2k+1n+1

n

k+1k

n+1nn

----------------------------------------------------------------------------

-------------------------------------------------

---------------------------------------------------------------

-----------------------------------------------

-----------------------

101

Input: 1 2 3 4 5 6 7 8

A 2 5 3 0 2 3 0 3

After lines 3-4: 0 1 2 3 4 5

C 2 0 2 3 0 1

After lines 6-7: 0 1 2 3 4 5

C 2 2 4 7 7 8

First 3 iterations of lines 9-11: 1 2 3 4 5 6 7 8

B 3

0 1 2 3 4 5

C 2 2 4 6 7 8

1 2 3 4 5 6 7 8

B 0 3

0 1 2 3 4 5

C 1 2 4 6 7 8

1 2 3 4 5 6 7 8

B 0 3 3

0 1 2 3 4 5

C 1 2 4 5 7 8

Output: 1 2 3 4 5 6 7 8

B 0 0 2 2 3 3 3 5

102

Analysis of CountingSort:

(nlgn) does not apply because CountingSort isn’t a comparison sort

CountingSort is stable because 4th loop goes from n downto 1

)(

then)( if and)(

45)(

n

nkkn

knnT

103

Radix Sort

This sort was originally used to sort computer punch-card decks.

It is currently used for multi-key sorts for example: year/month/day

Consider each digit of the number as a separate key

104

Radix Sort (cont)

Idea 1: Sort on most significant digit n, then sort on digit n-1, etc.

Problem: For old sorters, sort into 10 bins, but subsequent recursive sorts require all 10 bins Operator must store the other 9 piles

Idea 2: Sort on least significant digit, then sort on next least significant digit, etc.

105

Radix Sort (cont)3 2 9 7 2 0 7 2 0 3 2 9

4 5 7 3 5 5 3 2 9 3 5 5

6 5 7 4 3 6 4 3 6 4 3 6

8 3 9 4 5 7 8 3 9 4 5 7

4 3 6 6 5 7 3 5 5 6 5 7

7 2 0 3 2 9 4 5 7 7 2 0

3 5 5 8 3 9 6 5 7 8 3 9

Radix-Sort(A, d)for i 1 to d

use a stable sort to sort array A on digit d

106

Radix Sort (cont)

Radix-n: another way of indicating the sort used Implies each digit can differentiate among

n different symbols. In the previous case we assumed radix-10.

This is why the name Radix Sort is given

107

Analysis of RadixSort

If each digit is in the range 1 to k (or 0 to k-1), use CountingSort for each pass

Each pass over a digit is (n + k) For d digits (d(n + k)) If d is a constant and k = (n) T(n) = (n)

108

Proof by induction that RadixSort works Base Case: d = 1

since only one digit, sorting by that digit sorts the list

Inductive Step: Assume it holds for d – 1digits Show: that it works for d digits

A radix sort of d digits is the same as a radix sort of d-1 digits followed by a sort on digit d

By our inductive hypothesis, the sort on d-1 digits works

• and the digits are in order according to their low-order d-1 digits

109

Proof by induction that RadixSort works (cont)

• The sort on digit d will order the elements by their dth digit

• Consider two elements a and b, with dth digits ad and bd respectively

• If ad < bd, the sort will put a before b, which is correct, since a < b regardless of their low order digits

110

Proof by induction that RadixSort works (cont)

• If ad > bd, the sort will put a after b, which is correct, since a > b regardless of their low order digits

• if ad = bd, the sort will leave a and b in the same order they were in, because it is stable. But that order is already correct, since the correct order of a and b is determined by the low-order d-1 digits when their dth digits are equal

111

Radix Sort Example Show how n integers in the range 1 to

n2 can be sorted in (n) time Subtract 1 from each number

• So they’re in the range 0 to n2-1• We’ll add the one back after they are sorted

Use a radix-n sort Each digit requires n symbols, and lognn2

digits are needed (d=2, k=n). • i.e., treat the numbers as 2-digit numbers in

radix n• Each digit ranges from 0 to n-1

112

Radix Sort Example (cont)

Sort these 2-digit numbers with radix sort There are 2 calls to counting sort

• For (2(n + n)) = (n) time

The passes to subtract 1 and add 1each take (n) time

Hence, total running time of (n)

113

Radix Sort Example (cont)

Take 15 integers in the range of 1 to 152 (225)

Subtract one from each number to get the range 0 to 224

We will use a radix-15 sort We will need 2 digits of n-many symbols

Each digit ranges from 0 to n-1 See Handout

114

Bucket Sort

Bucket Sort assumes that the input values are uniformly distributed over the range [0,1), 0 x < 1

Procedure: Divide inputs into n equal-sized subintervals

(buckets) over the range [0,1). Sort each bucket and concatenate the buckets.

T(n) = (n)

115

Bucket Sort (cont)A B

1 .78 0

2 .17 1 .12 .17

3 .39 2 .21 .23 .26

4 .26 3 .39

5 .72 4

6 .94 5

7 .21 6 .68

8 .12 7 .72 .78

9 .23 8

10 .68 9 .94

116

Bucket Sort (cont)

Bucket-Sort(A)1. n Length(A) 12. for i 1 to n n+13. insert A[i] into list B[nA[i]] n4. for i 0 to n – 1 n+15. sort list B[i] with InsertionSort

n*T(n)6. concatenate the lists B[0], B[1], … ,B[n-1] n

together in order

117

Analysis of Bucket Sort

All lines except Line 5 take (n) time What is the cost of the calls to

InsertionSort? Let ni be the random variable denoting the

number of elements in bucket B[i] And we know that InsertionSort runs in

(n2) time

118

Analysis of Bucket Sort (cont)

1,,1,012

)(

)(

)()]([

)()(

2

1

0

2

1

0

2

1

0

2

1

0

2

ninn

nn

nn

nnnT

nnnT

i

n

ii

n

ii

n

ii

n

ii

for E

that claim We

E

E

EE

By equation c.21

EQ 8.2

EQ 8.1

119


To prove the above equation, define indicator random variables

• Xij = I{A[j] falls into bucket i]– for i = 0, 1, …, n-1– for j = 1, 2, …, n

Thus…

n

jiji Xn

1

120


: Ecompute To 2in

njjk

nkikij

n

jij

njjk

nkikij

n

jij

n

j

n

kikij

n

jiji

XXX

XXX

XX

Xn

1 11

2

1 11

2

1 1

2

1

2

EE

E

E

EE

EQ 8.3

121311131312111213111211213

212

211

213121311131312

212111213111211

211

131211131211

XXXXXXXXXXXXXXX

XXXXXXXXXXXXXXX

XXXXXX

121


Evaluate the two summations separately

n

nnin1

12122 101E

21

11

EEE

n

nn

ikijikij XXXX

Lemma 5.1E[XA] = Pr{A}

122

Analysis of Bucket Sort (cont) Substitute these into EQ 8.3

n

n

nn

nnn

n

nnn

njjk

nk

n

ji

12

11

11

1

11E

2

1 12

1

2

Which proves EQ 8.2

123

Thus, when input is drawn from a

uniform distribution,

BucketSort runs in linear time


Use the expected value in EQ 8.1

)(

n

12)(

n

1-2)(

E)()(

1

0

1

0

2

n

nn

n

nnnT

n

i

n

ii

124

Strassen’s Algorithm for Matrix Multiplication It is another divide-and-conquer

algorithm The naïve (straightforward) method for

multiplying two n x n matrices is (n3) n3 multiplications n2(n-1) additions

Strassen’s algorithm runs in (nlg7) = (n2.81)

125

Strassen’s Algorithm for Matrix Multiplication (cont) We wish to compute C = AB

All are n x n matrices Assume n is a power of 2

Divide each of A, B, and C into four n/2 x n/2 matrices, rewriting C = AB

hg

fe

dc

ba

ut

sr

126

Strassen’s Algorithm for Matrix Multiplication (cont) To compute each of the sub-matrices of

C: r = ae + bg s = af + bh t = ce + dg u = cf + dh

These are the eight “essential terms”

127

Strassen’s Algorithm for Matrix Multiplication (cont) To solve C

We need to make 8 recursive callsto “multiply” (with input size n/2)

And then perform 4 matrix additions Therefore,

No better than the “naïve” method

)(

)()(8)(3

22

n

nTnT n

The addition of

twon x n

matrices takes (n2)

time

128

Strassen’s Algorithm for Matrix Multiplication (cont) Strassen was the first to discover an

asymptotically faster algorithm (in the 1970s)

)(

)(

)()(7)(

81.2

7lg

22

n

n

nTnT n

By case 1 of the Master

Method

129

Strassen’s Algorithm for Matrix Multiplication (cont) His method

Divide the input matrices into n/2 x n/2 submatrices

Using (n2) scalar additions and subtractions, compute 14 matrices A1, B1, A2, B2, …, A7, B7, each of which is n/2 x n/2

Recursively compute the seven matrix products Pi = AiBi for i = 1, 2, …, 7

130

Strassen’s Algorithm for Matrix Multiplication (cont)

Compute the desired submatrices r, s, t, u of the result matrix C by adding and/or subtracting various combinations of the Pi matrices, using only (n2) scalar additions and subtractions

131


The Pi matrices:

fecaBAP

hgdbBAP

hedaBAP

egdBAP

edcBAP

hbaBAP

hfaBAP

777

666

555

444

333

222

111

Notice that each of these 7 Pi matrices

requires ONE multiplication only

132


Use the Pi matrices to compute:

7315

43

21

6245

PPPPu

PPt

PPs

PPPPr

133

Strassen’s Algorithm for Matrix Multiplication (cont) The simple recurrence to compute C is:

)(18)(7)( 81.2222 nTnT nn

The cost of the 7

recursive calls

The cost of the 18 n/2 x n/2 matrix additions and subtractions

134

An Example of Strassen’s Algorithm

Let C, A, and B be 2x2 matrices with

Therefore:

2221

1211

2221

1211

2221

1211 ,,bb

bbB

aa

aaA

cc

ccC

22

21

12

11

ad

ac

ab

aa

22

21

12

11

bh

bg

bf

be

22

21

12

11

cu

ct

cs

cr

135

An Example of Strassen’s Algorithm (cont)

121121117

222122126

221122115

1121224

1122213

2212112

2212111

bbaap

bbaap

bbaap

bbap

baap

baap

bbap

136

An Example of Strassen’s Algorithm (cont)

731522

4321

2112

624511

ppppc

ppc

ppc

ppppc

137

Discussion of Strassen’s Algorithm Strassen’s algorithm is numerically

stable, but not always as accurate as the naïve algorithm

It requires a significant amount of workspace to store intermediate results

The constant factor in the running time is greater than that in the naive algorithm

138

Discussion of Strassen’s Algorithm (cont) When matrices are sparse, there are

faster algorithms designed for sparse matrices

A potential attraction is its natural block decomposition; instead of continuing the recursion until n = 1, one could instead use a more standard matrix multiply as soon as n is small enough

139

Final Notes on Strassen’s Algorithm What if n is not a power of 2?

And you want to compute the product of two n x n matrices A and B

Set

Consider the following two matrices

nnn

nn n

2

2 2log

that such two of

power a is that so

nn

00

0

00

0 BB

AA

and

140

Final Notes on Strassen’s Algorithm (cont)

Since is a power of two, Strassen’s algorithm can be used to multiply

Since , the resulting algorithm still uses operations to multiply A and B together

BA

nnBA

is submatrix

left top whosematrix a is

n

BA

by nn 2

7log2n

141

An Example

0000

090114138

0546984

0182430

0000

0123

0456

0789

0000

0987

0654

0321

4

90114138

546984

182430

123

456

789

987

654

321

3

C

BA

n

CBA

nBAC

therefore and

then If

then and

where Find

142

Polynomials and FFT

To add two polynomials of degree n takes (n) time

To multiply two polynomials of degree n takes (n2) time

Using the Fast Fourier Transform, we can reduce the time of multiplying to (nlgn) time

143

Polynomials

Coefficients

Degree k The highest nonzero coefficient is ak

Degree-bound Any integer strictly greater than the degree of a

polynomial• Any polynomial with degree 0 to n-1 has a degree-bound

n

1

0

)(n

j

jj xaxA

110 ,,, naaa

144

Polynomials (cont)

Polynomial addition

If A(x) and B(x) are polynomials of degree-bound n, then A(x) + B(x) = C(x) is also of degree-bound n

1

0

1

0

1

0

)()()(n

j

jj

n

j

jj

n

j

jjj xbxaxba

xBxAxC

145

Polynomials (cont)

Polynomial multiplication

degree(C) = degree(A) + degree(B) • which implies• Degree-bound (C) = degree-bound(A) + degree-bound(B) – 1

• <= degree-bound(A) + degree-bound(B)

1

0

1

0

22

0 0

)()()(n

j

jj

n

j

jj

n

j

jj

kkjk xbxaxba

xBxAxC

146

Review of Complex Numbers

Have the form a + bi where i = 1

22

2222

)()(

)()(

)()(

babia

dc

iadbc

dc

bdac

dic

bia

ibcadbdacdicbia

idbcadicbia

idbcadicbia

:Value Absolute

:Division

:tionMultiplica

:Difference

:Sum

147

Coefficient Representation of Polynomials Is a vector of coefficients

Evaluating the polynomial A(x0) using Horner’s rule:

Takes time (n)

),,,( 110 naaaa

)))((()( 10201000 naxaxaxaxAy 0i nwhile i >= 0

y ai + x * yi i - 1

148

Coefficient Representation of Polynomials (cont) Adding two polynomials

Simply add the two coefficient vectors a and b representing A(x) and B(x)

Takes time (n)

149

Coefficient Representation of Polynomials (cont) Multiplying two polynomials

Multiply each element of coefficient vector a by each element in coefficient vector b giving coefficient vector c

c is called the convolution of vectors a and b

• It is denoted by c = a b

Takes time (n2)

150

Coefficient Representation of Polynomials (cont) Multiplying polynomials and computing

convolutions is done often in practice We need efficient algorithms for them

151

Point-Value Representation of Polynomials

A point-value representation of a polynomial A(x) of degree-bound n is a set of n point-value pairs

such that all of the xk are distinct and

yk = A(xk)

for k = 0, 1, …, n-1

111100 ,,,,,, nn yxyxyx

152

Point-Value Representation of Polynomials (cont) Uniquely specified by knowing A(x) at n different values of x

)(),,),,),, 110 kkn xAyyyy where,(x(x(x 1-n10

x

y = A(x)

xj

yj

153

Point-Value Representation of Polynomials (cont) To compute the point-value

representation for a polynomial in coefficient form Select n distinct points x0, x1, …, xn-1

Evaluate A(xk) for k = 0, 1, …, n-1

Using Horner’s method this takes (n2) time

154

Point-Value Representation of Polynomials (cont) Interpolation

Taking the point-value representation and determining the coefficient form of the polynomial

• This is the same as “evaluating” the polynomial in point-value form

155

Point-Value Representation of Polynomials (cont) Theorem 30.1 (Uniqueness of an

interpolating polynomial) For any set

of n point-value pairs such that all the xk

values are distinct, there is a unique polynomial A(x) of degree-bound n such that yk = A(xk) for k = 0, 1, …, n-1

Proof – but first a review of some matrix operations

111100 ,,,,,, nn yxyxyx

156

Review of Matrix Operations

n x n identity matrix (In) =

Inverse of an n x n matrix A is the n x n matrix A-1 (if it exists), such that

100

010

001

AAIAA n11

157

Review of Matrix Operations (cont) If a matrix has an inverse, it is called

Invertible

Nonsingular

If a matrix has no inverse, it is called Noninvertible

Singular

When a matrix inverse exists, it is unique

158

Review of Matrix Operations (cont) Minor

The ijth minor of an n x n matrix A, for n > 1, is the (n-1) x (n-1) matrix A[ i j ]

obtained by deleting the ith row and jth column of A

159

Review of Matrix Operations (cont) Determinant of an n x n matrix A

Defined recursively in terms of its minors by

1)det()1(

1)det(

1]1[1

1

11

nAa

naAn

jjj

j if

if

160

Review of Matrix Operations (cont) To easily find the determinant of a 2 x 2

matrix:

211222112221

1211 aaaaaa

aaA

161

Review of Matrix Operations (cont) To find the determinant of a 3 x 3 matrix

223113322113233112332112233211332211

223132211323313321122332332211

3231

222113

3331

232112

3332

232211

333231

232221

131211

aaaaaaaaaaaaaaaaaa

aaaaaaaaaaaaaaa

aa

aaa

aa

aaa

aa

aaa

aaa

aaa

aaa

A

162

Review of Matrix Operations (cont) Example: Find the determinant of:

2

)4)(1()2)(3(

)4()2(

)det()1()det()1()det(

24

13

1211

]12[1221

]11[1111

aa

AaAaA

A

163

Review of Matrix Operations (cont) Same example, different notation

2

46

)1)(4()2)(3(24

13

24

13

A

A

164

Review of Matrix Operations (cont)

)det()1()det()1(

)det()1()det()1(

)det()1()det()1(

)det()1(

)det()1()det()1()det(

101

312

121

312221

322111

13

312321

332111

12

322321

332211

11

]13[1331

]12[1221

]11[1111

aaaaa

aaaaa

aaaaa

Aa

AaAaA

A

Example: Find the determinant of:

165

Review of Matrix Operations (cont)

10

1101

1)1(521)1(

)1)(1()0)(2()1(

)1)(3()1)(2(2)0)(3()1)(1()1(

)1()0(

)1()1()0()1(

222113

232112232211

aaa

aaaaaa

166

Review of Matrix Operations (cont) Same example, different notation

10

1101

)1)(1()5)(2()1)(1(

)1)(1()0)(2()1()3)(1()1)(2()2()3)(0()1)(1()1(

01

12)1(

11

32)2(

10

31)1(

101

312

121

,

101

312

121

AA

167

Review of Matrix Operations (cont) Theorem 28.5

An n x n matrix A is singular (is noninvertible) if and only if det(A) = 0

Conversely, an n x n matrix A is nonsingular (is invertible) if and only if det(A) 0

168

Review of Matrix Operations (cont) Theorem: Square Systems with Exactly One

Solution If A is an invertible square matrix, then the system

of equations given by AX = B has the solution X = A-1B

Proof:

BAX

BAXI

BAAXA

BAX

n

1

1

11

169

Back To Point-Value Representation of Polynomials Theorem 30.1 (Uniqueness of an

interpolating polynomial) For any set

of n point-value pairs such that all the xk

values are distinct, there is a unique polynomial A(x) of degree-bound n such that yk = A(xk) for k = 0, 1, …, n-1

111100 ,,,,,, nn yxyxyx

170

Point-Value Representation of Polynomials (cont) Proof of Theorem 30.1:

yk = A(xk) for k = 0, 1, …, n-1 is equivalent

to the matrix equation

1

1

0

1

1

0

11

211

11

211

10

200

1

1

1

nnnnnn

n

n

y

y

y

a

a

a

xxx

xxx

xxx

171

Point-Value Representation of Polynomials (cont) The Vandermonde matrix

(shown on previous slide), has the determinant

And is therefore invertible if the xk are

distinct. Thus, we can solve for vector a:

10

)(nkj

jk xx

),...,,( 110 nxxxV

yxxxVa n1

110 ),...,,(

See problem 28.1-11

172

Point-Value Representation of Polynomials (cont) To interpolate by solving the set of

linear equations using the standard method of Gaussian Elimination takes (n3) time

A faster algorithm for n-point interpolation based on Lagrange’s formula

173

Point-Value Representation of Polynomials (cont)

To compute the coefficients takes time (n2)

1

0

)(n

kkj

jk

kjj

k xx

xx

yxA

174

Point-Value Representation of Polynomials (cont) An example using Lagrange’s Formula

345

20725.155875.25

29719875.249375.38875.3

8

1657

8

1242

8

207

2971984

99

8

744

8

310

8

31

8

86207

4

12899

8

241031

)46)(26(

)4)(2(207

)64)(24(

)6)(2(99

)62)(42(

)6)(4(31)(

:)}207,6(),99,4(),31,2{(

2

2

22

2

22

222

xx

xx

xxxx

xx

xxxx

xxxxxx

xxxxxxxA

:Given

Why is thisalgorithm only(n2) time?

175

Point-Value Representation of Polynomials (cont) Adding two polynomials:

Given point-value representation for two polynomials A and B:

Then the point-value representation for C:

The time is (n)

111100

111100

,,,,,,

,,,,,,

nn

nn

yxyxyx

yxyxyx

111111000 ,,,,,, nnn yyxyyxyyx

The x’s are the same!

176

Point-Value Representation of Polynomials (cont) Multiplying two polynomials:

We just need to multiply the corresponding values of A and B to get the point-value representation of C

A problem arises because the degree-bound of C (2n) is the sum of the degree-bounds for A and B (n)

• And we need 2n point-value pairs to represent C

177

Point-Value Representation of Polynomials (cont)

So, we must extend A and B to include 2n point-value pairs each

Then the point-value representation for C:

The time is (n)

12121100

12121100

,,,,,,

,,,,,,

nn

nn

yxyxyx

yxyxyx

121212111000 ,,,,,, nnn yyxyyxyyx

178

Best of Both Worlds

Can we get "fast" multiplication and evaluation?

coefficient

Representation

O(n2)

Multiplication

O(n)

Evaluation

point-value O(n) O(n2)

FFT O(n log n) O(n log n)

179

Best of Both Worlds (cont)

Yes! Convert back and forth between two representations

110

110

,,,

,,,

n

n

bbb

aaa

2-2n10 ,,,c cc

)},(,),,{(

)},(,),,{(

121200

121200

nn

nn

yxyx

yxyx

)},(),,{( 12120,0 nn yxyx O(n)

point-value multiplication

O(n2)

coefficient multiplication

O(n log n)evaluationFFT

interpolationinverse FFT

O(n log n)

180

Best of Both Worlds (cont)

Key idea: choose {x0, x1, . . . , xn-1 } to

make computation easier This will allow us to convert between

representations in (nlgn) time We will use “complex roots of unity” as

the evaluation points

181

Complex Roots of Unity

A complex nth root of unity is a complex number such that n = 1

There are exactly n many nth roots of unity

1,,1,0/2 nke nik for

182

Complex Roots of Unity (cont)

The principal nth root of unity

All the complex nth roots of unity are powers of n

For each term,

nin e /2

1210 ,,, nnnnn

1nk

n

183


Definition of the exponential of a complex number:

And

We get an nth root of unity by

)2sin()2cos(1

)sin()cos(

)2sin()2cos(1

2

ie

uiue

i

i

iu therefore

and

)sin()cos(1 22/21

nnni ien

)sin()cos( uiue iu

184


But 1 can be written using different arguments as follows:

etc.

)6sin()6cos(1

)4sin()4cos(1

)2sin()2cos(1

)0sin()0cos(1

6

4

2

0

ie

ie

ie

ie

i

i

i

i

185


Hence dividing the argument in each case by n gives the following nth roots of unity

etc.

)sin()cos(1

)sin()cos(1

)sin()cos(1

)sin()cos(1

66

44

22

00

61

41

21

01

nn

nn

nn

nn

ie

ie

ie

ie

ni

n

ni

n

ni

n

ni

n

186


An example: Find the cube roots of unity

)31()sin()cos(

)31()sin()cos(

1)0sin()0cos(

2,1,0

21

34

343/222

3

21

32

323/121

3

3/0203

3/2

iie

iie

ie

ke

i

i

i

ik

for

187


Another example: Find the fourth roots of unity

iie

ie

iie

ie

ke

i

i

i

i

ik

)sin()cos(

1)sin()cos(

)sin()cos(

1)0sin()0cos(

3,2,1,0

23

234/323

4

4/2224

224/121

4

4/0204

4/2 for

188


The n complex roots of unity are equally spaced around the circle of unit radius

The cube rootsof unity

189

Euler’s Formula:ei +1 = 0

Properties of Roots of Unity Lemma 30.3 (Cancellations Lemma)

For any integers n 0, k 0, and d > 0,

Proof:

Corollary 30.4 For any even integer n > 0, Proof:

kn

dkdn

kn

kni

dkdnidkdn

e

e

/2

/2

122/ n

n

12

2//22//2 inninn

nin eee

190

Properties of Roots of Unity (cont)

Corollary 99

Let n > 0 be even, and let n and n/2 be the

principal nth and (n/2)th roots of unity. Then

Proof: k

nkn v 2/

2

kn

iknikkn vee

n

2//2/222

2

191

Properties of Roots of Unity (cont) Lemma 30.5 (Halving lemma)

Let n > 0 be even. Then, the squares of the n complex nth roots of unity are the n/2 complex (n/2)th roots of unity.

Proof: If we square all of the nth roots of unity, then each (n/2) th root is obtained exactly twice since:

square same the have and these of both 99 Cor

thus

30.4 Cor

kn

nkn

k

kn

kn

nkn

kn

nkn

v

2/

2222/

2/ ,

192

Properties of Roots of Unity (cont)

Lemma 30.6 (Summation Lemma) For any integer n 1 and nonnegative

integer k not divisible by n Proof:

01

0

n

j

jkn

0

1

11

1

1

1

11

0

kn

k

kn

knn

kn

nkn

n

j

jkn

193

The DFT

Remember, we want to evaluate a polynomial of degree-bound n at

Actually, since we will eventually multiply, n is actually 2n

• We are working with complex (2n)th roots of unity

Also assume that n is a power of 2• We can add new high-order zero coefficients as

necessary

1210 ,,,, nnnnn

194

The DFT (cont)

Polynomial A is in coefficient form

The vector y is the Discrete Fourier Transform of the coefficient vector a

1

0

)(n

j

kjnj

knk

a

Ay

195

The FFT

The FFT allows us to compute the DFTn(a) in time (nlgn)

Define “even” and “odd” polynomials:

)()()(

)(

)(

2][2][

11

37

25

131

][

12

36

24

120

][

2

2

xxAxAxA

xaxaxaxaaxA

xaxaxaxaaxA

OE

nO

nE

n

n

196

The FFT (cont)

Reduces problem of Evaluating degree n polynomial A(x) at

0, 1, . . . , n-1

to evaluating two degree n/2 polynomials at:

(0)2, (1)2, . . . , (n-1)2 . Halving Lemma A[E](x) and A[O](x)

evaluated only at n/2 complex (n/2) th roots of unity

197

The FFT Algorithm

),,,,(

;

;

12/0

),...,,,,2/),...,,(

),...,,,,2/),...,,(

;

;

)1(

)...,,,,

1210

2/

153112/10

242012/10

/2

0

110

n

Ok

kn

Eknk

Ok

kn

Ekk

nOn

OO

nEn

EE

nin

n

yyyy

aay

aay

ntok

aaaanaaa

aaaanaaa

e

a

n

aaan

return

for

FFT(

FFT(

return

2 of power a is n // if

FFT(

O(n) time multiplies if we pre-compute k

198

The FFT Algorithm (cont)

Recursion Tree

76,5432,10 ,,,,,,, aaaaaaaa

7531 ,,, aaaa6420 ,,, aaaa

73 ,aa51 ,aa62 ,aa40 ,aa

0a1a2a3a

4a5a6a7a

199

The FFT Algorithm (cont)

The recurrence for this algorithm is

)lg(

)()2/(2)(

nn

nnTnT

200

Proof of the FFT Algorithm

We need to show thatfor each k = 0, …, n-1

Base case: n = 1

Induction step: Assume algorithm correct for n/2

1

0

)(n

j

kjnj

knk aAy

00100 aay

201

Proof of the FFT Algorithm (cont) Let be the principal (n/2)th root of unity

)()()( Recall

99Corollary by )()(

99Corollary by )()(

2][2][

2][][

2][][

xxAxAxA

AAa

AAa

OE

kOkOOk

kEkEEk

202

Proof of the FFT Algorithm (cont)

)(

)()(

)()(

)()(

2/

2][2/2][

2][2/2][

2][2][

2/

nk

nkn

Onkn

nkn

E

kn

Onkn

kn

E

kn

Okn

kn

E

Ok

kn

Eknk

A

AA

AA

AA

aay

)(

)()( 2][2][

kn

kn

Okn

kn

E

Ok

kn

Ekk

A

AA

aay

203

The Inverse FFT

110

110

,,,

,,,

n

n

bbb

aaa

2-2n10 ,,,c cc

)(

)(

)(

,

)(

)(

)(

122

12

02

122

12

02

nn

n

n

nn

n

n

B

B

B

A

A

A

)(

)(

)(

122

12

02

nn

n

n

C

C

C

O(n)

point-value multiplication

O(n2)

coefficient multiplication

O(n log n)evaluationFFT

interpolationinverse FFT

O(n log n)

204

The Inverse FFT (cont)

Forward FFT given {a0, a1, . . . , an-1 }

compute {y0, y1, . . . , yn-1 }

1

3

2

1

0

)1)(1()1(3)1(21

)1(3963

)1(2642

1321

1

3

2

1

0

1

1

1

1

11111

nnn

nn

nn

nnn

nnnnn

nnnnn

nnnnn

n a

a

a

a

a

y

y

y

y

y

Vn

205


Inverse FFT given {y0, y1, . . . , yn-1 }

compute {a0, a1, . . . , an-1}

1

3

2

1

0

1

)1)(1()1(3)1(21

)1(3963

)1(2642

1321

1

3

2

1

0

1

1

1

1

11111

nnn

nn

nn

nnn

nnnnn

nnnnn

nnnnn

n y

y

y

y

y

a

a

a

a

a

Vn-1

206


It is the same algorithm as FFT, except use n

-1 as “principal” nth root of unity (and divide by n).

)1)(1()1(3)1(2)1(

)1(3963

)1(2642

)1(321

1

1

1

1

1

11111

1

nnnnn

n

n

n

n nF

207


Theorem 30.7 For j, k = 0, 1, …, n-1, the (j, k) entry of

Proof:• We will show that Vn

-1Vn = In

• Consider the (j, j ) entry of Vn-1Vn

nV

kjn

n

is 1

208


•

• If j = j , the summation = 1• If j j , the summation = 0

Therefore, the interpolation/DFTn-1(y) is

1

0

)(

1

0

1

n

k

jjkn

n

k

jkkj

nnn

n

nVV

By the Summation

Lemma

1,,1,01 1

0

njyn

an

k

kjnkj for

209

The Inverse FFT Algorithm

),,,,(

;

;

12/0

),...,,,,2/),...,,(

),...,,,,2/),...,,(

;

;

)1(

)...,,,,

1210

2/

153112/10

242012/10

/2

0

110

n

Ok

kn

Ek

nk

Ok

kn

Ek

k

nOn

OO

nEn

EE

nin

n

yyyyn

aay

naay

ntok

aaaanaaa

aaaanaaa

e

a

n

aaan

return

for

FFT(

FFT(

return

2 of power a is n // if

FFT(-Inverse

210

Summary Slides

Divide & Conquer Algorithms Recursion Review Designing Algorithms Analyzing Divide & Conquer Algorithms MergeSort Analysis of MergeSort QuickSort Performance of QuickSort

211

Summary Slides (cont.)

Randomized QuickSort Analysis of QuickSort Indicator Random Variables Median and Order Statistics The Selection Problem Minimum and Maximum Selection in Expected Linear Time Analysis of Selection Algorithm

212


Selection in Worst-Case Linear Time Analyzing “Median of Medians” Review of Sorts Lower Bounds for Sorting The Decision-Tree Model Analysis of Decision-Tree Model Theorem 8.1 Sorting in Linear Time

213


Counting Sort Analysis of CountingSort: Radix Sort Analysis of RadixSort Proof by induction that RadixSort works Radix Sort Example Bucket Sort Analysis of Bucket Sort

214


Strassen’s Algorithm for Matrix Multiplication An Example of Strassen’s Algorithm Discussion of Strassen’s Algorithm Final Notes on Strassen’s Algorithm Polynomials and FFT Polynomials Review of Complex Numbers

215


Coefficient Representation of Polynomials

Point-Value Representation of Polynomials

Review of Matrix Operations Best of Both Worlds Complex Roots of Unity Properties of Roots of Unity

216


The DFT The FFT The FFT Algorithm Proof of the FFT Algorithm The Inverse FFT The Inverse FFT Algorithm

1 Divide & Conquer Algorithms Part 4. 2 Recursion Review A function that calls itself either...

Documents

Transcript of 1 Divide & Conquer Algorithms Part 4. 2 Recursion Review A function that calls itself either...