DivideandConquer.ppt

8/14/2019 DivideandConquer.ppt

1/77

Design and Analysis of Algorithm

Divide and Conquer


2/77

Divide and Conquer Technique2

Acknowledgement

This lecture note has been summarized from lecture noteon Data Structure and Algorithm, Design and Analysis of

Computer Algorithm all over the world. I cant remember

where those slide come from. However, Id like to thank

all professors who create such a good work on thoselecture notes. Without those lectures, this slide cant be

finished.


3/77


Review: What is an algorithm?

A step by step procedure for performing sometask in a finite amount of time

A precise set of instructions for achieving aparticular goal


4/77


Review: General Concepts

Algorithm strategy Approach to solving a problem

May combine several approaches

Algorithm structure Iterative execute action in loop

Recursive reapply action to subproblem(s)

Problem type Satisfying find any satisfactory solution

Optimization find best solutions (vs. cost

metric)


5/77


Review: What you need to know about

algorithms?

Running time Memory requirements

Understand how the complexity of an algorithm

is measured Develop techniques to measure these

complexities experimentally

In other words, measure the efficiency of an

algorithm!


6/77


Review: Measures of efficiency

Three main measures Best

Worst

Average E.g. consider a list

Best: first item in the list

Worst: last in the list, or not in the list

Average: in the middle of the list


7/77Divide and Conquer Technique7

Review: An example of an algorithm

(pseudocode)

BEGIN

Get WAGES

IF WAGES



Some Algorithm Strategies

Recursive algorithms Backtracking algorithms

Divide and conquer algorithms

Dynamic programming algorithms Greedy algorithms

Brute force algorithms

Branch and bound algorithms Heuristic algorithms



Divide-and-Conquer

A general methodology for using recursion todesign efficient algorithms

It solves a problem by:

Diving the data into parts Finding sub solutions for each of the parts

Constructing the final answer from the sub

solutions



Divide and Conquer

Based on dividing problem into subproblems Approach

1. Divide problem into smaller subproblems

Subproblems must be of same typeSubproblems do not need to overlap

2. Solve each subproblem recursively

3. Combine solutions to solve original problem

Usually contains two or more recursive calls



Divide-and-conquer technique

subproblem 2

of size n/2

subproblem 1

of size n/2

a solution to

subproblem 1

a solution to

the original problem

a solution to

subproblem 2

a problem of size n



Divide and Conquer Algorithms

Based on dividing problem into subproblems Divide problem into sub-problems

Subproblems must be of same type

Subproblems do not need to overlap Conquerby solving sub-problems recursively.

If the sub-problems are small enough, solve

them in brute force fashion

Combine the solutions of sub-problems into a

solution of the original problem (tricky part)



D-A-C

For Divide-and-Conquer algorithms the runningtime is mainly affected by 3 criteria:

The number of sub-instances into which a

problem is split. The ratio of initial problem size to sub-

problem size.

The number of steps required to divide the

initial instance and to combine sub-solutions.



An example of D-A-C

Given an array of n values find, in order, the lowest twovalues efficiently.

To make life easier we assume n is at least 2 and that it

is always divisible by 2

Have you understood the problem??

The solution should be 2 followed by 4

4 569102

9 37282What if??



Possible Solutions

Sort the elements and take the two first We get more information than we require (not

very efficient!!)

Find the lowest, then find the second lowest Better than the first but still we have to go through

the array two times

Be careful not to count the lowest value twice

Search through the array keeping track of thelowest two found so far

Lets see how the divide and conquer technique

works!



Divide and conquer approach

There are many ways to divide the array It seems reasonable that the division should be

related to the problem

Division based on 2!! Divide the array into two parts?

Divide the array into pairs?

Go ahead with the second option!



The algorithm

Lets assume the following array

We divide the values into pairs

We sort each pair

Get the first pair (both lowest values!)

4295 6 13762

4295 6 13762

1925 6 47362



The algorithm (2)

We compare these values (2 and 6) with the values ofthe next pair (3 and 7)

Lowest 2,3 The next one (5 and 6)

Lowest 2,3

The next one (2 and 9)

Lowest 2,2 The next one (1 and 4)

Lowest 1,2

1925 6 47362


19/77Divide and Conquer Technique

19

Example: Divide and Conquer

Binary Search Heap Construction

Tower of Hanoi

Exponentiation Fibonnacci Sequence

Quick Sort

Merge Sort Multiplying large Integers Matrix Multiplications Closest Pairs



20

Exponentiation

Power can be easily computed in O(logn) time as follows:

long Power (int x, int n) {

if(n==0) return 1; else {

y =Power (x, n/2);

if(n % 2 == 0) return(y*y) // n even

else return (y*y*x);

}

Ti C l it E l 1



21

Time Complexity Example 1:

Fibonnacci Sequence

Fibonnacci sequence F(n) = F(n-1) + F(n-2)

Approach 1:

int Fib(n)

{ if (n < 2) return (n);else {

return (Fib (n-1) + Fib (n-2));}

}

Approach 2:

int Fib(n) {

if (n < 2) return (n);else {

int F1 = 1; F2 = 0;

for (i = 2; i



22

Fibonnacci Sequence

Approach 3: the complexity is O(logn)Note that we can rewrite | F(n) | | 1 1 | | F(n-1) |

| | = | | * | || F(n-1)| | 1 0 | | F(n-2) |

Thus,2

| F(n) | | 1 1 | | F(n-1) | | 1 1 | | F(n-2) || | = | | * | | = | | * | || F(n-1)| | 1 0 | | F(n-2) | | 1 0 | | F(n-3) |

(n-1)| 1 1 | | F(1) |

= | | * | || 1 0 | | F(0) |

Therefore, computing F(n) is reduced to computing

(n-1)| 1 1 || || 1 0 |



23

Heap Construction

Construct heap of T =>Construct heap T1,

Construct heap T2

Adjust heap with root T

T

T1 T2



24

Divide and Conquer Examples

Quicksort Partition array into two parts around pivot

Recursively quicksort each part of array

Concatenate solutions Mergesort

Partition array into two parts

Recursively mergesort each half

Merge two sorted arrays into single sorted array


25/77

Quick Sort


26/77


Sorting Problem Revisited

Given: an unsorted array

Goal: sort it

62317425

76543221


27/77


Quick Sort

Approach Select pivot value (near median of list)

Partition elements (into 2 lists) using pivot value

Recursively sort both resulting lists Concatenate resulting lists

For efficiency pivot needs to partition list evenly

Performance

O( n log(n) ) average case

O( n2 ) worst case


28/77


Quick Sort Algorithm

1. If list below size K Sort w/ other algorithm

Else pick pivot x and

partition S into L elements < x

E elements = x

G elements > x

3. Quicksort L & G

4. Concatenate L, E & G

If not sorting in place

x

x

L GE

x


29/77


Quick Sort Code

void quickSort(int[] a, int x, int y) {int pivotIndex;

if ((y x) > 0) {

pivotIndex = partionList(a, x, y);

quickSort(a, x, pivotIndex 1);quickSort(a, pivotIndex+1, y);

}

}

int partionList(int[] a, int x, int y) {

// partitions list and returns index of pivot

}

Lowerend ofarray

regionto be

sorted

Upperend ofarray

regionto be

sorted


30/77


Quick Sort Example

7 2 8 5 4

2 5 4 7 8

5 42

4 5

2 4 5 7 8

2 4 5 7 8

4 52

4 5

Partition & Sort Result


31/77


Quick Sort Code

int partitionList(int[] a, int x, int y) {int pivot = a[x];

int left = x;

int right = y;

while (left < right) {while ((a[left] < pivot) && (left < right))

left++;

while (a[right] > pivot)

right--;

if (left < right)swap(a, left, right);

}

swap(a, x, right);

return right;

Use firstelementas pivot

Partition elementsin array relative to

value of pivot

Place pivot in middleof partitioned array,

return index of pivot


32/77

Merge Sort


33/77


Merge Sort

Approach Partition list of elements into 2 lists

Recursively sort both lists

Given 2 sorted lists, merge into 1 sorted lista) Examine head of both lists

b) Move smaller to end of new list

Performance

O( n log(n) ) average / worst case


34/77


Merge Example

2 4

7 5 8

2 7 4 5 8

2

7 4 5 8

2 4 5

7 8

2 4 5 7

8

2 4 5 7 8


35/77


36/77


Merge Sort Code

void mergeSort(int[] a, int x, int y) {int mid = (x + y) / 2;

if (y == x) return;

mergeSort(a, x, mid);

mergeSort(a, mid+1, y);merge(a, x, y, mid);

}

void merge(int[] a, int x, int y, int mid) {

// merges 2 adjacent sorted lists in array

}

Lowerend ofarray

regionto be

sorted

Upperend ofarray

region

to besorted

Upper


37/77


Merge Sort Code

void merge (int[] a, int x, int y, int mid) {

int size = y x;int left = x;

int right = mid+1;

int[] tmp; int j;

for (j = 0; j < size; j++) {if (left > mid) tmp[j] = a[right++];

else if (right > y) || (a[left] < a[right])

tmp[j] = a[left++];

else tmp[j] = a[right++];

}

for (j = 0; j < size; j++)

a[x+j] = tmp[j];

}

Lowerend of

1st arrayregion

Upperend of

2nd arrayregion

Upperend of

1st arrayregion

Copy merged

array back

Copy smaller of two

elements at head of 2array regions to tmpbuffer, then move on


38/77


Mergesort: Divide Step

Step 1 Divide62317425

62317425

62317425

62317425

log(n) divisions to split an array of size n into single elements


39/77


Mergesort: Conquer Step

Step 2 Conquer

76543221

63217542

62317452

62317425

O(n)

O(n)

O(n)

O(n)

O(n logn)logn iterations, each iteration takes O(n) time. Total Time:


40/77


Mergesort: Combine Step

Step 3 Combine

2 arrays of size 1 can be easily merged toform a sorted array of size 2

2 sorted arrays of size n and m can bemerged in O(n+m) time to form a sorted

array of size n+m

2 525


41/77


Mergesort: Combine Step

Combining 2 arrays of size 4

6321

75421

32

54221

632

754221

3

543221

6

75443221 Etcetera

76543221


42/77


Mergesort: Example

20 4 7 6 1 3 9 5

20 4 7 6 1 3 9 5

20 4 7 6 1 3 9 5

20 4 7 6 1 3 9 5

4 20 6 7 1 3 5 9

4 6 7 20 1 3 5 9

1 3 4 5 6 7 9 20

Divide

Conquer


43/77


MergeSort: Running Time

The problem is simplified to baby steps for the ith merging iteration, the

complexity of the problem is O(n)

number of iterations is O(log n) running time: O(n logn)


44/77

Integer Multiplication


45/77


46/77


To multiply two n-digit integers: Multiply four n-digit integers.

Add two n-digit integers, and shift to obtain

result.

Divide-and-Conquer Multiplication

assumes n is a power of 2


47/77


To multiply two n-digit integers: Add two n digit integers.

Multiply threen-digit integers.

Add, subtract, and shift n-digit integers toobtain result.

Karatsuba Multiplication

A B CA C


48/77


Karatsuba Multiplication (cont)

Theorem. [Karatsuba-Ofman, 1962] Canmultiply two n-digit integers in O(n1.585) bit

operations.


49/77


Karatsuba: Recursion Tree

n

3(n/2)

9(n/4)

3k (n / 2k)

3 lg n (2)

. . .

. . .

T(n)

T(n/2)

T(n/4) T(n/4)

T(2) T(2) T(2) T(2) T(2) T(2) T(2) T(2)

T(n / 2k)

T(n/4)

T(n/2)

T(n/4) T(n/4)T(n/4)

T(n/2)

T(n/4) T(n/4)T(n/4)

. . .

. . .


50/77


Multiplying Large Integers

High school multiplications

A = an-1 an-2 a1 a0

B = bn-1bn-2 b1b0

eg)1 1 0 0 1 11 0 1 0 0 1

1 1 0 0 1 10 0 0 0 0 0

0 0 0 0 0 0

0 0 0 0 0 01 1 0 0 1 1

1 1 0 0 1 1

Time O(N2)

l i l i ( )


51/77


Multiplying Large Integers (cont)

A1 = an-1 an-2 an/2 A0 = an/2-1 a1 a0

B1 = bn-1 bn-2 bn/2 B0 = bn/2-1 b1 b0Define X = A1 + A0

Y = B1 + B0

AB = A1B12n + (XY A1B1 A0B0) 2

n/2 + A0B0

Complexity? An integer multiplication of two n bit integer is

reduced to

=> three integer multiplications of n/2 bits, and 6 additions of n bits

T(n) = 3T(n/2) + O(n) = O(nlog23) = O(n1.59)

Multiplying integers


52/77


AB = (A1104 +A0 ) ( B1104 + B0 )= A1B110

8 + (A1B0+A0B1)104 +A0B0

4 multiplications

3 additions

2 shiftings

Multiplying integers(Decimal numbers)

2 3 6 72 3 6 7 8 1 8 3 8 1 8 3

1 3 0 4 6 4 9 11 3 0 4 6 4 9 1

A =

B =

A1 = A0 =

B1 = B0 =

T(n) = 4T(n/2) + O(n)T(1) = O(1)

T(n) = O(n2)

Multiplying integers(Decimal numbers)


53/77


Multiplying integers(Decimal numbers)D-A-C Approach

Let X = (A1 +A0 ) , Y = ( B1 + B0 )XY =A1B1 +A1B0 +A0B1 +A0B0

XY -A1B1 - A0B0

= A1

B0

+A0

B1

AB = (A1104 +A0 ) ( B110

4 + B0 )

= A1B1108 + (A1B0+A0B1)10

4 +A0B0= A1B110

8 + (XY -A1B1 - A0B0)104 +A0B0

Recursively compute XY, A1B1 , A0B0,

3 multiplications

6 additions2 shiftin s

T(n) = 3T(n/2) + O(n)T(1) = O(1)

T(n) = nlog23= O(n1.59)


54/77

Matrix Multiplication

M i M l i li i


55/77


Matrix Multiplication

Matrix multiplication. Given two n-by-n matrices A and B,compute C = AB.

Brute force. (n3) arithmetic operations.

Fundamental question. Can we improve upon brute

force?

M i M l i li i W


56/77


Matrix Multiplication: Warmup

Divide-and-conquer. Divide: partition A and B into n-by-n blocks.

Conquer: multiply 8 n-by-n recursively.

Combine: add appropriate products using 4matrix additions.

M t i M lti li ti K Id


57/77


Matrix Multiplication: Key Idea

Key idea. multiply 2-by-2 block matrices withonly 7 multiplications.

7 multiplications.

18 = 10 + 8 additions (or subtractions).

F t M t i M lti li ti


58/77


Fast Matrix Multiplication

Fast matrix multiplication. (Strassen, 1969) Divide: partition A and B into n-by-n blocks.

Compute: 14 n-by-n matrices via 10 matrix

additions.

Conquer: multiply 7 n-by-n matrices recursively. Combine: 7 products into 4 terms using 8 matrix

additions.

Analysis.

Assume n is a power of 2.

T(n) = # arithmetic operations.

F t M t i M lti li ti i P ti


59/77


Fast Matrix Multiplication in Practice

Implementation issues. Sparsity.

Caching effects.

Numerical stability.

Odd matrix dimensions. Crossover to classical algorithm around n = 128.

Common misperception: "Strassen is only a theoretical

curiosity." Advanced Computation Group at Apple Computer reports

8x speedup on G4 Velocity Engine when n ~ 2,500.

Range of instances where it's useful is a subject of

controversy.

F t M t i M lti li ti i Th


60/77


Fast Matrix Multiplication in Theory

Q. Multiply two 2-by-2 matrices with only 7 scalar multiplications? A. Yes! [Strassen, 1969]

Q. Multiply two 2-by-2 matrices with only 6 scalar multiplications?

A. Impossible. [Hopcroft and Kerr, 1971]

Q. Two 3-by-3 matrices with only 21 scalar multiplications?

A. Also impossible.

Q. Two 70-by-70 matrices with only 143,640 scalar multiplications? A. Yes! [Pan, 1980]

(nlog3 21)=O(n

2.77)

(nlog2 6)=O(n

2.59)

(nlog2 7 )=O(n

2.81)

(n log70143640)=O(n 2.80 )

F t M t i M lti li ti i Th


61/77


Fast Matrix Multiplication in Theory

Decimal wars. December, 1979: O(n2.521813). January, 1980: O(n2.521801).

Best known. O(n2.376) [Coppersmith-Winograd, 1987.]

Conjecture. O(n2+) for any > 0.

Caveat. Theoretical improvements to Strassen are

progressively less practical.

Ob ti


62/77


Observations

Comparison: n= 70 Direct multiplication: 703 = 343,000 Strassen: 70lg 7 is approximately 150,000 Crossover point typically around n = 20

Hopcroft and Kerr have shown 7 multiplications arenecessary to multiply 2 by 2 matrices But we can do better with larger matrices

Best lower bound is (n2) (since there are n2 entries)

Matrix multiplication can be used in some graph

algorithms as a fundamental step, so theoreticalimprovements in the efficiency of this operation can leadto theoretical improvements in those algorithms


63/77

Closest Pair of Points

Cl t P i f P i t


64/77



Closest pair. Given n points in the plane, find a pair with smallestEuclidean distance between them.

Fundamental geometric primitive.

Graphics, computer vision, geographic information systems,

molecular modeling, air traffic control.

Special case of nearest neighbor, Euclidean MST, Voronoi.

Brute force. Check all pairs of points p and q with (n2)

comparisons. 1-D version. O(n log n) easy if points are on a line.

Assumption. No two points have same x coordinate.

to make presentation cleaner

fast closest pair inspired fast algorithms for these problems


65/77

Closest Pair of Points: First Attempt


66/77


Closest Pair of Points: First Attempt

Divide. Sub-divide region into 4 quadrants. Obstacle. Impossible to ensure n/4 points in

each piece.

L



67/77



Algorithm.

Divide: draw vertical line L so that roughly n points on each side.

L



68/77



Algorithm.

Divide: draw vertical line L so that roughly n points on each side. Conquer: find closest pair in each side recursively.

12

21

L



69/77



Algorithm.

Divide: draw vertical line L so that roughly n points on each side. Conquer: find closest pair in each side recursively.

Combine: find closest pair with one point in each side.

Return best of 3 solutions.

12

218

L

seems like (n2)



70/77



Find closest pair with one point in each side, assuming that distance < .

12

21

= min(12, 21)

L



71/77




Observation: only need to consider points within of line L.

12

21

L

= min(12, 21)



72/77


12

21

1

2

3

45

6

7




Sort points in 2-strip by their y coordinate.

L

= min(12, 21)



73/77


12

21

1

2

3

45

6

7




Sort points in 2-strip by their y coordinate.

Only check distances of those within 11 positions in sorted list!

L

= min(12, 21)



74/77



Def. Let si be the point in the 2-strip, with

the ith smallest y-coordinate.

Claim. If |i j| 12, then the distance between

si and sj is at least .

Pf. No two points lie in same -by- box.

Two points at least 2 rows apart

have distance 2().

Fact. Still true if we replace 12 with 7.

27

2930

31

28

26

25

2 rows

39

i

j

Closest Pair Algorithm


75/77


Closest Pair Algorithm

Closest-Pair(p1, , p

n) {

Compute separation line L such that half the points

are on one side and half on the other side.

1 = Closest-Pair(left half)

2= Closest-Pair(right half)

= min(1, 2)

Delete all points further than from separation line L Sort remaining points by y-coordinate.

Scan points in y-order and compare distance between

each point and next 11 neighbors. If any of thesedistances is less than , update .

return.}

O(n log n)

2T(n / 2)

O(n)

O(n log n)

O(n)

Closest Pair of Points: Analysis


76/77


Closest Pair of Points: Analysis

Running time.

Q. Can we achieve O(n log n)?

A. Yes. Don't sort points in strip from scratch each time.

Each recursive returns two lists: all points sorted by y

coordinate, and all points sorted by x coordinate.

Sort by merging two pre-sorted lists.

Other Applications


77/77

Other Applications

Find median (Probabilistic) Convex Hull

Reversing array

Maximum consecutive subsequence

Etc

DivideandConquer.ppt

Documents

Transcript of DivideandConquer.ppt