Divide & Conquer

ThemesReasoning about code (correctness and cost) recursion, induction, and recurrence relationsDivide and Conquer

Examples sorting (insertion sort & merge sort)computing powers

Sorting

Given a list L = (L1 … Ln) of elements from a totally ordered set (e.g. integers with ≤) return a list M = (M1 … Mn) such that

(sorted M), i.e. M1 ≤ M2 ≤ ≤ Mn

M = (permutation L), i.e. Mi = L(i) where is a bijection from {1,…,n} to {1,…,n}

Use divide and conquer to derive several recursive algorithms and the principle of balance to obtain an efficient algorithm

sortedp

Predicate to test to see if a list is sortedShould check to see that L is a list of integers

(x1 x2 … xn) with x1 ≤ x2 ≤ ≤ xn

xi ≤ xj for 1 ≤ i < j ≤ n

(defun sortedp (L) (cond ((equal L nil) t) ((equal (length L) 1) t) ((and (<= (first L) (second L)) (sortedp (rest L))))))

Correctness of sortedp

L = (x1 x2 … xn)

By induction on len(L)Base cases L nil or of length 1

Show xi ≤ xj for 1 ≤ i < j ≤ n

x1 ≤ x2 and by induction xi ≤ xj for 2 ≤ i < j ≤ n

x1 ≤ x2 ≤ xj x1 ≤ xj for 2 ≤ j ≤ n

Precise Proof is Difficult

We take advantage of informal reasoning and names which provide intuitive meaningL = (x1 x2 … xn)

Prove (fn123 L) = t if xi ≤ xj for 1 ≤ i < j ≤ n and nil otherwise using equational properties of fn13, fn14, fn15, fn17, fn81, fn100 and fn150

(defun fn123 (L) (fn13 ((fn81 L nil) t) ((fn81 (fn150 L) 1) t) ((fn82 (fn100 (fn14 L) (fn17 L)) (fn123 (fn15 L))))))

Insertion Sort

To sort x1,…,xn,

recursively sort x2,…,xn

insert x1 into x2,…,xn

(see code for details)Prove correctness by induction

Insertion Sort

(defun insertionsort (L)

(if (equal L nil)

nil

(insert (first L) (insertionsort (rest L)))))

(defun insert (x L) (cond ((equal L nil) (list x)) ((<= x (first L)) (cons x L)) ((cons (first L) (insert x (rest L))))))

Insertion Sort (Example)(insertionsort ‘(7,6,5,4,3,2,1,0))recursive call returns (0,1,2,3,4,5,6)(insert 7 ‘(0,1,2,3,4,5,6))(0,1,2,3,4,5,6,7)

Number of comparisons to sort inputs that are in reverse sorted order (worst case)

C(n) = C(n-1) + (n-1)

C(1) = 0

Correctness of insertionsort

L = (L1 … Ln)

1.(insertionsort L) terminates and returns a list of length n

2.(insertionsort L) is sorted

3.(insertionsort L) is a permutation of L

Proof by induction on n

1. For n 0, recursive call with smaller n

2. Recursive call produces a sorted list of (rest L) by induction and (insert (first L) (insertionsort (rest L))) returns a sorted list (by induction)

3. Recursive call returns a permutation of (rest L) and insert returns a list containing (first L) and elements of (rest L)

Solve Recurrence for C(n)

C (n)=C (n−1)+(n−1)=C (n−2 )+( n−2)+(n−1)

=C (n−k )+∑i=1

k

(n−i )=C (1)+∑i=1

n−1

(n−i )

¿∑i=1

n−1

i=n(n−1 )/2

Merge Sort

To sort x1,…,xn,

recursively sort x1,…,xa and xa+1,…,xn, where a = n/2

merge two sorted lists

Insertion sort is a special case where a=1

Mergesort

(defun mergesort (L)

(cond

((equal L nil) nil)

((equal (length L) 1) L)

((merge (mergesort (take (floor (length L) 2) L))

(mergesort (drop (floor (length L) 2) L))))))

Merge

(defun merge (L M)

(cond

((equal L nil) M)

((equal M nil) L)

((<= (first L) (first M)) (cons (first L) (merge (rest L) M)))

((cons (first M) (merge L (rest M))))))

Take and Drop

(defun take (n L)

(cond

((equal L nil) nil)

((equal n 0) nil)

((cons (first L) (take (- n 1) (rest L))))))

(defun drop (n L) (cond ((equal L nil) nil) ((equal n 0) L) ((drop (- n 1) (rest L)))))

Merge Sort (Example)

(7,6,5,4,3,2,1,0)after recursive calls (4,5,6,7) and (0,1,2,3)Number of comparisons needed to sort, worst case

(the merged lists are perfectly interleaved)

M(n) = 2M(n/2) + n-1

M(1) = 0What is the best case (all in one list > other list)?

M(n) = 2M(n/2) + n/2

Comparison of Insertion and Merge Sort

Count the number of comparisons for different n=2k

M(n)/C(n) 0 as n increasesC(n) grows faster than M(n)

C(2n)/C(n) 4So, apparently quadratic.C(n) = Θ(n2)

M(2n)/M(n) 2 as n increases

n M(n) C(n) M(n)/C(n)

2 1 1 1

4 4 6 0.667

8 12 28 0.429

16 32 120 0.267

32 80 496 0.161

64 192 2016 0.095

128 4481 8128 0.055

Solve Recurrence for M(n)

Assume n =2k

M (n )=2M (n /2)+cn¿2[ 2M (n/ 4 )+cn /2 ]+cn=

¿22

M (n/22

)+2cn

¿22

[ 2M( n/23

)+cn /22

]+2cn

¿23

M ( n/23

)+3 cn.. .

2k

M (n/2k

)+kcn=ckn=c lg (n )∗n

Mergesort

What is wrong with the following?

(defun mergesort (L)

(if (equal L nil)

nil

(merge (mergesort (mytake (floor (length L) 2) L))

(mergesort (drop (floor (length L) 2) L)))))

Correctness of mergesort

L = (L1 … Ln)

1.(mergesort L) terminates and returns a list of length n

2.(mergesort L) is sorted

3.(mergesort L) is a permutation of L

4.Proof by induction on n

1. For n 2, recursive calls with smaller n

2. Recursive calls produce sorted lists by induction and merge produces a sorted list

3. Recursive calls contain lists whose concatenation is L and hence by induction return lists whose concatenation is L. merge returns a permutation of L

Properties of nth

L = (L1 … Llen(L))

1.0 < n ≤ len(L), then (nth n L) = Ln

Proof by induction on n

2.n > len(L), then (nth n L) = nil Proof by induction on len(L)

(defun nth (n L) (if (equal L nil) nil (if (equal n 1) (first L) (nth (- n 1) (rest L)))))

Properties of nth

L = (L1 … Llen(L))

1.0 < n ≤ len(L), then (nth n L) = Ln

Proof by induction on n

1. Base case. n = 1 return (first L)

2. By induction (nth (- n 1) (rest L)) returns the (n-1)st element of (rest L) which is the nth element of L (n > 1)

Properties of nth

L = (L1 … Llen(L))

1.n > len(L), then (nth n L) = nil Proof by induction on len(L)

1. Base case. Len(L) = 0. L = nil and return nil

2. n > len(L) n-1 > len((rest L)), therefore by induction (nth (- n 1) (rest L)) returns nil

Properties of take and drop

L = (L1 … Llen(L))

1.0 ≤ n ≤ len(L), then len((take n L)) = n Proof by induction on n

2.n > len(L), then (take n L) = L Proof by induction on len(L)

3.0 < k ≤ n ≤ len(L), then (nth k (take n L)) = Lk

Proof by property of nth and induction on k

4.0 ≤ n ≤ len(L), then len((drop n L)) = len(L) – n

5.n > len(L), then (drop n L) = nil

6.0 ≤ n ≤ len(L), 0 < k ≤ len(L)-n, then (nth k (drop n L)) = Ln+k

7.0 ≤ n ≤ len(L), (append (take n L) (drop n L)) = L

Properties of merge

L = (L1 … Lm) and M = (M1 … Mn)

•Assume L is sorted and M is sorted

•(merge L M) is a list of length m+n

•(merge L M) is sorted

•(merge L M) is a permutation of (L1 … Lm M1 … Mn)

Exercise

Use check= to verify properties of take and drop for several examples

Use induction to prove the properties

Solution Property 1

L = (L1 … Llen(L))

1.0 ≤ n ≤ len(L), then len((take n L)) = n Proof by induction on n

1. Base case. n = 0 return nil and len(nil) = 0.

2. Assume n > 0 and show that len((take n L)) = n. Assume that take returns a list of length k for k < n [Inductive hypothesis]

3. In this case take returns (cons (first L) (take (- n 1) (rest L))) and since n-1 ≤ len((rest L)) (take (- n 1) (rest L)) by induction is a list of length n-1. Therefore (cons (first L) (take (- n 1) (rest L))) has length (n-1) + 1 = n.

Solution Property 2

L = (L1 … Llen(L))

1.n > len(L), then (take n L) = L Proof by induction on len(L)

1. Base case. len(L) = 0 and take returns L = nil independent of n.

2. Assume n > len(L) = and show that (take n L) = L. Assume that take returns M when len(M) < [Inductive hypothesis]

3. In this case take returns (cons (first L) (take (- n 1) (rest L)). By induction, (take (- n 1) (rest L)) returns (rest L) and therefore (take n L) returns (cons (first L) (rest L)) = L.

Solution Property 3

L = (L1 … Llen(L))

1.0 < k ≤ n ≤ len(L), then (nth k (take n L)) = Lk

Proof by property of nth and induction on k1. Base case. k = 1and by the first property of nth, (nth 1 (take n L)) =

the first element of (take n L) which by the definition of take is the first element of L.

2. Assume the jth element of (take n L) = Lj for j < k [Inductive hypothesis] and show that the kth element of (take n L) = Lk.

3. By the first property of nth (nth k (take n L)) is the kth element of (take n L) and by the definition of take, that is the (k-1)st element of (take (- n 1) (rest L)) which by induction is the (k-1)st element of (rest L) which is the kth element of L.

Solution of Properties 4-7

L = (L1 … Llen(L))

Properties 4-6 for drop are analogous to properties 1-3 for take and their proofs are similar to those for properties 1-3.Property 7 follows from properties 3 and 6.

Computing Powers

Recursive definition:an = a an-1, n > 0a0 = 1

Number of multiplicationsM(n) = M(n-1) + 1, M(0) = 0M(n) = n

Exercise

Write a recursive ACL program to compute the power of a number using the recursive definition on the previous slide

Prove by induction on n that (power a n) = an

Solution

; inputs a rational number a and a natural number n

; output a rational number equal to a^n

(defun power (a n)

(if (= n 0)

1

(* a (power a (- n 1)))))

(check= (power 2 0) 1)

(check= (power 2 10) 1024)

Solution Continued

(defun power (a n)

(if (= n 0)

1

(* a (power a (- n 1)))))

Base case n = 0 (power a 0) = 1 = a0

Assume n > 0 and (power a (- n 1)) = an-1 [Inductive Hypothesis] (power a n) = (* a (power a (- n 1)) = a×an-1 [by induction] = an

Binary Powering (Recursive)

Binary poweringx16 = (((x2)2)2)2, 16 = (10000)2

x23 = (((x2)2x)2x)2x, 23 = (10111)2

Recursive (right-left)

xn = (xn/2)2 x(n % 2)

M(n) = M(n/2) + [n % 2] + 1

Exercise

Write a recursive ACL program to compute the power of a number using the binary powering algorithm

Prove by induction on n that (fastpower a n) = an

Solution

; inputs a rational number a and a natural number n

; output a rational number equal to a^n

(defun fastpower (a n)

(if (= n 0)

1

(let ((b (fastpower a (floor n 2))))

(if (evenp n)

(* b b)

(* b b a)))))

(check= (power 2 0) 1)

(check= (fastpower 2 16) 65536)

(check= (fastpower 2 15) 32768)

Solution Continued

Base case n = 0 (fastpower a 0) = 1 = a0

Assume n > 0 and (fastpower a k) = ak for k < n [Inductive Hypothesis] Even case n = 2k

(fastpower a n) = (* (fastpower a k) (fastpower a k)) = ak×ak [by induction] = a2k = an

Odd case n = 2k+1

(fastpower a n) = (* (fastpower a k) (fastpower a k) a) = ak×ak×a [by induction] = a2k+1 = an

Binary Powering

Number of multiplicationsLet (n) = number of 1bits in binary

representation of nnum bits = lg(n) +1

M(n) = lg(n) + (n)(n) ≤ lg(n) =>M(n) ≤ 2lg(n)

Asymptotic Growth Rates

f(n) = O(g(n)) [grows at the same rate or slower]

There exists positive constants c and n0

such that f(n) c g(n) for all n n0

Ignore constants and low order terms

Asymptotic Growth Rates

f(n) = o(g(n)) [grows slower]

f(n) = O(g(n)) and g(n) O(f(n))

limn f(n)/g(n) = 0

f(n) = (g(n)) [grows at the same rate]

f(n) = O(g(n)) and g(n) = O(f(n))

Asymptotic Growth Rates

(log(n)) – logarithmic [log(2n)/log(n) = 1 + log(2)/log(n)]

(n) – linear [double input double output](n2) – quadratic [double input quadruple output](n3) – cubit [double input output increases by

factor of 8](nk) – polynomial of degree k(cn) – exponential [double input square output]

Merge Sort and Insertion Sort

Insertion SortTI(n) = TI(n-1) + (n) =(n2) [worst case]TI(n) = TI(n-1) + (1) =(n) [best case]

Merge SortTM(n) = 2TM(n/2) + (n) =(nlogn) [worst case]TM(n) = 2TM(n/2) + (n) =(nlogn) [best case]

Divide & Conquer Recurrence

Assume T(n) = aT(n/b) + (n)

T(n) = (n) [a < b]

T(n) = (nlog(n)) [a = b]

T(n) = (nlogb(a)) [a > b]

Karatsuba’s AlgorithmUsing the classical pen and paper algorithm two n digit

integers can be multiplied in O(n2) operations. Karatsuba came up with a faster algorithm.

Let A and B be two integers with

A = A110k + A0, A0 < 10k

B = B110k + B0, B0 < 10k

C = A*B = (A110k + A0)(B110k + B0)

= A1B1102k + (A1B0 + A0 B1)10k + A0B0

Instead this can be computed with 3 multiplications

T0 = A0B0

T1 = (A1 + A0)(B1 + B0)

T2 = A1B1

C = T2102k + (T1 - T0 - T2)10k + T0

Complexity of Karatsuba’s Algorithm

Let T(n) be the time to compute the product of two n-digit numbers using Karatsuba’s algorithm. Assume n = 2k. T(n) = (nlg(3)), lg(3) 1.58

T(n) 3T(n/2) + cn

3(3T(n/4) + c(n/2)) + cn = 32T(n/22) + cn(3/2 + 1)

32(3T(n/23) + c(n/4)) + cn(3/2 + 1)

= 33T(n/23) + cn(32/22 + 3/2 + 1)

…

3iT(n/2i) + cn(3i-1/2i-1 + … + 3/2 + 1)

...

cn[((3/2)k - 1)/(3/2 -1)] --- Assuming T(1) c

2c(3k - 2k) 2c3lg(n) = 2cnlg(3)

Identities for Sums∑icf ( i )=cf 1+cf 2+…=c ( f 1+ f 2+… )=c∑

if ( i )

∑i

( f (i )+ g ( i ) )=∑if ( i )+∑

ig ( i )

∑if ( i ) g ( i )≠∑

if ( i )∑

ig ( i )

Some Common Sums

∑i=a

b

c=(b−a+1)c ,b≥a

∑i=0

m

i=∑i=1

m

i=m (m+1 )2

∑i=1

m

i2=m (m+1 ) (2m+1 )6

∑i=0

m

r i=rm+1−1r−1

, r≠1

∑i=0

∞r i=1

1−r,0<r<1

Arithmetic Series

Sum of consecutive integers (written slightly differently):

∑i=1

n−1

bi=b∑i=1

n−1

i=b+2b+3b+…+(n−1)b

¿b(n−1)n2

Arithmetic Series (Proof)

Base case (n=1) : ∑i=1

1

i=1=1(1+1 )/2

Assume the theorem is true for size n and show the assumption implies it is true for size n+1

∑i=1

n+1

i=( n+1)+∑i=1

n

i

By induction¿( n+1)+n(n+1)/2

¿2(n+1)+n(n+1 )2

=(n+2)( n+1)2

Geometric Series

= 1 + x + x2 + … + xn-1 = ∑i=0

n−1

xi { n , x=1

xn

−1

x−1, x≠1

Geometric Series (Proof)

¿ 0=1=( x−1 )( x−1 )

¿ ¿Assume the theorem is true for size n-1 and show for size n . ¿∑i=0

n

xi

=xn

+∑i=0

n−1

xi

¿which by induction =xn

+(x

n

−1 )( x−1) ¿=

xn

( x−1)+(xn

−1)( x−1)

=(x

n+1

−1 )( x−1)

¿ ¿

Floor and Ceiling

Let x be a real numberThe floor of x, x, is the largest integer less than

or equal to x If an integer k satisfies k x < k+1, k = xE.G. 3.7 = 3, 3 = 3

The ceiling of x, x, is the smallest integer greater than or equal to x If an integer k satisfies k-1 < x k, k = xE.G. 3.7 = 4, 3 = 3

logarithm

y = logb(x) by = x

Two important cases ln(x) = loge(x) lg(x) = log2(x) [frequently occurs in CS]

Properties log(cd) = log(c) + log(d) log(cx) = xlog(c) logb(bx) = x = blogb(x) d ln(x)/dx = 1/x

logarithm

2k x < 2k+1 k = lg(x)E.G. 16 25 < 32 4 lg(25) < 5lg(25) 4.64

Change of baselogc(x) = logb(x) / logb(c)

Proof. y = logc(x) cy = x ylogb(c) = logb(x) y = logb(x) / logb(c)