Asymptotic Notation and Recurrences - Uppsala University€¦ · Asymptotic notation. O. bounds):...

September 12, 2005 Copyright © 2001-5 Erik D. Demaine and Charles E. Leiserson L2.1

Introduction to Algorithms6.046J/18.401J

LECTURE 2Asymptotic Notation• O-, !-, and "-notationRecurrences• Substitution method• Iterating the recurrence• Recursion tree• Master method

Prof. Erik Demaine


Asymptotic notation

O-notation (upper bounds):

We write f(n) = O(g(n)) if there exist constants c > 0, n0 > 0 such that 0 # f(n) # cg(n) for all n $ n0.


exis


Asymptotic notation




EXAMPLE: 2n2 = O(n3) (c = 1, n0 = 2)

exis


Asymptotic notation




EXAMPLE: 2n2 = O(n3) (c = 1, n0 = 2)

functions, not values

exis


Asymptotic notation




EXAMPLE: 2n2 = O(n3) (c = 1, n0 = 2)

functions, not values

funny, “one-way”equality

exis


Set definition of O-notation

O(g(n)) = { f(n) : there exist constants c > 0, n0 > 0 such that 0 # f(n) # cg(n)for all n $ n0 }


exist





EXAMPLE: 2n2 % O(n3)

exist





EXAMPLE: 2n2 % O(n3)(Logicians: &n.2n2 % O(&n.n3), but it’s convenient to be sloppy, as long as we understand what’s really going on.)

exist


Macro substitution

Convention: A set in a formula represents an anonymous function in the set.


Macro substitution


f(n) = n3 + O(n2) means f(n) = n3 + h(n)for some h(n) % O(n2) .

EXAMPLE:

some


Macro substitution


n2 + O(n) = O(n2)meansfor any f(n) % O(n):

n2 + f(n) = h(n)for some h(n) % O(n2) .

EXAMPLE:

any

some


!-notation (lower bounds)

O-notation is an upper-bound notation. It makes no sense to say f(n) is at least O(n2).




!'g(n)) = { f(n) : there exist constants c > 0, n0 > 0 such that 0 # cg(n) # f(n)for all n $ n0 }


exist






EXAMPLE: (c = 1, n0 = 16))(lgnn !(

exist


"-notation (tight bounds)

"'g(n)) = O(g(n)) ) !(g(n))"'g(n)) = O(g(n)) ) !(g(n))


"-notation (tight bounds)

"'g(n)) = O(g(n)) ) !(g(n))"'g(n)) = O(g(n)) ) !(g(n))

)(2 2221 nnn "(*EXAMPLE:


+-notation and ,-notation

O-notation and !-notation are like # and $.o-notation and ,-notation are like < and >.

+'g(n)) = { f(n) : for any constant c > 0, there is a constant n0 > 0such that 0 # f(n) - cg(n)for all n $ n0 }

+'g(n)) = { f(n) : for any constant c > 0, there is a constant n0 > 0such that 0 # f(n) - cg(n)for all n $ n0 }

EXAMPLE: (n0 = 2/c)2n2 = o(n3)

any

a


+-notation and ,-notation

O-notation and !-notation are like # and $.o-notation and ,-notation are like < and >.

,'g(n)) = { f(n) : for any constant c > 0, there is a constant n0 > 0such that 0 # cg(n) - f(n)for all n $ n0 }

,'g(n)) = { f(n) : for any constant c > 0, there is a constant n0 > 0such that 0 # cg(n) - f(n)for all n $ n0 }

EXAMPLE: (n0 = 1+1/c))(lg nn ,(

anany

a


Solving recurrences

• The analysis of merge sort from Lecture 1required us to solve a recurrence.

• Recurrences are like solving integrals, differential equations, etc.!Learn a few tricks.

• Lecture 3: Applications of recurrences to divide-and-conquer algorithms.


Substitution method

1. Guess the form of the solution.2. Verify by induction.3. Solve for constants.

The most general method:


Substitution method

1. Guess the form of the solution.2. Verify by induction.3. Solve for constants.

The most general method:

EXAMPLE: T(n) = 4T(n/2) + n• [Assume that T(1) = "(1).]• Guess O(n3) . (Prove O and ! separately.)• Assume that T(k) # ck3 for k < n .• Prove T(n) # cn3 by induction.


Example of substitution

3

33

3

3

))2/(()2/(

)2/(4)2/(4)(

cnnnccn

nncnnc

nnTnT

#**(

.(.#.(

desired – residual

whenever (c/2)n3 – n $ 0, for example, if c $ 2 and n $ 1.

desired

residual

by the induction hypothesis


Example (continued)• We must also handle the initial conditions,

that is, ground the induction with base cases.

• Base: T(n) = "(1) for all n < n0, where n0is a suitable constant.

• For 1 # n < n0, we have “"(1)” # cn3, if we pick c big enough.


Example (continued)• We must also handle the initial conditions,

that is, ground the induction with base cases.

• Base: T(n) = "(1) for all n < n0, where n0is a suitable constant.

• For 1 # n < n0, we have “"(1)” # cn3, if we pick c big enough.

This bound is not tight!


A tighter upper bound?

We shall prove that T(n) = O(n2).




Assume that T(k) # ck2 for k < n:

)(

)2/(4)2/(4)(

2

2

2

nOncn

nncnnTnT

(.(

.#.(






)(

)2/(4)2/(4)(

2

2

2

nOncn

nncnnTnT

(.(

.#.(

Wrong! We must prove the I.H.


for k=n





)(

)2/(4)2/(4)(

2

2

2

nOncn

nncnnTnT

(.(

.#.(

Wrong! We must prove the I.H.

2

2 )(cn

ncn#

**(for no choice of c > 0. Lose!

[ desired – residual ]


for k=n


A tighter upper bound!IDEA: Strengthen the inductive hypothesis.• Subtract a low-order term.Inductive hypothesis: T(k) # c1k2 – c2k for k < n.



T(n) = 4T(n/2) + n= 4(c1(n/2)2 – c2(n/2)) + n= c1n2 – 2c2n + n= c1n2 – c2n – (c2n – n)# c1n2 – c2n if c2 $ 1.


=

should be ≤



T(n) = 4T(n/2) + n= 4(c1(n/2)2 – c2(n/2)) + n= c1n2 – 2c2n + n= c1n2 – c2n – (c2n – n)# c1n2 – c2n if c2 $ 1.

Pick c1 big enough to handle the initial conditions.


should be ≤

=


Recursion-tree method

• A recursion tree models the costs (time) of a recursive execution of an algorithm.

• The recursion-tree method can be unreliable, just like any method that uses ellipses (…).

• The recursion-tree method promotes intuition, however.

• The recursion tree method is good for generating guesses for the substitution method.


Example of recursion treeSolve T(n) = T(n/4) + T(n/2) + n2:



T(n)



T(n/4) T(n/2)

n2



n2

(n/4)2 (n/2)2

T(n/16) T(n/8) T(n/8) T(n/4)



n2

(n/16)2 (n/8)2 (n/8)2 (n/4)2

(n/4)2 (n/2)2

"(1)

…



n2 2n

(n/16)2 (n/8)2 (n/8)2 (n/4)2

(n/4)2 (n/2)2

"(1)

…



(n/16)2 (n/8)2 (n/8)2 (n/4)2

(n/4)2 (n/2)2

"(1)

…

2165 n

2nn2



(n/16)2 (n/8)2 (n/8)2 (n/4)2

(n/4)2

"(1)

…

2165 n

2n

225625 n

n2

(n/2)2

…



(n/16)2 (n/8)2 (n/8)2 (n/4)2

(n/4)2

"(1)

…

2165 n

2n

225625 n

' / ' /' / 1 31652

165

1652 "....n

= "(n2)

…

Total =

n2

(n/2)2

geometric series


The master method

The master method applies to recurrences of the form

T(n) = a T(n/b) + f (n) , where a $ 1, b > 1, and f is asymptotically positive.


Three common casesCompare f (n) with nlogba:1. f (n) = O(nlogba – 0) for some constant 0 > 0.

• f (n) grows polynomially slower than nlogba

(by an n0 factor).Solution: T(n) = "(nlogba) .


Three common casesCompare f (n) with nlogba:1. f (n) = O(nlogba – 0) for some constant 0 > 0.

• f (n) grows polynomially slower than nlogba

(by an n0 factor).Solution: T(n) = "(nlogba) .

2. f (n) = "(nlogba lgkn) for some constant k $ 0.• f (n) and nlogba grow at similar rates.Solution: T(n) = "(nlogba lgk+1n) .

CLRS: k=0


Three common cases (cont.)Compare f (n) with nlogba:

3. f (n) = !(nlogba + 0) for some constant 0 > 0.• f (n) grows polynomially faster than nlogba (by

an n0 factor),and f (n) satisfies the regularity condition that a f (n/b) # c f (n) for some constant c < 1.Solution: T(n) = "( f (n)) .


Examples

EX. T(n) = 4T(n/2) + na = 4, b = 2 1 nlogba = n2; f (n) = n.CASE 1: f (n) = O(n2 – 0) for 0 = 1.2 T(n) = "(n2).


Examples

EX. T(n) = 4T(n/2) + na = 4, b = 2 1 nlogba = n2; f (n) = n.CASE 1: f (n) = O(n2 – 0) for 0 = 1.2 T(n) = "(n2).

EX. T(n) = 4T(n/2) + n2

a = 4, b = 2 1 nlogba = n2; f (n) = n2.CASE 2: f (n) = "(n2lg0n), that is, k = 0.2 T(n) = "(n2lg n).


Examples

EX. T(n) = 4T(n/2) + n3

a = 4, b = 2 1 nlogba = n2; f (n) = n3.CASE 3: f (n) = !(n2 + 0) for 0 = 1and 4(n/2)3 # cn3 (reg. cond.) for c = 1/2.2 T(n) = "(n3).


Examples

EX. T(n) = 4T(n/2) + n3

a = 4, b = 2 1 nlogba = n2; f (n) = n3.CASE 3: f (n) = !(n2 + 0) for 0 = 1and 4(n/2)3 # cn3 (reg. cond.) for c = 1/2.2 T(n) = "(n3).

EX. T(n) = 4T(n/2) + n2/lgna = 4, b = 2 1 nlogba = n2; f (n) = n2/lgn.Master method does not apply. In particular, for every constant 0 > 0, we have n0 (3,(lgn).


f (n/b)

Idea of master theoremRecursion tree:

f (n/b) f (n/b)

4 (1)

…

…f (n) a

f (n/b2)f (n/b2) f (n/b2)…a


f (n/b)


f (n/b) f (n/b)

4 (1)

…

…f (n) a

f (n/b2)f (n/b2) f (n/b2)…a

f (n)

a f (n/b)

a2 f (n/b2)

…


f (n/b)

Idea of master theorem

f (n/b) f (n/b)

4 (1)

…Recursion tree:

…f (n) a

f (n/b2)f (n/b2) f (n/b2)…ah = logbn

f (n)

a f (n/b)

a2 f (n/b2)

…


nlogba4 (1)

f (n/b)

Idea of master theorem

f (n/b) f (n/b)

4 (1)

…Recursion tree:

…f (n) a


f (n)

a f (n/b)

a2 f (n/b2)

#leaves = ah

= alogbn

= nlogba

…


f (n/b)


f (n/b) f (n/b)

4 (1)

…

…f (n) a


f (n)

a f (n/b)

a2 f (n/b2)

CASE 1: The weight increases geometrically from the root to the leaves. The leaves hold a constant fraction of the total weight.

CASE 1: The weight increases geometrically from the root to the leaves. The leaves hold a constant fraction of the total weight.

"(nlogba)

nlogba4 (1)

…


f (n/b)


f (n/b) f (n/b)

4 (1)

…

…f (n) a


f (n)

a f (n/b)

a2 f (n/b2)

CASE 2: (k = 0) The weight is approximately the same on each of the logbn levels.

CASE 2: (k = 0) The weight is approximately the same on each of the logbn levels.

"(nlogbalg n)

nlogba4 (1)

…


f (n/b)


f (n/b) f (n/b)

4 (1)

…

…f (n) a


f (n)

a f (n/b)

a2 f (n/b2)

…CASE 3: The weight decreases geometrically from the root to the leaves. The root holds a constant fraction of the total weight.

CASE 3: The weight decreases geometrically from the root to the leaves. The root holds a constant fraction of the total weight.

nlogba4 (1)

"( f (n))


Appendix: geometric series

1

111

2x

xxxxn

n*

*(.....

" for x 5 1

1

11 2x

xx*

(... " for |x| < 1

Return to last slide viewed.

Asymptotic Notation and Recurrences - Uppsala University€¦ · Asymptotic notation. O. bounds):...

Documents

Transcript of Asymptotic Notation and Recurrences - Uppsala University€¦ · Asymptotic notation. O. bounds):...