Reccurence - Algorithm

8/6/2019 Reccurence - Algorithm

1/41

CSC2105: AlgorithmsCSC2105: Algorithms

RecurrencesRecurrences

&&

Master MethodMaster Method

Mashiour RahmanMashiour RahmanAssistant ProfessorAssistant Professor

[email protected]@aiub.edu

American International University BangladeshAmerican International University Bangladesh


2/41

Mashiour Rahman AIUB::CSC2105::Algorithm Recurrences 2

RecursionRecursion

AnAn objectobject is recursiveis recursive ifif

it containsit contains itselfitself as part of it, oras part of it, or

iit is defined in terms oft is defined in terms ofitselfitself..

Factorial:Factorial: n!n!

How do you compute 10!?How do you compute 10!?

n! = 1 * 2 * 3 *...* nn! = 1 * 2 * 3 *...* n

n! = n * (nn! = n * (n--1)!1)!


3/41


Factorial FunctionFactorial Function

Pseudocode of factorial:Pseudocode of factorial:

A recursive procedure includes aA recursive procedure includes a Termination condition (determines when andTermination condition (determines when and

how to stop the recursion).how to stop the recursion).

One (or more) recursive calls.One (or more) recursive calls.

fac1fac1INPUTINPUT:: nn a natural number.a natural number.OUTPUTOUTPUT:: n!n! (factorial of(factorial ofnn))

fac1(n)fac1(n)ifif n


4/41


Tracing the ExecutionTracing the Execution

fac(3)

fac(1)

1

fac(2)

2 * fac(1)

1

2

6

3 * fac(2)

fac1fac1

INPUTINPUT:: nn a natural number.a natural number.OUTPUTOUTPUT:: n!n! (factorial of(factorial ofnn))

fac1(n)fac1(n)ifif n


5/41


BookkeepingBookkeeping

fac(5) 5*fac(4)

fac(4) 4*fac(3)

fac(3) 3*fac(2)

fac(2) 2*fac(1)

fac(1) 1 = 1

1 = 2

2 = 6

6 = 24

24 = 120

The computer maintains an activation stack for activeThe computer maintains an activation stack for activeprocedure calls (procedure calls ( compiler construction). Example forcompiler construction). Example forfac(5).fac(5).


6/41


Variants ofFactorialVariants ofFactorial

fac2fac2

INPUTINPUT: n: n a natural number.a natural number.OUTPUTOUTPUT: n! (factorial of n): n! (factorial of n)

facfac22(n)(n)

ifif nn 22 thenthen returnreturn 11returnreturn n * fac2(nn * fac2(n--1)1)

fac3fac3INPUTINPUT: n: n a natural number.a natural number.OUTPUTOUTPUT: n! (factorial of n): n! (factorial of n)

facfac33(n)(n)returnreturn n * fac3(nn * fac3(n--1)1)

ifif nn ee22 thenthen returnreturn 11

fac2 is correct: The returnfac2 is correct: The return

statement in the if clausestatement in the if clause

terminates the function and,terminates the function and,

thus, the entire recursion.thus, the entire recursion.

fac3 is incorrect: Infinitefac3 is incorrect: Infinite

recursion. The terminationrecursion. The termination

condition is never reached.condition is never reached.


7/41


Mutual RecursionMutual Recursion

Recursion doesRecursion does notnot always occur because aalways occur because a

procedureprocedure calls itself.calls itself.

Mutual recursionMutual recursion occurs ifoccurs if twotwo proceduresprocedures callcall

each other.each other.

B A


8/41


Mutual Recursion ExampleMutual Recursion Example

Problem: Determine whether a givenProblem: Determine whether a given naturalnatural

numbernumber is even.is even.

Definition of even:Definition of even:

0 is even (let)0 is even (let)

N is odd if NN is odd if N--1 is1 is eveneven

N is even if NN is even if N--1 is odd1 is odd


9/41


ImplementationImplementation of evenof even

Can it be used to determine whether a numberCan it be used to determine whether a number

is odd?is odd?

evenevenINPUTINPUT: n: n a natural numbera natural number..OUTPUTOUTPUT:: true if n is even; false otherwisetrue if n is even; false otherwise

oddodd(n)(n)

ifif n = 0n = 0 then returnthen return TRUETRUEreturn !return !even(neven(n--1)1)

even(n)even(n)ifif n = 0n = 0 thenthen returnreturn TRUETRUEelse return !else return !odd(nodd(n--1)1)


10/41


The divideThe divide-- andand-- conquer designconquer design

paradigmparadigm

Divide and conquer is just one of several Divide and conquer is just one of several

powerful techniques for algorithm design.powerful techniques for algorithm design. Divide Divide-- andand-- conquer algorithms can beconquer algorithms can be

analyzed using recurrences and the masteranalyzed using recurrences and the master

method (so practice this math).method (so practice this math).

Can lead to more efficient algorithms Can lead to more efficient algorithms


11/41


RecurrencesRecurrences

Running times of algorithms withRunning times of algorithms with recursiverecursivecallscalls can be described using recurrences.can be described using recurrences.

AA recurrencerecurrence is an equation or inequality thatis an equation or inequality that

describes a function in terms of its value ondescribes a function in terms of its value onsmaller inputs.smaller inputs.

For Fordivide and conquerdivide and conqueralgorithms:algorithms:

Example: Merge SortExample: Merge Sort (1) i 1( )

2 ( / 2) ( ) i 1

nT n

T n n n

5 !!

5 "

solving_trivial_problem if 1

( ) num_pieces ( / subproblem_size_factor) dividing combining if 1

nn

n n

!

!


12/41



paradigmparadigm

1.1. DivideDivide the problem (instance) intothe problem (instance) into

subproblems.subproblems.

2.2. ConquerConquerthe subproblems by solving themthe subproblems by solving themrecursively.recursively.

3.3. CombineCombine subproblem solutions.subproblem solutions.


13/41


The divideThe divide-- andand-- conquerconquer

design paradigmdesign paradigmExample:Example: merge sortmerge sort

1.1. Divide:Divide: Trivial (array is halved).Trivial (array is halved).

2.2. Conquer:Conquer: Recursively sort 2 subarrays.Recursively sort 2 subarrays.

3.3. Combine:Combine: LinearLinear-- time merge.time merge.

Recurrence for merge sortRecurrence for merge sort

Let T(Let T( nn ) = Time required for size) = Time required for size nn

T(T( nn )) ==

*

+

22 T( n / 2)T( n / 2) O( n)O( n)* +Number Of

Subproblems

Time required for

each SubproblemWork Dividing

and Combining

Sub-

problemSize (n/2)


14/41



paradigmparadigm

Binary searchBinary search

Find an element in a sorted array:Find an element in a sorted array:

1. Divide:1. Divide: Check middle element.Check middle element.2. Conquer:2. Conquer: Recursively search 1 subarray.Recursively search 1 subarray.

3. Combine:3. Combine: Trivial.Trivial.

Recurrence for binary searchRecurrence for binary search

T( n) =T( n) =

* +

11 T( n / 2)T( n / 2) (1)(1)* +Number Of

Subproblems

Time required for

each SubproblemWork Dividing

and Combining


15/41


Solving RecurrencesSolving Recurrences

Repeated (backward) substitution methodRepeated (backward) substitution method

Expanding the recurrence by substitution andExpanding the recurrence by substitution and

noticing a pattern (this is not a strictly formal proof).noticing a pattern (this is not a strictly formal proof).

Substitution methodSubstitution method guessing the solutionsguessing the solutions

verifying the solution by the mathematical inductionverifying the solution by the mathematical induction

RecursionRecursion--treestrees Master methodMaster method

templates for different classes of recurrencestemplates for different classes of recurrences


16/41


Repeated SubstitutionRepeated Substitution Lets find the running time of the merge sortLets find the running time of the merge sort

1 if 1( )

2 ( / 2) if 1

nn

n n n

!!

T(n) = 2 T(n/2)T(n/2) + n substitutesubstitute

= 2(2T(n/4) + n/22T(n/4) + n/2) + n expandexpand= 22T(n/4)T(n/4) + 2n substitutesubstitute

= 22(2T(n/8) + n/42T(n/8) + n/4) + 2n expandexpand

= 23T(n/8)T(n/8) + 3n observe patternobserve pattern

= 23T(n/23) + 3n observe patternobserve pattern

T( n ) = 2kT(n/2k) + kn= n T(n/n) + nlg n= n + nlg n

assumeassume 22kk--11< n < n 22kk..

So upper bound forkis

n = 22kk

KK== loglog22n =n = lglg nn

T(n) = O(nlgn)


17/41


Repeated Substitution MethodRepeated Substitution Method

The procedure is straightforward:The procedure is straightforward:

SubstituteSubstitute,, ExpandExpand,, SubstituteSubstitute,, ExpandExpand, ,

Observe aObserve apatternpattern and determine the expression afterand determine the expression after

thethe kk--th substitution.th substitution.

Find out what the highest value ofFind out what the highest value ofkk(number of iterations,(number of iterations,

e.g.,e.g., lglg nn) should be to get to the base case of the) should be to get to the base case of the

recurrence (e.g.,recurrence (e.g., T(1)T(1)).).

Insert the value ofInsert the value ofT(1)T(1) and the expression ofand the expression ofkkinto yourinto your

expression.expression.


18/41


2 if 1( ) 2 ( / 2) 2 3 if 1

n

nn n n

!!


19/41


Another ExampleAnother Example2 if 1

( )2 ( / 2) 2 3 if 1

nn

n n n

!!

T(n) = 2 T(n/2)T(n/2) + 2n + 3 substitutesubstitute

= 2(2T(n/4) + n + 32T(n/4) + n + 3) + 2n +3 expandexpand

= 22T(n/4)T(n/4) + 4n + 2x3 + 3 substitutesubstitute= 22(2T(n/8) + n/2 + 32T(n/8) + n/2 + 3) + 4n + 2x3 + 3 expandexpand

= 23T(n/23) + 2x3n + 3x(22+21+20)observe patternobserve pattern

T(n) = 2 kT(n/2k) + 2nk + 3

= 2kT(n/2k) + 2nk + 3 (2k-1)=nT(n/n) + 2nlg n + 3(n - 1)= 2n+2nlg n + 3n - 3= 5n + 2nlg n 3

!

1k

0j

j2

assumeassume 22kk--11< n < n 22kk..

So upper bound forkisn = 22kk

KK== loglog22n =n = lglg nn

T(n) = O(nlgn)


20/41


Substitution MethodSubstitution Method

The substitution method to solve recurrencesThe substitution method to solve recurrences

entails two steps:entails two steps:

GuessGuess the solution.the solution.

UseUse inductioninduction to prove the solution.to prove the solution.

Example:Example:

T(n) = 4T(n/2) + nT(n) = 4T(n/2) + n


21/41


Substitution MethodSubstitution Method

T(n) = 4T(n/2) + nT(n) = 4T(n/2) + n

1) Guess T(n) = O(n3), i.e., T(n) is of the form cn3

2) Prove T(n)e cn3by induction

T(n)T(n) = 4T(n/2) + n= 4T(n/2) + n recurrencerecurrence

ee 4c(n/2)4c(n/2)33+ n+ n induction hypothesisinduction hypothesis

= 0.5cn= 0.5cn33+ n+ n simplifysimplify

= cn= cn33 (0.5cn(0.5cn33 n)n) rearrangerearrange

ee cncn33 if c>=2 and n>=1if c>=2 and n>=1

ThusT(n) = O(n3)


22/41

Tighter bound forT(n) = 4T(n/2) + n:Tighter bound forT(n) = 4T(n/2) + n:

Try to show T(n) = O(nTry to show T(n) = O(n22))



23/41


...Substitution Method...Substitution Method

Tighter bound forT(n) = 4T(n/2) + n:Tighter bound forT(n) = 4T(n/2) + n:

Try to show T(n) = O(nTry to show T(n) = O(n22))

Prove T(n)Prove T(n) ee cncn22 by inductionby induction

T(n) = 4T(n/2) + nT(n) = 4T(n/2) + n

ee4c(n/2)4c(n/2)

22

+ n+ n= cn= cn22 + n+ n

ee cncn22


24/41



What is the problem? RewritingWhat is the problem? Rewriting

T(n) = O(nT(n) = O(n22) = cn) = cn22+ (something positive)+ (something positive)

AsAs T(n)T(n)ee cncn22 does not work with the inductivedoes not work with the inductive

proof.proof.

Solution:Solution: Strengthen the hypothesisStrengthen the hypothesis for thefor theinductive proof:inductive proof:

T(n)T(n) ee ((answer you wantanswer you want)) -- (something > 0)(something > 0)


25/41



Fixed proof: strengthen the inductiveFixed proof: strengthen the inductivehypothesis by subtracting lowerhypothesis by subtracting lower--order terms:order terms:

ProveProve T(n)T(n)ee cncn22-- dndn by inductionby induction

T(n) = 4T(n/2) + nT(n) = 4T(n/2) + n

ee 4(c(n/2)4(c(n/2)22 d(n/2)) + nd(n/2)) + n

= cn= cn22

2dn + n2dn + n= (cn= (cn22 dn)dn) (dn(dn n)n)

ee cncn22 dn if ddn if d uu


26/41


Recursion TreeRecursion Tree

A recursion tree is a convenient way to visualize whatA recursion tree is a convenient way to visualize whathappens when a recurrence is iterated.happens when a recurrence is iterated.

Good for guessing asymtotic solutions to recurrencesGood for guessing asymtotic solutions to recurrences

2( ) ( / 4) ( / 2)T n T n T n n!


27/41


...Recursion Tree...Recursion Tree


28/41



The idea is to solve a class of recurrences that have the formThe idea is to solve a class of recurrences that have the form

T(n) = aT(n/b) + f(n)T(n) = aT(n/b) + f(n)

Assumptions:Assumptions: aa uu 11 andand bb >> 11, and, and f(n)f(n) is asymptoticallyis asymptotically

positive.positive.

Abstractly speaking,Abstractly speaking, TT((nn)) is the runtime for an algorithm andis the runtime for an algorithm and

we know thatwe know that

aa number of subproblems of sizenumber of subproblems of size n/bn/b are solved recursively, each inare solved recursively, each intimetime TT((n/bn/b).).

f(n)f(n) is the cost of dividing the problem and combining the results. e.g.,is the cost of dividing the problem and combining the results. e.g.,

In mergeIn merge--sort .sort .( ) 2 ( / 2) ( )T n T n n! 5


29/41



nbalog

Split problem into a parts. There are

logbn levels. There are leaves.log logb bn aa n!

nbalog

nbalog


30/41



Iterating the recurrence (expanding the tree) yieldsIterating the recurrence (expanding the tree) yields

T(n) = f(n) + aT(n/b)T(n) = f(n) + aT(n/b)

= f(n) + af(n/b) + a= f(n) + af(n/b) + a22T(n/bT(n/b22))

= f(n) + af(n/b) + a= f(n) + af(n/b) + a

22

f(n/bf(n/b

22

) + ) + aaloglogbbnn--11f(n/bf(n/bloglogbbnn--11) + a) + aloglogbbnnT(1)T(1)

T(n) =T(n) =

The first term is a division/recombination cost (totaled acrossThe first term is a division/recombination cost (totaled acrossall levels of the tree).all levels of the tree).

The second term isThe second term is thethe cost of doing all subproblems of sizecost of doing all subproblems of size1 (total of all work pushed to leaves).1 (total of all work pushed to leaves).

!

51nlog

0j

alogjjb

b )n()b/n(fa


31/41


Master Method, IntuitionMaster Method, Intuition

ThreeThree common cases:common cases:

1.1. Running time dominated by cost at leaves.Running time dominated by cost at leaves.

2.2. Running time evenly distributed throughout the tree.Running time evenly distributed throughout the tree.

3.3. Running time dominated by cost at the root.Running time dominated by cost at the root.

To solve the recurrence, we need to identify theTo solve the recurrence, we need to identify the

dominant term.dominant term.

In each case compare withIn each case compare with


32/41


Master Method, Case 1Master Method, Case 1

If for some constant thenIf for some constant then

f(n)f(n) grows polynomially slower than (bygrows polynomially slower than (by

factor ).factor ).

The work at the leaf level dominatesThe work at the leaf level dominates

Cost of all the leavesCost of all the leaves

log( ) ( )b af n O n I!logb an

0I "

nI


33/41



If thenIf then

andand are asymptotically the sameare asymptotically the same

The work is distributed equally throughoutThe work is distributed equally throughoutthethe treetree

(level cost)(number of levels)(level cost)(number of levels)

log( ) ( )b af n n! 5

( )f n logb an


34/41



If for some constant thenIf for some constant then

Inverse of Case 1Inverse of Case 1

f(n)f(n) grows polynomiallygrows polynomially fasterfaster thanthan

Also need a regularity conditionAlso need a regularity condition

The work at the root dominatesThe work at the root dominates

division/recombination costdivision/recombination cost

log( ) ( )b af n n I! ;

logb an

0 01 and 0 such that ( / ) ( )c n af n b cf n n n " "

0I "


35/41


MasterTheorem, SummarizedMasterTheorem, Summarized

Given: recurrence of the formGiven: recurrence of the form

log

log

log

log

log

0

1. ( )

( )

2. ( )

( ) lg

3. ( ) and ( / ) ( ), orsome 1,

( ) ( )

b

b

b

b

b

a

a

a

a

a

f n O n

T n n

f n n

T n n n

f n n af n b cf n c n n

T n f n

I

I

!

! 5

! 5

! 5

! ; " ! 5


36/41


StrategyStrategy

1.1. ExtractExtract aa,,bb,,andand f(n)f(n) from a given recurrencefrom a given recurrence

2.2. DetermineDetermine nnloglogbb aa

3.3. CompareCompare f(n)f(n) andand nnloglogbb aa asymptoticallyasymptotically

4.4. DetermineDetermine appropriateappropriate MT case and apply itMT case and apply it

Merge sort: T(n) = 2T(n/2) +Merge sort: T(n) = 2T(n/2) + 55(n)(n)

1.1. a=2, b=2, f(n) =a=2, b=2, f(n) = 55(n)(n)

2.2. nnloglog2222= n= n

3.3. 55(n) =(n) = 55((n)n)

Case 2: T(n) =Case 2: T(n) = 55((nnloglogbbaalog n) =log n) = 55(n log n)(n log n)


37/41


Examples ofExamples of Master MethodMaster Method

BinarySearch(A, l, r, q):

m := (l+r)/2

if A[m]=q then return m

else if A[m]>q then

BinarySearch(A, l, m-1, q)else BinarySearch(A, m+1, r, q)

T(n) = T(n/2) + 1T(n) = T(n/2) + 1

1.1. a=1, b=2, f(n) = 1a=1, b=2, f(n) = 12.2. nnloglog2211 = 1= 1

3.3. 1 =1 = 55((1)1)

Case 2: T(n) =Case 2: T(n) = 55(log n)(log n)


38/41


...Examples of...Examples of Master MethodMaster Method

T(n) = 9T(n/3) + nT(n) = 9T(n/3) + n

1.1. a=9, b=3, f(n) = na=9, b=3, f(n) = n

2. n2. nloglog3399

= n= n22

3.3. n =n = 55(n(nloglog3399 -- II)) withwith II= 1= 1

Case 1:Case 1: T(n) =T(n) = 55(n(n22))


39/41


...Examples of Master Method...Examples of Master Method

T(n) = 3T(n/4) + n log nT(n) = 3T(n/4) + n log n

1.1. a=3, b=4, f(n) = n log na=3, b=4, f(n) = n log n

2.2. nnloglog4433= n= n0.7920.792

3.3. n log n =n log n = ;;(n(nloglog443 +3 + II) with) with II= 0.208= 0.208

Case 3:Case 3:

regularity condition:regularity condition: af(n/b)


40/41


Is Recursion Necessary?Is Recursion Necessary?

TheoryTheory:: YouYou cancan alwaysalways resortresort toto iterationiteration andand

explicitlyexplicitly maintainmaintain aa recursionrecursion stackstack..

PracticePractice:: RecursionRecursion isis elegantelegant andand inin somesome casescases thethe

bestbest solutionsolution byby farfar..

RecursionRecursion isis nevernever appropriateappropriate ifif therethere existexist simplesimple

iterativeiterative solutionssolutions..

RecursionRecursion isis moremore expensiveexpensive thanthan correspondingcorresponding

iterativeiterative solutionssolutions sincesince bookkeepingbookkeeping isis necessarynecessary..


41/41


References & ReadingsReferences & Readings

CLRSCLRS Chapter: 4 (4.1Chapter: 4 (4.1--4.3)4.3)

4.4 for bedtime reading4.4 for bedtime reading

ExercisesExercises 4.1, 4.2, 4.34.1, 4.2, 4.3

ProblemsProblems 44--1, 41, 4--44

HSRHSR Chapter: 3 (3.1Chapter: 3 (3.1--3.6)3.6)

Examples: 3.1Examples: 3.1--3.53.5

Exercises: 3.1 (1, 2), 3.2 (1, 3Exercises: 3.1 (1, 2), 3.2 (1, 3--6),6),

Reccurence - Algorithm

Documents

Transcript of Reccurence - Algorithm