9/14/2016

1

• A more intricate example is the recurrence( ) = ( /3) + ( /3) + ( )

• represents the constant in the term• Adding the values across the levels of the

recursion tree yields value for every level• The longest simple path from the root to a

leaf is (2 3) (2 3) 1• (2 3) = 1 when = log /

• The height of the tree is log /

14-Sep-16MAT-72006 AADS, Fall 2016 116

14-Sep-16MAT-72006 AADS, Fall 2016 117

9/14/2016

2

• We expect the solution to the recurrence tobe number of levels × cost of each level:

( log ) = ( lg )• Not every level contributes a cost of• If this recursion tree were a complete binary

tree of height log , there would be2 = leaves

• The cost of each leaf is a constant• The total cost of all leaves would then be

( ) which, since log 2 > 1, is( lg )

14-Sep-16MAT-72006 AADS, Fall 2016 118

• This recursion tree is not a complete binary tree,however, and has fewer than leaves

• As we go down from the root, more and moreinternal nodes are absent

• Consequently, levels toward the bottomcontribute less than to the total cost

• we can use the substitution method to verify that( lg ) is an upper bound for the solution to

the recurrence

14-Sep-16MAT-72006 AADS, Fall 2016 119

9/14/2016

3

4.5 The master method

Theorem 4.1 (Master theorem)

Let 1 and > 1, and ( ) a function, ( )is de ned on nonneg. integers by recurrence

= ( ) + ,where we / means either or .

( ) has the following asymptotic bounds:

14-Sep-16MAT-72006 AADS, Fall 2016 120

1. If = ( ) for some constant > 0,then ( ) =

2. If ( ) = ,then ( ) = lg

3. If ( ) = for some constant > 0,and if ( ) ( ) for some constant < 1and all suf ciently large ,

then ( ) = ( )

14-Sep-16MAT-72006 AADS, Fall 2016 121

9/14/2016

4

• In each of the three cases, we compare thefunction ( ) with the function

• The larger of the two determines the solutionto the recurrence– In case 1, the function is the larger,

then the solution is ( ) =– In case 3, the function ( ) is the larger, then

the solution is ( ) = ( )– In case 2, the two functions are the same

size, we multiply by a logarithmic factor, andthe solution is ( ) = lg

14-Sep-16MAT-72006 AADS, Fall 2016 122

• Be aware of some technicalities:• In case 1, not only must ( ) be smaller than

, it must be polynomially smaller• ( ) must be asymptotically smaller than

by a factor of for some > 0• In case 3, not only must ( ) be larger than

, it also must be polynomially largerand satisfy the “regularity” condition that

( ) ( )14-Sep-16MAT-72006 AADS, Fall 2016 123

9/14/2016

5

Using the master method

( ) = ( 3) +

• We have = 9, = 3, ( ) = , and thus= ( )

• Since ( ) = ( ), where = 1, wecan apply case 1 of the master theorem andconclude that the solution is ( )

14-Sep-16MAT-72006 AADS, Fall 2016 124

( ) = (2 3) + 1

• Here = 1, = 3 2, ( ) = 1, and == = 1

• Case 2 applies, since =(1), and thus the solution to the recurrence

is (lg )

14-Sep-16MAT-72006 AADS, Fall 2016 125

9/14/2016

6

( ) = ( 4) + lg

• We have = 3, = 4, = lg , and= = ( . )

• Since ( ) = ( ), where 0.2,case 3 applies provided that the regularitycondition holds for ( )

• For suf ciently large , we have that( ) = 3( 4) lg( 4)

(3 4) lg = ( ) for = 3 4• By case 3, the solution is ( lg )

14-Sep-16MAT-72006 AADS, Fall 2016 126

• The master method does not apply to( ) = ( 2) + lg

even if it seems to have the proper form:= 2, = 2, = lg , and =

• One might think that case 3 applies, sinceis asymptotically larger than

• It is, however, not polynomially larger: ratio= lg is asymptotically less than

for any positive constant• Consequently, the recurrence falls into the

gap between case 2 and case 314-Sep-16MAT-72006 AADS, Fall 2016 127

9/14/2016

7

• The recurrence of the running time of thesimple divide-and-conquer algorithm formatrix multiplication

( ) = ( 2) ( )

• Now = 8, = 2, and and so= =

• Since is polynomially larger than– i.e., = for = 1case 1 applies, and ( ) =

14-Sep-16MAT-72006 AADS, Fall 2016 128

• The recurrence of the running time ofStrassen’s algorithm

( ) = ( 2) ( )

• Here we have = 7, = 2, andand so = =

• Recalling that 2.80 < lg 7 < 2.81, we see that= O( ) for = 0.8

• Again, case 1 applies, and we have thesolution ( ) =

14-Sep-16MAT-72006 AADS, Fall 2016 129

9/14/2016

8

5 Probabilistic Analysis andRandomized Algorithms

• We need to hire a new of ce assistant with thehelp of an employment agency that sends us acandidate each day

• We interview that person and then decide eitherto hire her or not

• We pay the agency a fee for each interview• To actually hire an applicant is more costly

– We must re the current of ce assistant and– pay a substantial hiring fee to the agency

14-Sep-16MAT-72006 AADS, Fall 2016 130

• We are committed to having, at all times, thebest possible person for the job

• After interviewing an applicant, if she is betterquali ed than the current assistant, we will

re the current of ce assistant and hire thenew applicant

• We are willing to pay the price of this strategy,but wish to estimate what that price will be

14-Sep-16MAT-72006 AADS, Fall 2016 131

5.1 The hiring problem

9/14/2016

9

HIRE-ASSISTANT( )

1. best 0 // candidate 0 is a least-quali ed dummy candidate

2. for 1 to3. interview candidate4. if candidate is better than best5. best6. hire candidate

14-Sep-16MAT-72006 AADS, Fall 2016 132

• Interviewing has a low cost, , whereas hiring isexpensive, costing

• Letting be the number of people hired, thetotal cost of the algorithm is ( + )

• No matter how many people we hire, we alwaysinterview candidates and thus always incur thecost

• Let us concentrate on analyzing , the hiringcost

• This quantity varies with each run of thealgorithm

14-Sep-16MAT-72006 AADS, Fall 2016 133

9/14/2016

10

Worst-case analysis

• In the worst case, we hire every candidate thatwe interview– candidates come in increasing order of quality– we hire times, for a total hiring cost of

• We have no idea about the order in which thecandidates arrive, nor do we have any controlover this order

• It is natural to ask what we expect to happen in atypical or average case

14-Sep-16MAT-72006 AADS, Fall 2016 134

Probabilistic analysis

• We can assume that the applicants come in arandom order

• Assume that we can compare any twocandidates and decide the better quali ed

there is a total order on the candidates• Rank each candidate with a unique number

1, … ,• Use ) to denote the rank of applicant

14-Sep-16MAT-72006 AADS, Fall 2016 135

9/14/2016

11

• A higher rank corresponds to a better quali edapplicant

• Ordered list 1 , 2 , … , ( ) is apermutation of the list 1,2, … ,

• “The applicants come in a random order”– “this list of ranks is equally likely to be any

one of the ! permutations of the numbers 1through ”

• Alternatively, the ranks form a uniform randompermutation; i.e., each of the possible !permutations appears with equal probability

14-Sep-16MAT-72006 AADS, Fall 2016 136

Randomized algorithms

• We call an algorithm randomized if its behavioris determined also by values produced by arandom-number generator RANDOM

• A call to RANDOM( , ) returns an integerbetween and , inclusive, with each suchinteger being equally likely

• E.g., RANDOM(0,1) produces 0 with probability1 2, and 1 with probability 1 2

14-Sep-16MAT-72006 AADS, Fall 2016 137

9/14/2016

12

• A call to RANDOM 3,7 returns either 3,4,5,6,7,each with probability 1 5

• Each integer returned by RANDOM isindependent of the integers returned on previouscalls

• In analyzing the running time of a randomizedalgorithm, we take the expectation of the runningtime over the distribution of values returned bythe RANDOM

• We refer to the running time of a randomizedalgorithm as an expected running time

14-Sep-16MAT-72006 AADS, Fall 2016 138

5.2 Indicator random variables

• Indicator random variables provide a convenientmethod for converting between probabilities andexpectations

• Suppose we are given a sample space and anevent

• Then the indicator random variable { }associated with is de ned as

= 1 if occurs0 if does not occur

14-Sep-16MAT-72006 AADS, Fall 2016 139

9/14/2016

13

• Let us determine the expected number of headsthat we obtain when ipping a fair coin

• Our sample space is = { , }, withPr = Pr = 1 2

• We de ne an indicator random variable ,associated with the coin coming up heads ( )

• This variable counts the number of headsobtained in this ip

= = 1 if H occurs0 if T occurs

14-Sep-16MAT-72006 AADS, Fall 2016 140

• Recall that E = Pr{ = }• The expected number of heads obtained in one

ip of the coin is simply the expected value ofour indicator variable

E = E= 1 Pr + 0 Pr= 1 (1 2) + 0 (1 2)= 1 2

• Thus the expected number of heads obtained byone ip of a fair coin is 1 2

14-Sep-16MAT-72006 AADS, Fall 2016 141

9/14/2016

14

• The expected value of an indicator randomvariable associated with an event is equal tothe probability that occurs

Lemma 5.1 Given a sample space and anevent in , let = { }. Then E = Pr{ }

Proof By the de nition of an indicator and thede nition of expected value, we have =

= 1 Pr + 0 Pr = Pr , wheredenotes , the complement of

14-Sep-16MAT-72006 AADS, Fall 2016 142

• Compute the number of heads in coin ips• Let be the indicator associated with the event

in which the th ip comes up heads:= {the th flip results in the event }

• Let denote the total number of heads in thecoin ips, so that

=

• To compute the expected number of heads, takethe expectation of both sides to obtain

E = E

14-Sep-16MAT-72006 AADS, Fall 2016 143

9/14/2016

15

• Linearity of expectationE + = E + E

makes the use of indicator random variables apowerful analytical technique– it applies even when there is dependence

among the variables• We now can easily compute the expected

number of heads:

E = E = E

1 2 = 2

14-Sep-16MAT-72006 AADS, Fall 2016 144

Analysis of the hiring problem

• Let be the random variable whose valuenumber of times we hire a new assistant

• let be the indicator random variableassociated with the event in which the thcandidate is hired

= candidate is hired

= 1 if candidate is hired0 if candidate is not hired

• = + + +14-Sep-16MAT-72006 AADS, Fall 2016 145

9/14/2016

16

• By Lemma 5.1E = Pr candidate is hired

and we must compute the probability that lines5–6 of HIRE-ASSISTANT are executed

• Candidate is hired (line 6) exactly when she isbetter than each of candidates 1, … , 1

• We assume that the candidates arrive in arandom order: the rst candidates haveappeared in a random order

• Any one of these rst candidates is equallylikely to be the best-quali ed so far

14-Sep-16MAT-72006 AADS, Fall 2016 146

• Candidate has a probability of 1/ of beingbetter quali ed than candidates 1, … , 1 andthus a probability of 1/ of being hired.

• By Lemma 5.1, we conclude that E = 1/• Now we can compute E :

E = E = E

= 1 = ln + (1)

by harmonic series

= 1 = ln + (1)

14-Sep-16MAT-72006 AADS, Fall 2016 147

9/14/2016

17

• We interview people, but actually hire onlyapprox. ln of them, on average

• The following lemma summarize this

Lemma 5.2 Assuming that candidates come inrandom order, algorithm HIRE-ASSISTANT has anaverage-case total hiring cost of ( ln )

Proof Follows immediately from our de nition ofthe hiring cost and the equation, which shows thatthe expected number of hires is approximately ln

14-Sep-16MAT-72006 AADS, Fall 2016 148

5.3 Randomized algorithms

• The algorithm above is deterministic• The number of times we hire a new of ce

assistant differs for different inputs• Given the rank list = 1,2,3,4,5,6,7,8,9,10 , a

new assistant is always hired 10 times• For the list of ranks = 10,9,8,7,6,5,4,3,2,1 , a

new assistant is hired only once• = 5,2,1,8,4,7,10,9,3,6 , a new assistant is

hired three times

14-Sep-16MAT-72006 AADS, Fall 2016 149

9/14/2016

18

• Consider an algorithm that rst permutes theinput before determining the best candidate

• Now we randomize in the algorithm, not in theinput distribution

• For input we cannot say how many times thebest is updated – it differs with each run

• Not even an enemy can produce a bad input,random permutation yields input order irrelevant

• The algorithm performs badly only if the random-number generator produces an “unlucky”permutation

14-Sep-16MAT-72006 AADS, Fall 2016 150

RANDOMIZED-HIRE-ASSISTANT( )

1. randomly permute the list of candidates2. best 0 // candidate 0 is a least-quali ed

// dummy candidate3. for 1 to4. interview candidate5. if candidate is better than candidate best6. best7. hire candidate

14-Sep-16MAT-72006 AADS, Fall 2016 151

9/14/2016

19

Randomly permuting arrays

• We assume that we are given an array which,w.l.o.g., contains the elements 1, 2, … ,

• Produce a random permutation of the array• Assign element [ ] a random priority [ ], and

sort the elements according to priorities• E.g., if our initial array is = 1,2,3,4 and we

choose random priorities = 36,3,62,19 , wewould produce array = 2,4,1,3

• We call this procedure PERMUTE-BY-SORTING

14-Sep-16MAT-72006 AADS, Fall 2016 152

PERMUTE-BY-SORTING( )

1. . length2. let [1. . ] be a new array3. for 1 to4. [ RANDOM(1, )5. sort , using as sort keys

• We use a range of 1 to for random numbersto make it likely that all the priorities in areunique

14-Sep-16MAT-72006 AADS, Fall 2016 153

9/14/2016

20

• It remains to prove that the procedure producesa uniform random permutation,– It is equally likely to produce every

permutation of the numbers 1 through

Lemma 5.4 PERMUTE-BY-SORTING produces auniform random permutation of the input,assuming that all priorities are distinct

Proof See the book.

14-Sep-16MAT-72006 AADS, Fall 2016 154

• It is better to permute the given array in place• RANDOMIZE-IN-PLACE does so in ( ) time• In its th iteration, it chooses the element [ ]

randomly from among elements through[ ]

• After the th iteration, is never altered

RANDOMIZE-IN-PLACE( )1 .2 for 1 to3 swap with [RANDOM( , )]

14-Sep-16MAT-72006 AADS, Fall 2016 155

9/14/2016

21

• A -permutation on a set of elements is asequence containing of the elements, withno repetitions

• The number of -permutations is ! !

loop invariant:• Just prior to the th iteration of the for loop of

lines 2–3, for each possible ( 1)-permutationof the elements, the subarray [1. . 1]contains this ( 1)-permutation with probability

+ 1 ! !

14-Sep-16MAT-72006 AADS, Fall 2016 156

Initialization: loop invariant trivially holds

Maintenance:• Consider a particular -permutation, and denote

the elements in it by , , … ,• This permutation consists of an ( 1)-

permutation , , … , followed by thevalue that the algorithm places in [ ]

• Let denote the event in which the rst ( 1)iterations have created the particular ( 1)-permutation , , … , in [1. . 1]

14-Sep-16MAT-72006 AADS, Fall 2016 157

9/14/2016

22

• By the loop invariant, Pr = + 1 ! !• Let be the event that th iteration puts in

position [ ]• The -permutation , , … , appears in

1. . precisely when both and occurPr = Pr Pr{ }

• Pr = 1 + 1 because in line 3 thealgorithm chooses randomly from the +1 values in positions . .

Pr =1

+ 1+ 1 !!

=!

!14-Sep-16MAT-72006 AADS, Fall 2016 158

Termination:• At termination, = + 1, and we have that the

subarray 1. . is a given -permutation withprobability

+ 1 + 1 ! ! = 0! ! = 1 !

• Thus, RANDOMIZE-IN-PLACE produces a uniformrandom permutation

14-Sep-16MAT-72006 AADS, Fall 2016 159

9/14/2016

23

5.4.1 The birthday paradox

• How many people must there be in a roombefore there is a 50% chance that two of themwere born on the same day of the year?

• The answer is surprisingly few• The paradox is that it is in fact far fewer

– than the number of days in a year, or– even half the number of days in a year

14-Sep-16MAT-72006 AADS, Fall 2016 160

An analysis using indicatorrandom variables

• We use indicator random variables to provide asimple but approximate analysis of the birthdayparadox

• For each pair ( , ) of the people in the room,de ne the indicator random variable , for

< , by= { and have the same birthday}

= 1 and have the same birthday0 otherwise

14-Sep-16MAT-72006 AADS, Fall 2016 161

9/14/2016

24

• Once birthday for is chosen, the probabilitythat is chosen to be the same day is 1 ,where = 365

E = Pr and have the same birthday = 1

• Let be a random variable counting the numberof pairs of individuals having the same birthday

=

14-Sep-16MAT-72006 AADS, Fall 2016 162

• Taking expectations of both sides and applyinglinearity of expectation, we obtain

E = E = [ ]

= 21

=( 1)

• When ( 1) 2 , the expected number ofpairs of people with the same birthday is at least1

14-Sep-16MAT-72006 AADS, Fall 2016 163

9/14/2016

25

• Thus, if we have at least + 1 individuals in aroom, we can expect at least two to have thesame birthday

• For = 365, if = 28, the expected number ofpairs with the same birthday is

(28 27) (2 365) 1.0356• With at least 28 people, we expect to nd at

least one matching pair of birthdays• Mars has = 687 days, need = 38 aliens• Analysis using only probabilities gives a different

exact number of people, but sameasymptotically:

14-Sep-16MAT-72006 AADS, Fall 2016 164