BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example:...

118

Transcript of BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example:...

Page 1: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

BBM402-Lecture 12: Randomized Algorithms

Lecturer: Lale Özkahya

Resources for the presentation:https://courses.engr.illinois.edu/cs473/fa2016/lectures.html

Page 2: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Outline

Randomization is very powerfulHow do you play R-P-S?Calculating insurance.

Our goalBasics of randomization – probability space, expectation, events,random variables, etc.

Randomized Algorithms – Two types

Las VegasMonte Carlo

Randomized Quick Sort

Chandra & Ruta (UIUC) CS473 3 Fall 2016 3 / 56

Page 3: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Outline

Randomization is very powerfulHow do you play R-P-S?Calculating insurance.

Our goalBasics of randomization – probability space, expectation, events,random variables, etc.

Randomized Algorithms – Two types

Las VegasMonte Carlo

Randomized Quick Sort

Chandra & Ruta (UIUC) CS473 3 Fall 2016 3 / 56

Page 4: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Part I

Introduction to RandomizedAlgorithms

Chandra & Ruta (UIUC) CS473 4 Fall 2016 4 / 56

Page 5: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Randomized Algorithms

Input x Output yDeterministic Algorithm

Input x Output yrRandomized Algorithm

random bits r

Chandra & Ruta (UIUC) CS473 5 Fall 2016 5 / 56

Page 6: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Randomized Algorithms

Input x Output yDeterministic Algorithm

Input x Output yrRandomized Algorithm

random bits r

Chandra & Ruta (UIUC) CS473 5 Fall 2016 5 / 56

Page 7: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Example: Randomized QuickSort

QuickSort ?1 Pick a pivot element from array

2 Split array into 3 subarrays: those smaller than pivot, thoselarger than pivot, and the pivot itself.

3 Recursively sort the subarrays, and concatenate them.

Randomized QuickSort1 Pick a pivot element uniformly at random from the array

2 Split array into 3 subarrays: those smaller than pivot, thoselarger than pivot, and the pivot itself.

3 Recursively sort the subarrays, and concatenate them.

Chandra & Ruta (UIUC) CS473 6 Fall 2016 6 / 56

Page 8: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Example: Randomized Quicksort

Recall: QuickSort can take Ω(n2) time to sort array of size n.

TheoremRandomized QuickSort sorts a given array of length n in O(n log n)expected time.

Note: On every input randomized QuickSort takes O(n log n) timein expectation. On every input it may take Ω(n2) time with somesmall probability.

Chandra & Ruta (UIUC) CS473 7 Fall 2016 7 / 56

Page 9: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Example: Randomized Quicksort

Recall: QuickSort can take Ω(n2) time to sort array of size n.

TheoremRandomized QuickSort sorts a given array of length n in O(n log n)expected time.

Note: On every input randomized QuickSort takes O(n log n) timein expectation. On every input it may take Ω(n2) time with somesmall probability.

Chandra & Ruta (UIUC) CS473 7 Fall 2016 7 / 56

Page 10: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Example: Randomized Quicksort

Recall: QuickSort can take Ω(n2) time to sort array of size n.

TheoremRandomized QuickSort sorts a given array of length n in O(n log n)expected time.

Note: On every input randomized QuickSort takes O(n log n) timein expectation. On every input it may take Ω(n2) time with somesmall probability.

Chandra & Ruta (UIUC) CS473 7 Fall 2016 7 / 56

Page 11: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Example: Verifying Matrix Multiplication

ProblemGiven three n× n matrices A, B, C is AB = C?

Deterministic algorithm:

1 Multiply A and B and check if equal to C.

2 Running time? O(n3) by straight forward approach. O(n2.37)with fast matrix multiplication (complicated and impractical).

Chandra & Ruta (UIUC) CS473 8 Fall 2016 8 / 56

Page 12: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Example: Verifying Matrix Multiplication

ProblemGiven three n× n matrices A, B, C is AB = C?

Deterministic algorithm:

1 Multiply A and B and check if equal to C.

2 Running time?

O(n3) by straight forward approach. O(n2.37)with fast matrix multiplication (complicated and impractical).

Chandra & Ruta (UIUC) CS473 8 Fall 2016 8 / 56

Page 13: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Example: Verifying Matrix Multiplication

ProblemGiven three n× n matrices A, B, C is AB = C?

Deterministic algorithm:

1 Multiply A and B and check if equal to C.

2 Running time? O(n3) by straight forward approach. O(n2.37)with fast matrix multiplication (complicated and impractical).

Chandra & Ruta (UIUC) CS473 8 Fall 2016 8 / 56

Page 14: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Example: Verifying Matrix Multiplication

ProblemGiven three n× n matrices A, B, C is AB = C?

Randomized algorithm:1 Pick a random n× 1 vector r.2 Return the answer of the equality ABr = Cr.3 Running time? O(n2)!

TheoremIf AB = C then the algorithm will always say YES. If AB 6= C thenthe algorithm will say YES with probability at most 1/2. Can repeatthe algorithm 100 times independently to reduce the probability of afalse positive to 1/2100.

Chandra & Ruta (UIUC) CS473 9 Fall 2016 9 / 56

Page 15: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Example: Verifying Matrix Multiplication

ProblemGiven three n× n matrices A, B, C is AB = C?

Randomized algorithm:1 Pick a random n× 1 vector r.2 Return the answer of the equality ABr = Cr.3 Running time?

O(n2)!

TheoremIf AB = C then the algorithm will always say YES. If AB 6= C thenthe algorithm will say YES with probability at most 1/2. Can repeatthe algorithm 100 times independently to reduce the probability of afalse positive to 1/2100.

Chandra & Ruta (UIUC) CS473 9 Fall 2016 9 / 56

Page 16: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Example: Verifying Matrix Multiplication

ProblemGiven three n× n matrices A, B, C is AB = C?

Randomized algorithm:1 Pick a random n× 1 vector r.2 Return the answer of the equality ABr = Cr.3 Running time? O(n2)!

TheoremIf AB = C then the algorithm will always say YES. If AB 6= C thenthe algorithm will say YES with probability at most 1/2. Can repeatthe algorithm 100 times independently to reduce the probability of afalse positive to 1/2100.

Chandra & Ruta (UIUC) CS473 9 Fall 2016 9 / 56

Page 17: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Example: Verifying Matrix Multiplication

ProblemGiven three n× n matrices A, B, C is AB = C?

Randomized algorithm:1 Pick a random n× 1 vector r.2 Return the answer of the equality ABr = Cr.3 Running time? O(n2)!

TheoremIf AB = C then the algorithm will always say YES. If AB 6= C thenthe algorithm will say YES with probability at most 1/2. Can repeatthe algorithm 100 times independently to reduce the probability of afalse positive to 1/2100.

Chandra & Ruta (UIUC) CS473 9 Fall 2016 9 / 56

Page 18: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Why randomized algorithms?

1 Many many applications in algorithms, data structures andcomputer science!

2 In some cases only known algorithms are randomized orrandomness is provably necessary.

3 Often randomized algorithms are (much) simpler and/or moreefficient.

4 Several deep connections to mathematics, physics etc.

5 . . .

6 Lots of fun!

Chandra & Ruta (UIUC) CS473 10 Fall 2016 10 / 56

Page 19: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Average case analysis vs Randomized algorithms

Average case analysis:

1 Fix a deterministic algorithm.

2 Assume inputs comes from a probability distribution.

3 Analyze the algorithm’s average performance over thedistribution over inputs.

Randomized algorithms:

1 Algorithm uses random bits in addition to input.

2 Analyze algorithms average performance over the given inputwhere the average is over the random bits that the algorithmuses.

3 On each input behaviour of algorithm is random. Analyzeworst-case over all inputs of the (average) performance.

Chandra & Ruta (UIUC) CS473 11 Fall 2016 11 / 56

Page 20: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Part II

Basics of Discrete Probability

Chandra & Ruta (UIUC) CS473 12 Fall 2016 12 / 56

Page 21: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Discrete Probability

We restrict attention to finite probability spaces.

DefinitionA discrete probability space is a pair (Ω, Pr) consists of finite set Ωof elementary events and function p : Ω→ [0, 1] which assigns aprobability Pr[ω] for each ω ∈ Ω such that

∑ω∈Ω Pr[ω] = 1.

Example

An unbiased coin. Ω = H, T and Pr[H] = Pr[T] = 1/2.

Example

A 6-sided unbiased die. Ω = 1, 2, 3, 4, 5, 6 and Pr[i] = 1/6 for1 ≤ i ≤ 6.

Chandra & Ruta (UIUC) CS473 13 Fall 2016 13 / 56

Page 22: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Discrete Probability

We restrict attention to finite probability spaces.

DefinitionA discrete probability space is a pair (Ω, Pr) consists of finite set Ωof elementary events and function p : Ω→ [0, 1] which assigns aprobability Pr[ω] for each ω ∈ Ω such that

∑ω∈Ω Pr[ω] = 1.

Example

An unbiased coin. Ω = H, T and Pr[H] = Pr[T] = 1/2.

Example

A 6-sided unbiased die. Ω = 1, 2, 3, 4, 5, 6 and Pr[i] = 1/6 for1 ≤ i ≤ 6.

Chandra & Ruta (UIUC) CS473 13 Fall 2016 13 / 56

Page 23: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Events

DefinitionGiven a probability space (Ω, Pr) an event is a subset of Ω. In otherwords an event is a collection of elementary events. The probabilityof an event A, denoted by Pr[A], is

∑ω∈A Pr[ω].

The complement event of an event A ⊆ Ω is the event Ω \ Afrequently denoted by A.

Example

A pair of independent dice. Ω = (i, j) | 1 ≤ i ≤ 6, 1 ≤ j ≤ 6.Let A be the event that the sum of the two numbers on the dice iseven.Then A =

(i, j) ∈ Ω

∣∣∣ (i + j) is even

.

Pr[A] = |A|/36 = 1/2.

Chandra & Ruta (UIUC) CS473 14 Fall 2016 14 / 56

Page 24: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Events

DefinitionGiven a probability space (Ω, Pr) an event is a subset of Ω. In otherwords an event is a collection of elementary events. The probabilityof an event A, denoted by Pr[A], is

∑ω∈A Pr[ω].

The complement event of an event A ⊆ Ω is the event Ω \ Afrequently denoted by A.

Example

A pair of independent dice. Ω = (i, j) | 1 ≤ i ≤ 6, 1 ≤ j ≤ 6.Let A be the event that the sum of the two numbers on the dice iseven.Then A =

(i, j) ∈ Ω

∣∣∣ (i + j) is even

.

Pr[A] = |A|/36 = 1/2.

Chandra & Ruta (UIUC) CS473 14 Fall 2016 14 / 56

Page 25: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Independent Events

DefinitionGiven a probability space (Ω, Pr) and two events A, B areindependent if and only if Pr[A ∩ B] = Pr[A] Pr[B]. Otherwisethey are dependent. In other words A, B independent implies onedoes not affect the other.

Example

Two coins. Ω = HH, TT, HT, TH andPr[HH] = Pr[TT] = Pr[HT] = Pr[TH] = 1/4.

1 A is the event that the first coin is heads and B is the eventthat second coin is tails. A, B are independent.

2 A is the event that both are not tails and B is event that secondcoin is heads. A, B are dependent.

Chandra & Ruta (UIUC) CS473 15 Fall 2016 15 / 56

Page 26: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Independent Events

DefinitionGiven a probability space (Ω, Pr) and two events A, B areindependent if and only if Pr[A ∩ B] = Pr[A] Pr[B]. Otherwisethey are dependent. In other words A, B independent implies onedoes not affect the other.

Example

Two coins. Ω = HH, TT, HT, TH andPr[HH] = Pr[TT] = Pr[HT] = Pr[TH] = 1/4.

1 A is the event that the first coin is heads and B is the eventthat second coin is tails. A, B are independent.

2 A is the event that both are not tails and B is event that secondcoin is heads. A, B are dependent.

Chandra & Ruta (UIUC) CS473 15 Fall 2016 15 / 56

Page 27: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Independent Events

DefinitionGiven a probability space (Ω, Pr) and two events A, B areindependent if and only if Pr[A ∩ B] = Pr[A] Pr[B]. Otherwisethey are dependent. In other words A, B independent implies onedoes not affect the other.

Example

Two coins. Ω = HH, TT, HT, TH andPr[HH] = Pr[TT] = Pr[HT] = Pr[TH] = 1/4.

1 A is the event that the first coin is heads and B is the eventthat second coin is tails. A, B are independent.

2 A is the event that both are not tails and B is event that secondcoin is heads.

A, B are dependent.

Chandra & Ruta (UIUC) CS473 15 Fall 2016 15 / 56

Page 28: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Independent Events

DefinitionGiven a probability space (Ω, Pr) and two events A, B areindependent if and only if Pr[A ∩ B] = Pr[A] Pr[B]. Otherwisethey are dependent. In other words A, B independent implies onedoes not affect the other.

Example

Two coins. Ω = HH, TT, HT, TH andPr[HH] = Pr[TT] = Pr[HT] = Pr[TH] = 1/4.

1 A is the event that the first coin is heads and B is the eventthat second coin is tails. A, B are independent.

2 A is the event that both are not tails and B is event that secondcoin is heads. A, B are dependent.

Chandra & Ruta (UIUC) CS473 15 Fall 2016 15 / 56

Page 29: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Union boundThe probability of the union of two events, is no bigger than the probability of thesum of their probabilities.

LemmaFor any two events E and F, we have that

Pr[E ∪ F

]≤ Pr

[E]

+ Pr[F].

Proof.Consider E and F to be a collection of elmentery events (which theyare). We have

Pr[E ∪ F

]=

x∈E∪FPr[x]

≤∑

x∈EPr[x] +

x∈FPr[x] = Pr

[E]

+ Pr[F].

Chandra & Ruta (UIUC) CS473 16 Fall 2016 16 / 56

Page 30: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Random Variables

DefinitionGiven a probability space (Ω, Pr) a (real-valued) random variable Xover Ω is a function that maps each elementary event to a realnumber. In other words X : Ω→ R.

DefinitionExpectation For a random variable X over a probability space(Ω, Pr) the expectation of X is defined as

∑ω∈Ω Pr[ω] X(ω). In

other words, the expectation is the average value of X according tothe probabilities given by Pr[·].

Chandra & Ruta (UIUC) CS473 17 Fall 2016 17 / 56

Page 31: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Random Variables

DefinitionGiven a probability space (Ω, Pr) a (real-valued) random variable Xover Ω is a function that maps each elementary event to a realnumber. In other words X : Ω→ R.

DefinitionExpectation For a random variable X over a probability space(Ω, Pr) the expectation of X is defined as

∑ω∈Ω Pr[ω] X(ω). In

other words, the expectation is the average value of X according tothe probabilities given by Pr[·].

Chandra & Ruta (UIUC) CS473 17 Fall 2016 17 / 56

Page 32: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Expectation

Example

A 6-sided unbiased die. Ω = 1, 2, 3, 4, 5, 6 and Pr[i] = 1/6 for1 ≤ i ≤ 6.

1 X : Ω→ R where X(i) = i mod 2. Then

E[X] =∑6

i=1 Pr[i] · X(i) = 16

∑6i=1 X(i) = 1/2.

2 Y : Ω→ R where Y(i) = i2. Then

E[Y] =∑6

i=116· i2 = 91/6.

Chandra & Ruta (UIUC) CS473 18 Fall 2016 18 / 56

Page 33: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Expectation

Example

A 6-sided unbiased die. Ω = 1, 2, 3, 4, 5, 6 and Pr[i] = 1/6 for1 ≤ i ≤ 6.

1 X : Ω→ R where X(i) = i mod 2. Then

E[X] =∑6

i=1 Pr[i] · X(i) = 16

∑6i=1 X(i) = 1/2.

2 Y : Ω→ R where Y(i) = i2. Then

E[Y] =∑6

i=116· i2 = 91/6.

Chandra & Ruta (UIUC) CS473 18 Fall 2016 18 / 56

Page 34: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Expected number of vertices?

Let G = (V, E) be a graph with n vertices and m edges. Let H bethe graph resulting from independently deleting every vertex of G withprobability 1/2. Compute the expected number of vertices in H.

(A) n/2.

(B) n/4.

(C) m/2.

(D) m/4.

(E) none of the above.

Chandra & Ruta (UIUC) CS473 19 Fall 2016 19 / 56

Page 35: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Expected number of vertices is:

Probability Space

Ω = 0, 1n. For ω ∈ 0, 1n, ωv = 1 if vertex v is presentin H, else is zero.

For each ω ∈ Ω, Pr[ω] = 12n .

X(ω) = # vertices in H as per ω = # 1s in ω.

E[X] =∑

ω∈Ω Pr[ω] X(ω)=

∑ω∈Ω

1/2nX(ω)= 1/2n

∑nk=0

(nk

)k

= 1/2n(2n n2)

= n/2

Chandra & Ruta (UIUC) CS473 20 Fall 2016 20 / 56

Page 36: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Expected number of vertices is:

Probability Space

Ω = 0, 1n. For ω ∈ 0, 1n, ωv = 1 if vertex v is presentin H, else is zero.

For each ω ∈ Ω, Pr[ω] = 12n .

X(ω) = # vertices in H as per ω = # 1s in ω.

E[X] =∑

ω∈Ω Pr[ω] X(ω)=

∑ω∈Ω

1/2nX(ω)= 1/2n

∑nk=0

(nk

)k

= 1/2n(2n n2)

= n/2

Chandra & Ruta (UIUC) CS473 20 Fall 2016 20 / 56

Page 37: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Expected number of vertices is:

Probability Space

Ω = 0, 1n. For ω ∈ 0, 1n, ωv = 1 if vertex v is presentin H, else is zero.

For each ω ∈ Ω, Pr[ω] = 12n .

X(ω) = # vertices in H as per ω = # 1s in ω.

E[X] =∑

ω∈Ω Pr[ω] X(ω)=

∑ω∈Ω

1/2nX(ω)= 1/2n

∑nk=0

(nk

)k

= 1/2n(2n n2)

= n/2

Chandra & Ruta (UIUC) CS473 20 Fall 2016 20 / 56

Page 38: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Expected number of edges?

Let G = (V, E) be a graph with n vertices and m edges. Let H bethe graph resulting from independently deleting every vertex of Gwith probability 1/2. The expected number of edges in H is

(A) n/2.

(B) n/4.

(C) m/2.

(D) m/4.

(E) none of the above.

Chandra & Ruta (UIUC) CS473 21 Fall 2016 21 / 56

Page 39: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Expected number of edges is:

Probability Space

Ω = 0, 1n. For ω ∈ 0, 1n, ωv = 1 if vertex v is presentin H, else is zero.

For each ω ∈ Ω, Pr[ω] = 12n .

X(ω) = # edges present in H as per ω = ??

How to compute E[X]?

Chandra & Ruta (UIUC) CS473 22 Fall 2016 22 / 56

Page 40: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Expected number of edges is:

Probability Space

Ω = 0, 1n. For ω ∈ 0, 1n, ωv = 1 if vertex v is presentin H, else is zero.

For each ω ∈ Ω, Pr[ω] = 12n .

X(ω) = # edges present in H as per ω = ??

How to compute E[X]?

Chandra & Ruta (UIUC) CS473 22 Fall 2016 22 / 56

Page 41: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Expected number of edges is:

Probability Space

Ω = 0, 1n. For ω ∈ 0, 1n, ωv = 1 if vertex v is presentin H, else is zero.

For each ω ∈ Ω, Pr[ω] = 12n .

X(ω) = # edges present in H as per ω = ??

How to compute E[X]?

Chandra & Ruta (UIUC) CS473 22 Fall 2016 22 / 56

Page 42: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Indicator Random Variables

DefinitionA binary random variable is one that takes on values in 0, 1.

Special type of random variables that are quite useful.

DefinitionGiven a probability space (Ω, Pr) and an event A ⊆ Ω the indicatorrandom variable XA is a binary random variable where XA(ω) = 1 ifω ∈ A and XA(ω) = 0 if ω 6∈ A.

Example

A 6-sided unbiased die. Ω = 1, 2, 3, 4, 5, 6 and Pr[i] = 1/6 for1 ≤ i ≤ 6. Let A be the even that i is divisible by 3. ThenXA(i) = 1 if i = 3, 6 and 0 otherwise.

Chandra & Ruta (UIUC) CS473 23 Fall 2016 23 / 56

Page 43: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Indicator Random Variables

DefinitionA binary random variable is one that takes on values in 0, 1.

Special type of random variables that are quite useful.

DefinitionGiven a probability space (Ω, Pr) and an event A ⊆ Ω the indicatorrandom variable XA is a binary random variable where XA(ω) = 1 ifω ∈ A and XA(ω) = 0 if ω 6∈ A.

Example

A 6-sided unbiased die. Ω = 1, 2, 3, 4, 5, 6 and Pr[i] = 1/6 for1 ≤ i ≤ 6. Let A be the even that i is divisible by 3. ThenXA(i) = 1 if i = 3, 6 and 0 otherwise.

Chandra & Ruta (UIUC) CS473 23 Fall 2016 23 / 56

Page 44: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Indicator Random Variables

DefinitionA binary random variable is one that takes on values in 0, 1.

Special type of random variables that are quite useful.

DefinitionGiven a probability space (Ω, Pr) and an event A ⊆ Ω the indicatorrandom variable XA is a binary random variable where XA(ω) = 1 ifω ∈ A and XA(ω) = 0 if ω 6∈ A.

Example

A 6-sided unbiased die. Ω = 1, 2, 3, 4, 5, 6 and Pr[i] = 1/6 for1 ≤ i ≤ 6. Let A be the even that i is divisible by 3. ThenXA(i) = 1 if i = 3, 6 and 0 otherwise.

Chandra & Ruta (UIUC) CS473 23 Fall 2016 23 / 56

Page 45: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Expectation

Proposition

For an indicator variable XA, E[XA] = Pr[A].

Proof.

E[XA] =∑

y∈Ω

XA(y) Pr[y]

=∑

y∈A

1 · Pr[y] +∑

y∈Ω\A0 · Pr[y]

=∑

y∈A

Pr[y]

= Pr[A] .

Chandra & Ruta (UIUC) CS473 24 Fall 2016 24 / 56

Page 46: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Linearity of Expectation

LemmaLet X, Y be two random variables (not necessarily independent) overa probability space (Ω, Pr). Then E[X + Y] = E[X] + E[Y].

Proof.

E[X + Y] =∑

ω∈Ω

Pr[ω] (X(ω) + Y(ω))

=∑

ω∈Ω

Pr[ω] X(ω) +∑

ω∈Ω

Pr[ω] Y(ω) = E[X] + E[Y] .

Corollary

E[a1X1 + a2X2 + . . . + anXn] =∑n

i=1 ai E[Xi].

Chandra & Ruta (UIUC) CS473 25 Fall 2016 25 / 56

Page 47: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Linearity of Expectation

LemmaLet X, Y be two random variables (not necessarily independent) overa probability space (Ω, Pr). Then E[X + Y] = E[X] + E[Y].

Proof.

E[X + Y] =∑

ω∈Ω

Pr[ω] (X(ω) + Y(ω))

=∑

ω∈Ω

Pr[ω] X(ω) +∑

ω∈Ω

Pr[ω] Y(ω) = E[X] + E[Y] .

Corollary

E[a1X1 + a2X2 + . . . + anXn] =∑n

i=1 ai E[Xi].

Chandra & Ruta (UIUC) CS473 25 Fall 2016 25 / 56

Page 48: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Expected number of edges?

Let G = (V, E) be a graph with n vertices and m edges. Let H bethe graph resulting from independently deleting every vertex of Gwith probability 1/2. The expected number of edges in H is

Event Ae = edge e ∈ E is present in H.

Pr[Ae=(u,v)

]= Pr[u and v both are present] =

Pr[u is present] · Pr[v is present] = 12· 1

2= 1

4.

XAe indicator random variables, then E[XAe] = Pr[Ae].

Let X =∑

e∈E XAe (Number of edges in H)

E[X] = E

[∑

e∈EXAe

]=∑

e∈EE[XAe] =

e∈EPr[Ae] =

m

4

It is important to setup random variables carefully.

Chandra & Ruta (UIUC) CS473 26 Fall 2016 26 / 56

Page 49: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Expected number of edges?

Let G = (V, E) be a graph with n vertices and m edges. Let H bethe graph resulting from independently deleting every vertex of Gwith probability 1/2. The expected number of edges in H is

Event Ae = edge e ∈ E is present in H.

Pr[Ae=(u,v)

]= Pr[u and v both are present] =

Pr[u is present] · Pr[v is present] = 12· 1

2= 1

4.

XAe indicator random variables, then E[XAe] = Pr[Ae].

Let X =∑

e∈E XAe (Number of edges in H)

E[X] = E

[∑

e∈EXAe

]=∑

e∈EE[XAe] =

e∈EPr[Ae] =

m

4

It is important to setup random variables carefully.

Chandra & Ruta (UIUC) CS473 26 Fall 2016 26 / 56

Page 50: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Expected number of edges?

Let G = (V, E) be a graph with n vertices and m edges. Let H bethe graph resulting from independently deleting every vertex of Gwith probability 1/2. The expected number of edges in H is

Event Ae = edge e ∈ E is present in H.

Pr[Ae=(u,v)

]= Pr[u and v both are present] =

Pr[u is present] · Pr[v is present] = 12· 1

2= 1

4.

XAe indicator random variables, then E[XAe] = Pr[Ae].

Let X =∑

e∈E XAe (Number of edges in H)

E[X] = E

[∑

e∈EXAe

]=∑

e∈EE[XAe] =

e∈EPr[Ae] =

m

4

It is important to setup random variables carefully.

Chandra & Ruta (UIUC) CS473 26 Fall 2016 26 / 56

Page 51: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Expected number of edges?

Let G = (V, E) be a graph with n vertices and m edges. Let H bethe graph resulting from independently deleting every vertex of Gwith probability 1/2. The expected number of edges in H is

Event Ae = edge e ∈ E is present in H.

Pr[Ae=(u,v)

]= Pr[u and v both are present] =

Pr[u is present] · Pr[v is present] = 12· 1

2= 1

4.

XAe indicator random variables, then E[XAe] = Pr[Ae].

Let X =∑

e∈E XAe (Number of edges in H)

E[X] = E

[∑

e∈EXAe

]=∑

e∈EE[XAe] =

e∈EPr[Ae] =

m

4

It is important to setup random variables carefully.

Chandra & Ruta (UIUC) CS473 26 Fall 2016 26 / 56

Page 52: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Expected number of edges?

Let G = (V, E) be a graph with n vertices and m edges. Let H bethe graph resulting from independently deleting every vertex of Gwith probability 1/2. The expected number of edges in H is

Event Ae = edge e ∈ E is present in H.

Pr[Ae=(u,v)

]= Pr[u and v both are present] =

Pr[u is present] · Pr[v is present] = 12· 1

2= 1

4.

XAe indicator random variables, then E[XAe] = Pr[Ae].

Let X =∑

e∈E XAe (Number of edges in H)

E[X] = E

[∑

e∈EXAe

]=∑

e∈EE[XAe] =

e∈EPr[Ae] =

m

4

It is important to setup random variables carefully.

Chandra & Ruta (UIUC) CS473 26 Fall 2016 26 / 56

Page 53: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Expected number of edges?

Let G = (V, E) be a graph with n vertices and m edges. Let H bethe graph resulting from independently deleting every vertex of Gwith probability 1/2. The expected number of edges in H is

Event Ae = edge e ∈ E is present in H.

Pr[Ae=(u,v)

]= Pr[u and v both are present] =

Pr[u is present] · Pr[v is present] = 12· 1

2= 1

4.

XAe indicator random variables, then E[XAe] = Pr[Ae].

Let X =∑

e∈E XAe (Number of edges in H)

E[X] = E

[∑

e∈EXAe

]=∑

e∈EE[XAe] =

e∈EPr[Ae] =

m

4

It is important to setup random variables carefully.

Chandra & Ruta (UIUC) CS473 26 Fall 2016 26 / 56

Page 54: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Expected number of triangles?

Let G = (V, E) be a graph with n vertices and m edges. Assume Ghas t triangles (i.e., a triangle is a simple cycle with three vertices).Let H be the graph resulting from deleting independently each vertexof G with probability 1/2. The expected number of triangles in H is

(A) t/2.

(B) t/4.

(C) t/8.

(D) t/16.

(E) none of the above.

Chandra & Ruta (UIUC) CS473 27 Fall 2016 27 / 56

Page 55: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Independent Random Variables

DefinitionRandom variables X, Y are said to be independent if

∀x, y ∈ R, Pr[X = x ∧ Y = y] = Pr[X = x] Pr[Y = y]

.

Examples

Two independent un-biased coin flips: Ω = HH, HT, TH, TT.X = 1 if first coin is H else 0. Y = 1 if second coin is H else 0.Independent.

X = #H, Y = #T. Dependent. Why?

Chandra & Ruta (UIUC) CS473 28 Fall 2016 28 / 56

Page 56: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Independent Random Variables

DefinitionRandom variables X, Y are said to be independent if

∀x, y ∈ R, Pr[X = x ∧ Y = y] = Pr[X = x] Pr[Y = y]

.

Examples

Two independent un-biased coin flips: Ω = HH, HT, TH, TT.X = 1 if first coin is H else 0. Y = 1 if second coin is H else 0.

Independent.

X = #H, Y = #T. Dependent. Why?

Chandra & Ruta (UIUC) CS473 28 Fall 2016 28 / 56

Page 57: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Independent Random Variables

DefinitionRandom variables X, Y are said to be independent if

∀x, y ∈ R, Pr[X = x ∧ Y = y] = Pr[X = x] Pr[Y = y]

.

Examples

Two independent un-biased coin flips: Ω = HH, HT, TH, TT.X = 1 if first coin is H else 0. Y = 1 if second coin is H else 0.Independent.

X = #H, Y = #T. Dependent. Why?

Chandra & Ruta (UIUC) CS473 28 Fall 2016 28 / 56

Page 58: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Independent Random Variables

DefinitionRandom variables X, Y are said to be independent if

∀x, y ∈ R, Pr[X = x ∧ Y = y] = Pr[X = x] Pr[Y = y]

.

Examples

Two independent un-biased coin flips: Ω = HH, HT, TH, TT.X = 1 if first coin is H else 0. Y = 1 if second coin is H else 0.Independent.

X = #H, Y = #T.

Dependent. Why?

Chandra & Ruta (UIUC) CS473 28 Fall 2016 28 / 56

Page 59: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Independent Random Variables

DefinitionRandom variables X, Y are said to be independent if

∀x, y ∈ R, Pr[X = x ∧ Y = y] = Pr[X = x] Pr[Y = y]

.

Examples

Two independent un-biased coin flips: Ω = HH, HT, TH, TT.X = 1 if first coin is H else 0. Y = 1 if second coin is H else 0.Independent.

X = #H, Y = #T. Dependent. Why?

Chandra & Ruta (UIUC) CS473 28 Fall 2016 28 / 56

Page 60: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

independent Randomized Variables

LemmaIf X and Y are independent then E[X · Y] = E[X] · E[Y]

Proof.

E[X · Y] =∑

ω∈Ω

Pr[ω] (X(ω) · Y(ω))

=∑

x,y∈RPr[X = x ∧ Y = y] (x · y)

=∑

x,y∈RPr[X = x] · Pr[Y = y] · x · y

= (∑

x∈RPr[X = x] x)(

y∈RPr[Y = y] y) = E[X] E[Y]

Chandra & Ruta (UIUC) CS473 29 Fall 2016 29 / 56

Page 61: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Types of Randomized Algorithms

Typically one encounters the following types:

1 Las Vegas randomized algorithms: for a given input xoutput of algorithm is always correct but the running time is arandom variable. In this case we are interested in analyzing theexpected running time.

2 Monte Carlo randomized algorithms: for a given input x therunning time is deterministic but the output is random; correctwith some probability. In this case we are interested in analyzingthe probability of the correct output (and also the running time).

3 Algorithms whose running time and output may both be random.

Chandra & Ruta (UIUC) CS473 30 Fall 2016 30 / 56

Page 62: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Types of Randomized Algorithms

Typically one encounters the following types:

1 Las Vegas randomized algorithms: for a given input xoutput of algorithm is always correct but the running time is arandom variable. In this case we are interested in analyzing theexpected running time.

2 Monte Carlo randomized algorithms: for a given input x therunning time is deterministic but the output is random; correctwith some probability. In this case we are interested in analyzingthe probability of the correct output (and also the running time).

3 Algorithms whose running time and output may both be random.

Chandra & Ruta (UIUC) CS473 30 Fall 2016 30 / 56

Page 63: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Analyzing Las Vegas Algorithms

Deterministic algorithm Q for a problem Π:1 Let Q(x) be the time for Q to run on input x of length |x|.2 Worst-case analysis: run time on worst input for a given size n.

Twc(n) = maxx:|x|=n

Q(x).

Randomized algorithm R for a problem Π:1 Let R(x) be the time for Q to run on input x of length |x|.2 R(x) is a random variable: depends on random bits used by R.3 E[R(x)] is the expected running time for R on x4 Worst-case analysis: expected time on worst input of size n

Trand−wc(n) = maxx:|x|=n

E[R(x)] .

Chandra & Ruta (UIUC) CS473 31 Fall 2016 31 / 56

Page 64: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Analyzing Las Vegas Algorithms

Deterministic algorithm Q for a problem Π:1 Let Q(x) be the time for Q to run on input x of length |x|.2 Worst-case analysis: run time on worst input for a given size n.

Twc(n) = maxx:|x|=n

Q(x).

Randomized algorithm R for a problem Π:1 Let R(x) be the time for Q to run on input x of length |x|.2 R(x) is a random variable: depends on random bits used by R.3 E[R(x)] is the expected running time for R on x4 Worst-case analysis: expected time on worst input of size n

Trand−wc(n) = maxx:|x|=n

E[R(x)] .

Chandra & Ruta (UIUC) CS473 31 Fall 2016 31 / 56

Page 65: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Analyzing Monte Carlo Algorithms

Randomized algorithm M for a problem Π:

1 Let M(x) be the time for M to run on input x of length |x|. ForMonte Carlo, assumption is that run time is deterministic.

2 Let Pr[x] be the probability that M is correct on x.

3 Pr[x] is a random variable: depends on random bits used by M.

4 Worst-case analysis: success probability on worst input

Prand−wc(n) = minx:|x|=n

Pr[x] .

Chandra & Ruta (UIUC) CS473 32 Fall 2016 32 / 56

Page 66: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Part III

Why does randomization help?

Chandra & Ruta (UIUC) CS473 33 Fall 2016 33 / 56

Page 67: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Ping and find.

Consider a deterministic algorithm A that is trying to find an elementin an array X of size n. At every step it is allowed to ask the value ofone cell in the array, and the adversary is allowed after each suchping, to shuffle elements around in the array in any way it seems fit.For the best possible deterministic algorithm the number of rounds ithas to play this game till it finds the required element is

(A) O(1)

(B) O(n)

(C) O(n log n)

(D) O(n2)

(E) ∞.

Chandra & Ruta (UIUC) CS473 34 Fall 2016 34 / 56

Page 68: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Ping and find randomized.

Consider an algorithm randFind that is trying to find an element inan array X of size n. At every step it asks the value of one randomcell in the array, and the adversary is allowed after each such ping, toshuffle elements around in the array in any way it seems fit. Thisalgorithm would stop in expectation after

(A) O(1)

(B) O(log n)

(C) O(n)

(D) O(n2)

(E) ∞.

steps.

Chandra & Ruta (UIUC) CS473 35 Fall 2016 35 / 56

Page 69: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Abundance of witnesses

Consider the problem of finding an “approximate median” of anunsorted array A[1..n]: an element of A with rank between n/4 and3n/4.

Finding an approximate median is not any easier than a propermedian.

n/2 elements of A qualify as approximate medians and hence arandom element is good with probability 1/2!

Chandra & Ruta (UIUC) CS473 36 Fall 2016 36 / 56

Page 70: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Part IV

Randomized Quick Sort

Chandra & Ruta (UIUC) CS473 37 Fall 2016 37 / 56

Page 71: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Randomized QuickSort

Randomized QuickSort1 Pick a pivot element uniformly at random from the array.

2 Split array into 3 subarrays: those smaller than pivot, thoselarger than pivot, and the pivot itself.

3 Recursively sort the subarrays, and concatenate them.

Chandra & Ruta (UIUC) CS473 38 Fall 2016 38 / 56

Page 72: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Analysis

What events to count?

Number of Comparisions.

What is the probability space?

All the coin tosses at all levels and parts of recursion.

Too Big!!

What random variables to define?What are the events of the algorithm?

Chandra & Ruta (UIUC) CS473 39 Fall 2016 39 / 56

Page 73: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Analysis

What events to count?

Number of Comparisions.

What is the probability space?

All the coin tosses at all levels and parts of recursion.

Too Big!!

What random variables to define?What are the events of the algorithm?

Chandra & Ruta (UIUC) CS473 39 Fall 2016 39 / 56

Page 74: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Analysis

What events to count?

Number of Comparisions.

What is the probability space?

All the coin tosses at all levels and parts of recursion.

Too Big!!

What random variables to define?What are the events of the algorithm?

Chandra & Ruta (UIUC) CS473 39 Fall 2016 39 / 56

Page 75: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Analysis

What events to count?

Number of Comparisions.

What is the probability space?

All the coin tosses at all levels and parts of recursion.

Too Big!!

What random variables to define?What are the events of the algorithm?

Chandra & Ruta (UIUC) CS473 39 Fall 2016 39 / 56

Page 76: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Analysis via Recurrence

1 Given array A of size n, let Q(A) be number of comparisons ofrandomized QuickSort on A.

2 Note that Q(A) is a random variable.

3 Let Aileft and Ai

right be the left and right arrays obtained if:

Let Xi be indicator random variable, which is set to 1 if pivot isof rank i in A, else zero.

Q(A) = n +n∑

i=1

Xi ·(

Q(Aileft) + Q(Ai

right)).

Since each element of A has probability exactly of 1/n of beingchosen:

E[Xi] = Pr[pivot has rank i] = 1/n.

Chandra & Ruta (UIUC) CS473 40 Fall 2016 40 / 56

Page 77: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Analysis via Recurrence

1 Given array A of size n, let Q(A) be number of comparisons ofrandomized QuickSort on A.

2 Note that Q(A) is a random variable.

3 Let Aileft and Ai

right be the left and right arrays obtained if:

Let Xi be indicator random variable, which is set to 1 if pivot isof rank i in A, else zero.

Q(A) = n +n∑

i=1

Xi ·(

Q(Aileft) + Q(Ai

right)).

Since each element of A has probability exactly of 1/n of beingchosen:

E[Xi] = Pr[pivot has rank i] = 1/n.

Chandra & Ruta (UIUC) CS473 40 Fall 2016 40 / 56

Page 78: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Independence of Random Variables

LemmaRandom variables Xi is independent of random variables Q(Ai

left) aswell as Q(Ai

right), i.e.

E[Xi · Q(Ai

left)]

= E[Xi] E[Q(Ai

left)]

E[Xi · Q(Ai

right)]

= E[Xi] E[Q(Ai

right)]

Proof.This is because the algorithm, while recursing on Q(Ai

left) andQ(Ai

right) uses new random coin tosses that are independent of thecoin tosses used to decide the first pivot. Only the latter decidesvalue of Xi.

Chandra & Ruta (UIUC) CS473 41 Fall 2016 41 / 56

Page 79: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Analysis via Recurrence

Let T(n) = maxA:|A|=n E[Q(A)] be the worst-case expected runningtime of randomized QuickSort on arrays of size n.

We have, for any A:

Q(A) = n +n∑

i=1

Xi

(Q(Ai

left) + Q(Airight)

)

By linearity of expectation, and independence random variables:

E[Q(A)

]= n +

n∑

i=1

E[Xi](

E[Q(Ai

left)]

+ E[Q(Ai

right)])

.

⇒ E[Q(A)

]≤ n +

n∑

i=1

1

n(T(i− 1) + T(n− i)) .

Chandra & Ruta (UIUC) CS473 42 Fall 2016 42 / 56

Page 80: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Analysis via Recurrence

Let T(n) = maxA:|A|=n E[Q(A)] be the worst-case expected runningtime of randomized QuickSort on arrays of size n.

We have, for any A:

Q(A) = n +n∑

i=1

Xi

(Q(Ai

left) + Q(Airight)

)

By linearity of expectation, and independence random variables:

E[Q(A)

]= n +

n∑

i=1

E[Xi](

E[Q(Ai

left)]

+ E[Q(Ai

right)])

.

⇒ E[Q(A)

]≤ n +

n∑

i=1

1

n(T(i− 1) + T(n− i)) .

Chandra & Ruta (UIUC) CS473 42 Fall 2016 42 / 56

Page 81: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Analysis via Recurrence

Let T(n) = maxA:|A|=n E[Q(A)] be the worst-case expected runningtime of randomized QuickSort on arrays of size n.

We have, for any A:

Q(A) = n +n∑

i=1

Xi

(Q(Ai

left) + Q(Airight)

)

By linearity of expectation, and independence random variables:

E[Q(A)

]= n +

n∑

i=1

E[Xi](

E[Q(Ai

left)]

+ E[Q(Ai

right)])

.

⇒ E[Q(A)

]≤ n +

n∑

i=1

1

n(T(i− 1) + T(n− i)) .

Chandra & Ruta (UIUC) CS473 42 Fall 2016 42 / 56

Page 82: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Analysis via Recurrence

Let T(n) = maxA:|A|=n E[Q(A)] be the worst-case expected runningtime of randomized QuickSort on arrays of size n.

We have, for any A:

Q(A) = n +n∑

i=1

Xi

(Q(Ai

left) + Q(Airight)

)

By linearity of expectation, and independence random variables:

E[Q(A)

]= n +

n∑

i=1

E[Xi](

E[Q(Ai

left)]

+ E[Q(Ai

right)])

.

⇒ E[Q(A)

]≤ n +

n∑

i=1

1

n(T(i− 1) + T(n− i)) .

Chandra & Ruta (UIUC) CS473 42 Fall 2016 42 / 56

Page 83: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Analysis via Recurrence

Let T(n) = maxA:|A|=n E[Q(A)] be the worst-case expected runningtime of randomized QuickSort on arrays of size n.

We derived:

E[Q(A)

]≤ n +

n∑

i=1

1

n(T(i− 1) + T(n− i)) .

Note that above holds for any A of size n. Therefore

maxA:|A|=n

E[Q(A)] = T(n) ≤ n +n∑

i=1

1

n(T(i− 1) + T(n− i)) .

Chandra & Ruta (UIUC) CS473 43 Fall 2016 43 / 56

Page 84: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Analysis via Recurrence

Let T(n) = maxA:|A|=n E[Q(A)] be the worst-case expected runningtime of randomized QuickSort on arrays of size n.We derived:

E[Q(A)

]≤ n +

n∑

i=1

1

n(T(i− 1) + T(n− i)) .

Note that above holds for any A of size n. Therefore

maxA:|A|=n

E[Q(A)] = T(n) ≤ n +n∑

i=1

1

n(T(i− 1) + T(n− i)) .

Chandra & Ruta (UIUC) CS473 43 Fall 2016 43 / 56

Page 85: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Solving the Recurrence

T(n) ≤ n +n∑

i=1

1

n(T(i− 1) + T(n− i))

with base case T(1) = 0.

LemmaT(n) = O(n log n).

Proof.(Guess and) Verify by induction.

Chandra & Ruta (UIUC) CS473 44 Fall 2016 44 / 56

Page 86: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Solving the Recurrence

T(n) ≤ n +n∑

i=1

1

n(T(i− 1) + T(n− i))

with base case T(1) = 0.

LemmaT(n) = O(n log n).

Proof.(Guess and) Verify by induction.

Chandra & Ruta (UIUC) CS473 44 Fall 2016 44 / 56

Page 87: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Solving the Recurrence

T(n) ≤ n +n∑

i=1

1

n(T(i− 1) + T(n− i))

with base case T(1) = 0.

LemmaT(n) = O(n log n).

Proof.(Guess and) Verify by induction.

Chandra & Ruta (UIUC) CS473 44 Fall 2016 44 / 56

Page 88: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Part V

Slick analysis of QuickSort

Chandra & Ruta (UIUC) CS473 45 Fall 2016 45 / 56

Page 89: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

A Slick Analysis of QuickSort

Let Q(A) be number of comparisons done on input array A:

1 For 1 ≤ i < j < n let Rij be the event that rank i element iscompared with rank j element.

2 Xij is the indicator random variable for Rij. That is, Xij = 1 ifrank i is compared with rank j element, otherwise 0.

Q(A) =∑

1≤i<j≤n

Xij

and hence by linearity of expectation,

E[Q(A)

]=

1≤i<j≤n

E[Xij

]=

1≤i<j≤n

Pr[Rij

].

Chandra & Ruta (UIUC) CS473 46 Fall 2016 46 / 56

Page 90: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

A Slick Analysis of QuickSort

Let Q(A) be number of comparisons done on input array A:

1 For 1 ≤ i < j < n let Rij be the event that rank i element iscompared with rank j element.

2 Xij is the indicator random variable for Rij. That is, Xij = 1 ifrank i is compared with rank j element, otherwise 0.

Q(A) =∑

1≤i<j≤n

Xij

and hence by linearity of expectation,

E[Q(A)

]=

1≤i<j≤n

E[Xij

]=

1≤i<j≤n

Pr[Rij

].

Chandra & Ruta (UIUC) CS473 46 Fall 2016 46 / 56

Page 91: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

A Slick Analysis of QuickSort

Rij = rank i element is compared with rank j element.

Question: What is Pr[Rij]?

7 5 9 1 3 4 8 6

Chandra & Ruta (UIUC) CS473 47 Fall 2016 47 / 56

Page 92: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

A Slick Analysis of QuickSort

Rij = rank i element is compared with rank j element.

Question: What is Pr[Rij]?

With ranks:

7 5 9 1 3 4 8 61 2 34 56 78

Chandra & Ruta (UIUC) CS473 47 Fall 2016 47 / 56

Page 93: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

A Slick Analysis of QuickSort

Rij = rank i element is compared with rank j element.

Question: What is Pr[Rij]?

With ranks:

7 5 9 1 3 4 8 61 2 34 56 78

As such, probability of comparing 5 to 8 is Pr[R4,7].

Chandra & Ruta (UIUC) CS473 47 Fall 2016 47 / 56

Page 94: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

A Slick Analysis of QuickSort

Rij = rank i element is compared with rank j element.

Question: What is Pr[Rij]?

With ranks:

7 5 9 1 3 4 8 61 2 34 56 78

1 If pivot too small (say 3 [rank 2]). Partition and call recursively:

7 5 9 1 3 4 8 6=⇒ 7 5 93 4 8 61

Decision if to compare 5 to 8 is moved to subproblem.

Chandra & Ruta (UIUC) CS473 47 Fall 2016 47 / 56

Page 95: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

A Slick Analysis of QuickSort

Rij = rank i element is compared with rank j element.

Question: What is Pr[Rij]?

With ranks:

7 5 9 1 3 4 8 61 2 34 56 78

1 If pivot too small (say 3 [rank 2]). Partition and call recursively:

7 5 9 1 3 4 8 6=⇒ 7 5 93 4 8 61

Decision if to compare 5 to 8 is moved to subproblem.

2 If pivot too large (say 9 [rank 8]):

7 5 9 1 3 4 8 67 5 9 1 3 4 8 6=⇒ 7 5 1 3 4 8 6 9

Decision if to compare 5 to 8 moved to subproblem.

Chandra & Ruta (UIUC) CS473 47 Fall 2016 47 / 56

Page 96: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

A Slick Analysis of QuickSortQuestion: What is Pr[Ri,j]?

7 5 9 1 3 4 8 61 2 34 56 78

As such, probability of com-paring 5 to 8 is Pr[R4,7].

1 If pivot is 5 (rank 4). Bingo!

7 5 9 1 3 4 8 6=⇒ 1 3 4 5 7 9 8 6

Chandra & Ruta (UIUC) CS473 48 Fall 2016 48 / 56

Page 97: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

A Slick Analysis of QuickSortQuestion: What is Pr[Ri,j]?

7 5 9 1 3 4 8 61 2 34 56 78

As such, probability of com-paring 5 to 8 is Pr[R4,7].

1 If pivot is 5 (rank 4). Bingo!

7 5 9 1 3 4 8 6=⇒ 1 3 4 5 7 9 8 6

2 If pivot is 8 (rank 7). Bingo!

7 5 9 1 3 4 8 6=⇒ 7 5 91 3 4 6 8

Chandra & Ruta (UIUC) CS473 48 Fall 2016 48 / 56

Page 98: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

A Slick Analysis of QuickSortQuestion: What is Pr[Ri,j]?

7 5 9 1 3 4 8 61 2 34 56 78

As such, probability of com-paring 5 to 8 is Pr[R4,7].

1 If pivot is 5 (rank 4). Bingo!

7 5 9 1 3 4 8 6=⇒ 1 3 4 5 7 9 8 6

2 If pivot is 8 (rank 7). Bingo!

7 5 9 1 3 4 8 6=⇒ 7 5 91 3 4 6 8

3 If pivot in between the two numbers (say 6 [rank 5]):

7 5 9 1 3 4 8 6=⇒ 75 91 3 4 6 8

5 and 8 will never be compared to each other.

Chandra & Ruta (UIUC) CS473 48 Fall 2016 48 / 56

Page 99: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

A Slick Analysis of QuickSortQuestion: What is Pr[Ri,j]?

Conclusion:Ri,j happens if and only if:

ith or jth ranked element is the first pivot out ofith to jth ranked elements.

How to analyze this?Thinking acrobatics!

1 Assign every element in the array a random priority (say in[0, 1]).

2 Choose pivot to be the element with lowest priority insubproblem.

3 Equivalent to picking pivot uniformly at random(as QuickSort do).

Chandra & Ruta (UIUC) CS473 49 Fall 2016 49 / 56

Page 100: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

A Slick Analysis of QuickSortQuestion: What is Pr[Ri,j]?

How to analyze this?Thinking acrobatics!

1 Assign every element in the array a random priority (say in[0, 1]).

2 Choose pivot to be the element with lowest priority insubproblem.

=⇒ Ri,j happens if either i or j have lowest priority out of elementsrank i to j,

There are k = j− i + 1 relevant elements.

Pr[Ri,j

]=

2

k=

2

j− i + 1.

Chandra & Ruta (UIUC) CS473 50 Fall 2016 50 / 56

Page 101: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

A Slick Analysis of QuickSortQuestion: What is Pr[Ri,j]?

How to analyze this?Thinking acrobatics!

1 Assign every element in the array a random priority (say in[0, 1]).

2 Choose pivot to be the element with lowest priority insubproblem.

=⇒ Ri,j happens if either i or j have lowest priority out of elementsrank i to j,There are k = j− i + 1 relevant elements.

Pr[Ri,j

]=

2

k=

2

j− i + 1.

Chandra & Ruta (UIUC) CS473 50 Fall 2016 50 / 56

Page 102: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

A Slick Analysis of QuickSort

Question: What is Pr[Rij]?

Lemma

Pr[Rij

]= 2

j−i+1.

Proof.Let a1, . . . , ai, . . . , aj, . . . , an be elements of A in sorted order. LetS = ai, ai+1, . . . , ajObservation: If pivot is chosen outside S then all of S either in leftarray or right array.Observation: ai and aj separated when a pivot is chosen from S forthe first time. Once separated no comparison.Observation: ai is compared with aj if and only if either ai or aj ischosen as a pivot from S at separation...

Chandra & Ruta (UIUC) CS473 51 Fall 2016 51 / 56

Page 103: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

A Slick Analysis of QuickSort

Question: What is Pr[Rij]?

Lemma

Pr[Rij

]= 2

j−i+1.

Proof.Let a1, . . . , ai, . . . , aj, . . . , an be elements of A in sorted order. LetS = ai, ai+1, . . . , ajObservation: If pivot is chosen outside S then all of S either in leftarray or right array.Observation: ai and aj separated when a pivot is chosen from S forthe first time. Once separated no comparison.Observation: ai is compared with aj if and only if either ai or aj ischosen as a pivot from S at separation...

Chandra & Ruta (UIUC) CS473 51 Fall 2016 51 / 56

Page 104: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

A Slick Analysis of QuickSort

Question: What is Pr[Rij]?

Lemma

Pr[Rij

]= 2

j−i+1.

Proof.Let a1, . . . , ai, . . . , aj, . . . , an be elements of A in sorted order. LetS = ai, ai+1, . . . , ajObservation: If pivot is chosen outside S then all of S either in leftarray or right array.Observation: ai and aj separated when a pivot is chosen from S forthe first time. Once separated no comparison.Observation: ai is compared with aj if and only if either ai or aj ischosen as a pivot from S at separation...

Chandra & Ruta (UIUC) CS473 51 Fall 2016 51 / 56

Page 105: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

A Slick Analysis of QuickSortContinued...

Lemma

Pr[Rij

]= 2

j−i+1.

Proof.Let a1, . . . , ai, . . . , aj, . . . , an be sort of A. LetS = ai, ai+1, . . . , ajObservation: ai is compared with aj if and only if either ai or aj ischosen as a pivot from S at separation.Observation: Given that pivot is chosen from S the probability thatit is ai or aj is exactly 2/|S| = 2/(j− i + 1) since the pivot ischosen uniformly at random from the array.

Chandra & Ruta (UIUC) CS473 52 Fall 2016 52 / 56

Page 106: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

How much is this?

Hn =∑n

i=11i

is the n’th harmonic number

(A) Hn = Θ(1).

(B) Hn = Θ(log log n).

(C) Hn = Θ(√

log n).

(D) Hn = Θ(log n).

(E) Hn = Θ(log2 n).

Chandra & Ruta (UIUC) CS473 53 Fall 2016 53 / 56

Page 107: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

And how much is this?

Tn =n−1∑

i=1

n−i∑

j=1

1

j

is equal to

(A) Tn = Θ(n).

(B) Tn = Θ(n log n).

(C) Tn = Θ(n log2 n).

(D) Tn = Θ(n2).

(E) Tn = Θ(n3).

Chandra & Ruta (UIUC) CS473 54 Fall 2016 54 / 56

Page 108: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

A Slick Analysis of QuickSortContinued...

E[Q(A)

]=

1≤i<j≤n

E[Xij] =∑

1≤i<j≤n

Pr[Rij] .

Lemma

Pr[Rij] = 2j−i+1

.

E[Q(A)

]=

1≤i<j≤n

2

j− i + 1

Chandra & Ruta (UIUC) CS473 55 Fall 2016 55 / 56

Page 109: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

A Slick Analysis of QuickSortContinued...

Lemma

Pr[Rij] = 2j−i+1

.

E[Q(A)

]=

1≤i<j≤n

Pr[Rij

]=

1≤i<j≤n

2

j− i + 1

Chandra & Ruta (UIUC) CS473 55 Fall 2016 55 / 56

Page 110: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

A Slick Analysis of QuickSortContinued...

Lemma

Pr[Rij] = 2j−i+1

.

E[Q(A)

]=

1≤i<j≤n

2

j− i + 1

Chandra & Ruta (UIUC) CS473 55 Fall 2016 55 / 56

Page 111: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

A Slick Analysis of QuickSortContinued...

Lemma

Pr[Rij] = 2j−i+1

.

E[Q(A)

]=

1≤i<j≤n

2

j− i + 1

=n−1∑

i=1

n∑

j=i+1

2

j− i + 1

Chandra & Ruta (UIUC) CS473 55 Fall 2016 55 / 56

Page 112: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

A Slick Analysis of QuickSortContinued...

Lemma

Pr[Rij] = 2j−i+1

.

E[Q(A)

]=

n−1∑

i=1

n∑

j=i+1

2

j− i + 1= 2

n−1∑

i=1

n∑

i<j

1

j− i + 1

Chandra & Ruta (UIUC) CS473 55 Fall 2016 55 / 56

Page 113: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

A Slick Analysis of QuickSortContinued...

Lemma

Pr[Rij] = 2j−i+1

.

E[Q(A)

]= 2

n−1∑

i=1

n∑

i<j

1

j− i + 1

Chandra & Ruta (UIUC) CS473 55 Fall 2016 55 / 56

Page 114: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

A Slick Analysis of QuickSortContinued...

Lemma

Pr[Rij] = 2j−i+1

.

E[Q(A)

]= 2

n−1∑

i=1

n∑

i<j

1

j− i + 1

Chandra & Ruta (UIUC) CS473 55 Fall 2016 55 / 56

Page 115: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

A Slick Analysis of QuickSortContinued...

Lemma

Pr[Rij] = 2j−i+1

.

E[Q(A)

]= 2

n−1∑

i=1

n∑

i<j

1

j− i + 1≤ 2

n−1∑

i=1

n−i+1∑

∆=2

1

Chandra & Ruta (UIUC) CS473 55 Fall 2016 55 / 56

Page 116: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

A Slick Analysis of QuickSortContinued...

Lemma

Pr[Rij] = 2j−i+1

.

E[Q(A)

]= 2

n−1∑

i=1

n∑

i<j

1

j− i + 1≤ 2

n−1∑

i=1

n−i+1∑

∆=2

1

≤ 2n−1∑

i=1

(Hn−i+1 − 1) ≤ 2∑

1≤i<n

Hn

Chandra & Ruta (UIUC) CS473 55 Fall 2016 55 / 56

Page 117: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

A Slick Analysis of QuickSortContinued...

Lemma

Pr[Rij] = 2j−i+1

.

E[Q(A)

]= 2

n−1∑

i=1

n∑

i<j

1

j− i + 1≤ 2

n−1∑

i=1

n−i+1∑

∆=2

1

≤ 2n−1∑

i=1

(Hn−i+1 − 1) ≤ 2∑

1≤i<n

Hn

≤ 2nHn = O(n log n)

Chandra & Ruta (UIUC) CS473 55 Fall 2016 55 / 56

Page 118: BBM402-Lecture 12: Randomized Algorithmsozkahya/classes/bbm402/Lectures/Lec12... · Example: Randomized Quicksort Recall: QuickSort can take (n 2) time to sort array of size n . Theorem

Where do I get random bits?

Question: Are true random bits available in practice?

1 Buy them!

2 CPUs use physical phenomena to generate random bits.

3 Can use pseudo-random bits or semi-random bits from nature.Several fundamental unresolved questions in complexity theoryon this topic. Beyond the scope of this course.

4 In practice pseudo-random generators work quite well in manyapplications.

5 The model is interesting to think in the abstract and is veryuseful even as a theoretical construct. One can derandomizerandomized algorithms to obtain deterministic algorithms.

Chandra & Ruta (UIUC) CS473 56 Fall 2016 56 / 56