Randomization at Work: An Introduction to Randomized ... · QuickSort I Proposed by C.A.R. Hoare in...

QuickSort Randomized QuickSort MAXCUT Randomized MAXCUT

Randomization at Work:An Introduction to Randomized Algorithms

Arpit AgarwalSuprovat Ghoshal

Indian Institute of Science, Bangalore

June 27, 2013

Randomization at Work IISc

QuickSort

I Proposed by C.A.R. Hoare in 1962.

I Divide-and-conquer algorithm.

I One of the fastest sorting algorithms in practice.

Pseudo-code

1. Divide: Select an element from the array and call it the pivot x . Partition thearray such that all elements which are ≤ x are in the lower subarray and allelements ≥ x are in the upper subarray.

2. Conquer: Apply the divide step above to both the lower and upper subarraysrecursively.

3. Combine: The array will be trivially sorted using the above steps.

QuickSort

Pseudo-code

QuickSort

Pseudo-code

QuickSort

Pseudo-code

QuickSort : Partitioning

27 24 22 38 35 25 30

The Input Array

1. Select the pivot

27 24 22 38 35 25 30

2. Partition

24 22 25 27 37 35 30

27 24 22 38 35 25 30

The Input Array

1. Select the pivot

27 24 22 38 35 25 30

2. Partition

24 22 25 27 37 35 30

27 24 22 38 35 25 30

The Input Array

1. Select the pivot

27 24 22 38 35 25 30

2. Partition

24 22 25 27 37 35 30

27 24 22 38 35 25 30

The Input Array

1. Select the pivot

27 24 22 38 35 25 30

2. Partition

24 22 25 27 37 35 30

27 24 22 38 35 25 30

The Input Array

1. Select the pivot

27 24 22 38 35 25 30

2. Partition

24 22 25 27 37 35 30

27 24 22 38 35 25 30

The Input Array

1. Select the pivot

27 24 22 38 35 25 30

2. Partition

24 22 25 27 37 35 30

27 24 22 38 35 25 30

The Input Array

1. Select the pivot

27 24 22 38 35 25 30

2. Partition

24 22 25 27 37 35 30

27 24 22 38 35 25 30

The Input Array

1. Select the pivot

27 24 22 38 35 25 30

2. Partition

24 22 25 27 38 35 30

27 24 22 38 35 25 30

The Input Array

1. Select the pivot

27 24 22 38 35 25 30

2. Partition

24 22 25 27 38 35 30

27 24 22 38 35 25 30

The Input Array

1. Select the pivot

27 24 22 38 35 25 30

2. Partition

24 22 25 27 38 35 30

27 24 22 38 35 25 30

The Input Array

1. Select the pivot

27 24 22 38 35 25 30

2. Partition

24 22 25 27 38 35 30

Final Partitioned Array

QuickSort: Recursion

Worst Case Analysis of QuickSort

I Worst Case occurs when you always partition around the minimum or maximumelement.

I One side of the partition always has n − 1 elements and the other has noelements.

T (n) = T (0) + T (n − 1) + Θ(n)

= T (n − 1) + Θ(n)

= T (n − 2) + Θ(n − 1) + Θ(n)

· · ·= Θ(1) + Θ(2) + · · ·+ Θ(n − 1) + Θ(n)

= Θ(n2)

Randomized QuickSort

MotivationI We might always get an input array that is almost sorted or almost reverse

sorted.

I Quick Sort will perform badly in this scenario.

I An adversery might give you the input array which will make you do a lot ofwork.

IdeaI Pick the pivot randomly to partition each time.

I Therefore, we almost never allow worst case to occur, whatever the type of theinput array.

I In adversarial environment, we confuse the adversary with the random choice ofpivot.

sorted.

Analysis of Randomized QuickSortLet X be the total number of comparisons. Let [x1, x2, · · · , xn] be the input array, and[y1, y2, · · · , yn] be the output sorted array.

Define : Xij to be a indicator random variable such that

{1 if yi and yj are compared0 otherwise

E [Xij ] =∞∑−∞

x .P(x)

= 1.P(Xij = 1) + 0.P(Xij = 0)

= P(Xij = 1)

= P(yi and yj are compared) (1)

n−1∑i=1

n∑j=i+1

E [Xij ] =∞∑−∞

x .P(x)

= 1.P(Xij = 1) + 0.P(Xij = 0)

= P(Xij = 1)

n−1∑i=1

n∑j=i+1

E [Xij ] =∞∑−∞

x .P(x)

= 1.P(Xij = 1) + 0.P(Xij = 0)

= P(Xij = 1)

n−1∑i=1

n∑j=i+1

E [Xij ] =∞∑−∞

x .P(x)

= 1.P(Xij = 1) + 0.P(Xij = 0)

= P(Xij = 1)

n−1∑i=1

n∑j=i+1

E [Xij ] =∞∑−∞

x .P(x)

= 1.P(Xij = 1) + 0.P(Xij = 0)

= P(Xij = 1)

n−1∑i=1

n∑j=i+1

Analysis of Randomized QuickSortTherefore,

E [X ] = E [

n−1∑i=1

n∑j=i+1

n−1∑i=1

n∑j=i+1

E [Xij ] (2)

Consider the subarray {yi , yi+1, · · · , yj−1, yj}.

Case 1: yi was chosen as the pivot. yi and yj are compared in this case.

yi · · · yj

Case 2: Some element from yi+1 to yj−1 was chosen as the pivot. yi and yj are notcompared in this case.

yi · · · yk · · · yj

E [X ] = E [

n−1∑i=1

n∑j=i+1

n−1∑i=1

n∑j=i+1

E [Xij ] (2)

yi · · · yj

yi · · · yk · · · yj

E [X ] = E [

n−1∑i=1

n∑j=i+1

n−1∑i=1

n∑j=i+1

E [Xij ] (2)

yi · · · yj

yi · · · yk · · · yj

Analysis of Randomized QuickSort

Case 3: yj was chosen as the pivot. yi and yj are compared in this case.

yi · · · yj

I The total number of elements in {yi , yi+1, · · · , yj−1, yj} is j − i + 1.

I Each element is chosen as pivot uniformly at random.

I Probability of yi being chosen as pivot(Case 1) is 1j−i+1

I Probability of yj being chosen as pivot(Case 3) is 1j−i+1

P(yi and yj are compared) = P(Case1) + P(Case3)

j − i + 1+

j − i + 1

yi · · · yj

j − i + 1+

j − i + 1

yi · · · yj

j − i + 1+

j − i + 1

yi · · · yj

j − i + 1+

j − i + 1

yi · · · yj

j − i + 1+

j − i + 1

yi · · · yj

j − i + 1+

j − i + 1

From Equation (1) and (2),

E [X ] =

n−1∑i=1

n∑j=i+1

j − i + 1

n−1∑i=1

3+ · · ·+

n − i + 1)

n−1∑i=1

3+ · · ·+

< 2n(1 +1

3+ · · ·+

The quantity 1 + 12

+ · · ·+ 1n

, denoted Hn, is called the nth Harmonic number

and is in the range [ln n, 1 + ln n]. Therefore,

E [X ] < 2n(Hn) ≤ 2n ln n + 2n = θ(n ln n)

MAX-CUT : Problem Definition

I Input : An undirected graph G(V ,E) where V : Set of vertices E : Set of Edges

I Output : A partition of vertices V1,V2 such that the no. of edges crossing fromV1 to V2 is maximised

Easy Example

Figure : A Bipartite Graph

Easy Example

Figure : A Bipartite Graph

A Not so Easy Example

Figure : A General Graph

A Not so Easy Example

Figure : A General Graph

Naive Approach

I The total no. of ways the vertex set can be partitioned is 2‖V‖

I Brute force would require exponential no. of trials, which is too much hardwork!!

I Can we do better ?

Standard Approaches

I Greedy?

I Divide and Conquer?

I Dynamic Programming?

Naive Approach

I The total no. of ways the vertex set can be partitioned is 2‖V‖

I Brute force would require exponential no. of trials, which is too much hardwork!!

I Can we do better ?

Standard Approaches

I Greedy?

I Divide and Conquer?

I Dynamic Programming?

Randomized Approach Example

I A Simple randomized algorithm that provides (probabilistic) guarantees a cut ofsize ‖E‖/2

I Algorithm Initialize the partitions V1 and V2 as empty setsFor each vertex v ∈ V , flip a fair coin. If outcome is heads then put vertex inpartition V1 else put it into partition V2.

Pseudo-code

1. Initialize: Select a vertex v ∈ V randomly2. Remove a vertex v ∈ V from the vertex set. Toss a ”fair” coin.

If the outcome is head the assign it to partition V1 else assignit to vertex V2.

3. Repeat above step till all vertices are assigned a partition .

I A Simple randomized algorithm that provides (probabilistic) guarantees a cut ofsize ‖E‖/2

I Algorithm Initialize the partitions V1 and V2 as empty setsFor each vertex v ∈ V , flip a fair coin. If outcome is heads then put vertex inpartition V1 else put it into partition V2.

Pseudo-code

1. Initialize: Select a vertex v ∈ V randomly2. Remove a vertex v ∈ V from the vertex set. Toss a ”fair” coin.

If the outcome is head the assign it to partition V1 else assignit to vertex V2.

3. Repeat above step till all vertices are assigned a partition .

Figure : Coin Toss Sequence = H

Figure : Coin Toss Sequence = H T

Figure : Coin Toss Sequence = H T H

Figure : Coin Toss Sequence = H T H T

Figure : Coin Toss Sequence = H T H T T

Does it Work??

Let Xe be a random variable s.t.

{1 if Edge e = (vi , vj ) crosses the partition0 otherwise

I Let X be a random variable indicating the no. of edges crossing the cut. ThenX = X1 + X2 + · · ·+ Xe

I Expected no. of edges crossing the cut =E [X ] = E [X1 + X2 + .+ Xe ] =

∑ei=1 E [Xi ]

I E [Xe ] = 1×Pr(Edge e is chosen)+0×Pr(Edge e is not chosen)

The 4 cases of partitioning

Figure : Case 1 i ∈ V1 j ∈ V1

Pr(i ∈ V1,j ∈ V1)= Pr(i ∈ V1) × Pr(j ∈ V1) = 12× 1

Pr(i ∈ V1,j ∈ V2)= Pr(i ∈ V1) × Pr(j ∈ V2) = 12× 1

Pr(i ∈ V2,j ∈ V1)= Pr(i ∈ V2) × Pr(j ∈ V1) = 12× 1

Pr(i ∈ V2,j ∈ V2)= Pr(i ∈ V2) × Pr(j ∈ V2) = 12× 1

Correctness Contd.

I E [X(i,j)] = 1×Pr(Edge e is chosen)+0×Pr(Edge e is not chosen) = 1× 12

0× 12

I Expected no. of edges =E [X ] = E [X1 + X2 + .+ Xe ] =

∑ei=1 E [Xi ] =

∑ei=1 1/2 = e/2

Logical and Practical Difficulties with Randomization

I True randomness is hard to replicate, as famously countered by Einstein :

GOD DOES NOT PLAY DICE

I A large scale randomized operation could need upto billions of randomized bitsper second

I Hard to produce that many no. of uncorrelated bits.

Derandomization

I Some Randomized algorithms have deterministic counterparts that providesimilar guarantees on output.

I The cost of de-randomization is the added complexity

I Can we derandomize maxcut ??

The Chimp Strategy

I A chimp is given a binary tree in which the leaf nodes are weighed by differentno. of bananas

I Objective : Starting from the root node, get to a leaf having atleast n/2 bananas

I Incentive?? : Obvious!!

The chimp is given a map of the binary tree but it never takes a look, but alwayssucceeds!!

The secret : At each node , the branch with the more no. of bananas is heavier andhence a little less Steep

The Chimp Strategy

I A chimp is given a binary tree in which the leaf nodes are weighed by differentno. of bananas

I Objective : Starting from the root node, get to a leaf having atleast n/2 bananas

I Incentive?? : Obvious!!

The chimp is given a map of the binary tree but it never takes a look, but alwayssucceeds!!The secret : At each node , the branch with the more no. of bananas is heavier andhence a little less Steep

Back to MAX-CUT

I Analogy: At iteration t, having already assigned vertices v1, v2, . . . vt−1 topartitions, compute expectations

I Compute e1 = E [X‖v1, v2.., vt−1, vt = 0] and e2 = E [X‖v1, v2, . . . vt−1, vt = 1]

I Assign vt to partition V1 if e1 ≥ e2 else assign it to partition V2 Assign vt topartition 1 if e1 ≥ e2 else assign it to partition 2

Correctness

I Proof by induction :

I Hypothesis: At any iteration i , E [X |v1, v2, , vi ] ≥ m/2

I Base case : E [X ] ≥ m/2 @ iteration 0

I Induction step: Assuming E [X |v1, v2, , vi − 1] ≥ m/2, where v1 . . . vi−1 are aspartitioned by the algorithm

Back to MAX-CUT

Correctness

Back to MAX-CUT

Correctness

Back to MAX-CUT

Correctness

More About Randomization

I Is randomization more powerful than determinism ??

I Are there upper bounds on increase in complexity on derandomization ??

Thank you!

Randomization at Work: An Introduction to Randomized ... · QuickSort I Proposed by C.A.R. Hoare in...

Documents

Transcript of Randomization at Work: An Introduction to Randomized ... · QuickSort I Proposed by C.A.R. Hoare in...