Lecture 5. Counting Sort / Radix Sort -...
Transcript of Lecture 5. Counting Sort / Radix Sort -...
![Page 1: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/1.jpg)
Intelligent Networking Laboratory H.CHOO 1/62Copyright 2000-2020 Intelligent Networking Laboratory
Lecture 5.
Counting Sort / Radix Sort
T. H. Cormen, C. E. Leiserson and R. L. Rivest
Introduction to Algorithms, 3rd Edition, MIT Press, 2009
Sungkyunkwan University
Hyunseung Choo
![Page 2: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/2.jpg)
Intelligent Networking Laboratory H.CHOO 2/62
Sorting So Far
Insertion Sort
Easy to code
Fast on small inputs (less than ~ 50 elements)
Fast on nearly-sorted inputs
O(n2) worst case
O(n2) average (equally-likely inputs) case
O(n2) reverse-sorted case
![Page 3: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/3.jpg)
Intelligent Networking Laboratory H.CHOO 3/62
Sorting So Far
Merge Sort
Divide-and-Conquer
Split array in half
Recursively sort subarrays
Linear-time merge step
O(n lg n) worst case
It does not sort in place
![Page 4: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/4.jpg)
Intelligent Networking Laboratory H.CHOO 4/62
Sorting So Far
Heap Sort
It uses the very useful heap data structure
Complete binary tree
Heap property: parent key > children’s keys
O(n lg n) worst case
It sorts in place
![Page 5: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/5.jpg)
Intelligent Networking Laboratory H.CHOO 5/62
Sorting So Far
Quick Sort
Divide-and-Conquer
Partition array into two subarrays, recursively sort
All of first subarray < all of second subarray
No merge step needed !
O(n lg n) average case
Fast in practice
O(n2) worst case
Naïve implementation: worst case on sorted input
Address this with randomized quicksort
![Page 6: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/6.jpg)
Intelligent Networking Laboratory H.CHOO 6/62
How Fast Can We Sort?
We will provide a lower bound, then beat it
How do you suppose we’ll beat it?
First, an observation: all of the sorting algorithms so
far are comparison sorts
The only operation used to gain ordering information about a
sequence is the pairwise comparison of two elements
Theorem: all comparison sorts are (n lg n)
![Page 7: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/7.jpg)
Intelligent Networking Laboratory H.CHOO 7/62
Decision Trees
5 7 3
5 7 3
5 3 7
3 7
3 5
![Page 8: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/8.jpg)
Intelligent Networking Laboratory H.CHOO 8/62
Decision Trees
Decision trees provide an abstraction of comparison
sorts
A decision tree represents the comparisons made by a
comparison sort.
What do the leaves represent?
How many leaves must there be?
![Page 9: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/9.jpg)
Intelligent Networking Laboratory H.CHOO 9/62
Practice Problems
What is the smallest possible depth of a leaf in a decision
tree for a comparison sort?
![Page 10: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/10.jpg)
Intelligent Networking Laboratory H.CHOO 10/62
Decision Trees
Decision trees can model comparison sorts
For a given algorithm:
One tree for each n
Tree paths are all possible execution traces
What’s the longest path in a decision tree for insertion sort?
For merge sort?
What is the asymptotic height of any decision tree for
sorting n elements?
Answer: (n lg n) (Now let’s prove it…)
![Page 11: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/11.jpg)
Intelligent Networking Laboratory H.CHOO 11/62
A Lower Bound for Comparison Sorting
Theorem: Any decision tree that sorts n elements has
height (n lg n)
What’s the minimum number of leaves?
What’s the maximum number of leaves of a binary tree
of height h?
Clearly the minimum number of leaves is less than or
equal to the maximum number of leaves
![Page 12: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/12.jpg)
Intelligent Networking Laboratory H.CHOO 12/62
A Lower Bound for Comparison Sorting
So we have…
n! 2h
Taking logarithms:
lg (n!) h
Stirling’s approximation tells us:
Thus:
n
e
nn
!
n
e
nh
lg
![Page 13: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/13.jpg)
Intelligent Networking Laboratory H.CHOO 13/62
A Lower Bound for Comparison Sorting
So we have
Thus the minimum height of a decision tree is (n lg n)
( )nn
ennn
e
nh
n
lg
lglg
lg
=
−=
![Page 14: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/14.jpg)
Intelligent Networking Laboratory H.CHOO 14/62
A Lower Bound for Comparison Sorts
Thus the time to comparison sort n elements is (n lg n)
Corollary: Heapsort and Mergesort are asymptotically
optimal comparison sorts
But the name of this lecture is “Sorting in linear time”!
How can we do better than (n lg n)?
![Page 15: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/15.jpg)
Intelligent Networking Laboratory H.CHOO 15/62
Sorting in Linear Time
Counting Sort
No comparisons between elements !
But…depends on assumption about the numbers being sorted
Basic Idea
For each input element x, determine the number of elements less than or equal to x
For each integer i (0 i k), count how many elements whose values are i
Then we know how many elements are less than or equal to i
Algorithm Storage
A[1..n]: input elements
B[1..n]: sorted elements
C[0..k]: hold the number of elements less than or equal to i
![Page 16: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/16.jpg)
Intelligent Networking Laboratory H.CHOO 16/62
Counting Sort Illustration
Networking Laboratory 16/55
(Range from 0 to 5)
![Page 17: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/17.jpg)
Intelligent Networking Laboratory H.CHOO 17/62
Counting Sort Illustration
2 5 3 0 2 3 0 3A 0 3 3B
1 2 4 5 7 8C
0 2 3 3B
1 2 3 5 7 8C
0 0 2 3 3B
0 2 3 5 7 8C
0 0 2 3 3 3B
0 2 3 4 7 8C
0 0 2 3 3 3 8B
0 2 3 4 7 7C
0 0 2 2 3 3 3 8
0 2 3 4 7 7C
B
2
4
0
1
3
5
5
8
2
2
![Page 18: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/18.jpg)
Intelligent Networking Laboratory H.CHOO 18/62
Counting Sort
(k)
(n)
(k)
(n)
(n+k)
![Page 19: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/19.jpg)
Intelligent Networking Laboratory H.CHOO 19/62
Counting Sort
![Page 20: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/20.jpg)
Intelligent Networking Laboratory H.CHOO 20/62
Counting Sort
Video Content
An illustration of Counting Sort.
![Page 21: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/21.jpg)
Intelligent Networking Laboratory H.CHOO 21/62
Counting Sort
![Page 22: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/22.jpg)
Intelligent Networking Laboratory H.CHOO 22/62
Counting Sort
Total time: O(n + k)
Usually, k = O(n)
Thus counting sort runs in O(n) time
But sorting is (n lg n)!
No contradiction
This is not a comparison sort
In fact, there are no comparisons at all !
Notice that this algorithm is stable
![Page 23: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/23.jpg)
Intelligent Networking Laboratory H.CHOO 23/62
Radix Sort
Intuitively, you might sort on the most significant
digit, then the second msd, etc.
Problem
Lots of intermediate piles of cards (read: scratch arrays) to
keep track of
Key idea
Sort the least significant digit first
RadixSort(A, d)
for i=1 to d
StableSort(A) on digit i
![Page 24: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/24.jpg)
Intelligent Networking Laboratory H.CHOO 24/62
Radix Sort Example
a b a b a c c a a a c b b a b c c a b b aa a c
Input data:
![Page 25: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/25.jpg)
Intelligent Networking Laboratory H.CHOO 25/62
Radix Sort Example
a b a b a c c a a a c b b a b c c a b b aa a c
a b c
Place into appropriate pile.
Pass 1: Looking at rightmost position.
![Page 26: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/26.jpg)
Intelligent Networking Laboratory H.CHOO 26/62
Radix Sort Example
a b a b a c
c a a
a c b
b a b
c c a
b b a
a a c
a b c
Join piles.
Pass 1: Looking at rightmost position.
![Page 27: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/27.jpg)
Intelligent Networking Laboratory H.CHOO 27/62
Radix Sort Example
a b a b a cc a a a c b b a bc c a b b a a a c
a b c
Pass 2: Looking at next position.
Place into appropriate pile.
![Page 28: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/28.jpg)
Intelligent Networking Laboratory H.CHOO 28/62
Radix Sort Example
a b a
b a c
c a a
a c bb a b
c c a
b b a
a a c
a b c
Join piles.
Pass 2: Looking at next position.
![Page 29: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/29.jpg)
Intelligent Networking Laboratory H.CHOO 29/62
Radix Sort Example
a b c
b a cc a a b a b a a c a b a b b a a c bc c a
Pass 3: Looking at last position.
Place into appropriate pile.
![Page 30: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/30.jpg)
Intelligent Networking Laboratory H.CHOO 30/62
Radix Sort Example
a b c
b a c
c a ab a ba a c
a b a
b b aa c b
c c a
Pass 3: Looking at last position.
Join piles.
![Page 31: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/31.jpg)
Intelligent Networking Laboratory H.CHOO 31/62
Radix Sort Example
b a c c a ab a ba a c a b a b b aa c b c c a
Result is sorted.
![Page 32: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/32.jpg)
Intelligent Networking Laboratory H.CHOO 32/62
Radix Sort Example
13
30
28
56
36
28
36
56
13
30
56
36
30
28
13
Radix Bins 0 1 2 3 4 5 6 7 8 9
36
30 13 56 28
36
13 28 30 56
28
36
56
13
30
13
30
28
56
36
![Page 33: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/33.jpg)
Intelligent Networking Laboratory H.CHOO 33/62
Radix Sort Example
d digits in total
![Page 34: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/34.jpg)
Intelligent Networking Laboratory H.CHOO 34/62
Radix Sort
Counting sort is obvious choice:
Sort n numbers on digits that range from 1..k
Time: O(n + k)
Each pass over n numbers with d digits takes time
O(n+k), so total time O(dn+dk)
When d is constant and k=O(n), takes O(n) time
In general, radix sort based on counting sort is
Fast
Asymptotically fast (i.e., O(n))
Simple to code
![Page 35: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/35.jpg)
Intelligent Networking Laboratory H.CHOO 35/62
Radix Sort
Video Content
An illustration of Radix Sort.
![Page 36: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/36.jpg)
Intelligent Networking Laboratory H.CHOO 36/62
Radix Sort
![Page 37: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/37.jpg)
Intelligent Networking Laboratory H.CHOO 37/62
Practice Problems
Show how to sort n integers in the range 0 to n3 – 1 in
O(n) time.
![Page 38: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/38.jpg)
Intelligent Networking Laboratory H.CHOO 38/62
Medians and Order Statistics
The ith order statistic of a set of n elements is the ith
smallest element
The median is the halfway point
Define the selection problem as:
Given a set A of n elements, find an element x A that is
larger than x-1 elements
Obviously, this can be solved in O(n lg n). Why?
Is it really necessary to sort all the numbers?
![Page 39: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/39.jpg)
Intelligent Networking Laboratory H.CHOO 39/62
Order Statistics
The ith order statistic in a set of n elements is the
ith smallest element
The minimum is thus the 1st order statistic
The maximum is the nth order statistic
The median is the n/2 order statistic
If n is even, there are 2 medians
The lower median (n+1)/2
The upper median (n+1)/2
How can we calculate order statistics?
What is the running time?
![Page 40: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/40.jpg)
Intelligent Networking Laboratory H.CHOO 40/62
Minimum and Maximum
The maximum can be found in exactly the same way
by replacing the > with < in the above algorithm.
![Page 41: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/41.jpg)
Intelligent Networking Laboratory H.CHOO 41/62
Simultaneous Min and Max
How many comparisons are needed to find the minimum element in a set? The maximum?
Can we find the minimum and maximum with less than twice the cost?
Yes:
Walk through elements by pairs
Compare each element in pair to the other
Compare the largest to maximum, smallest to minimum
o Pair (ai, ai+1), assume ai < ai+1, then
o Compare (min,ai), (ai+1,max)
o 3 comparisions for each pair.
Total cost: 3 comparisons per 2 elements
O( 3 n/2 )
![Page 42: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/42.jpg)
Intelligent Networking Laboratory H.CHOO 42/62
The Selection Problem
A more interesting problem is selection
Finding the ith smallest element of a set
We will show
A practical randomized algorithm with O(n) expected running
time
A cool algorithm of theoretical interest only with O(n) worst-
case running time
![Page 43: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/43.jpg)
Intelligent Networking Laboratory H.CHOO 43/62
Randomized Selection
Key idea: use partition() from quicksort
But, only need to examine one subarray
This savings shows up in running time: O(n)
Select the i=7th smallest
i - k
![Page 44: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/44.jpg)
Intelligent Networking Laboratory H.CHOO 44/62
Randomized Selection
A[q] A[q]
k
qp r
![Page 45: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/45.jpg)
Intelligent Networking Laboratory H.CHOO 45/62
Randomized Selection
Analyzing Randomized-Select()
Worst case: partition always 0:n-1
T(n) = T(n-1) + O(n)
= O(n2) (arithmetic series)
No better than sorting !
Best case: suppose a 9:1 partition
T(n) = T(9n/10) + O(n)
= O(n) (Master Theorem, case 3)
Better than sorting!
What if this had been a 99:1 split?
![Page 46: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/46.jpg)
Intelligent Networking Laboratory H.CHOO 46/62
Randomized Selection
Analysis of expected time
The analysis follows that of randomized quicksort, but it’s a
little different.
Let T(n) be the random variable for the running time of Randomized-Select on an input of size n, assuming
random numbers are independent.
For k = 0, 1, …, n–1, define the indicator random variable
![Page 47: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/47.jpg)
Intelligent Networking Laboratory H.CHOO 47/62
Randomized Selection
To obtain an upper bound, assume that the ith
element always falls in the larger side of the
partition:
![Page 48: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/48.jpg)
Intelligent Networking Laboratory H.CHOO 48/62
Randomized Selection
Take expectations of both sides
Let’s show that T(n) = O(n) by substitution
( ) ( )( ) ( )
( ) ( )
−
=
−
=
+
+−−
1
2/
1
0
2
1,max1
n
nk
n
k
nkTn
nknkTn
nT
![Page 49: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/49.jpg)
Intelligent Networking Laboratory H.CHOO 49/62
Randomized Selection
Assume T(k) ck for sufficiently large c:
What happened here?“Split” the recurrence
What happened here?
What happened here?
What happened here?
( )
( )
( )
( ) ( )
( ) ( )nnc
nc
nnn
nnn
c
nkkn
c
nckn
nkTn
nT
n
k
n
k
n
nk
n
nk
+
−−−=
+
−−−=
+
−=
+
+
−
=
−
=
−
=
−
=
122
1
21
22
11
2
12
2
2
)(2
)(
12
1
1
1
1
2/
1
2/
The recurrence we started with
Substitute T(n) cn for T(k)
Expand arithmetic series
Multiply it out
![Page 50: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/50.jpg)
Intelligent Networking Laboratory H.CHOO 50/62
Randomized Selection
Assume T(k) ck for sufficiently large c:
What happened here?Subtract c/2
What happened here?
What happened here?
What happened here?
The recurrence so far
Multiply it out
Rearrange the arithmetic
What we set out to prove
( ) ( )
( )
( )
( )
enough) big is c if(
24
24
24
122
1)(
cn
nccn
cn
nccn
cn
nccn
ccn
nnc
ncnT
−+−=
+−−=
++−−=
+
−−−
![Page 51: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/51.jpg)
Intelligent Networking Laboratory H.CHOO 51/62
Selection in Worst-Case Linear Time
Randomized algorithm works well in practice
What follows is a worst-case linear time algorithm,
really of theoretical interest only
Basic idea:
Generate a good partitioning element
Call this element x
![Page 52: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/52.jpg)
Intelligent Networking Laboratory H.CHOO 52/62
Selection in Worst-Case Linear Time
The algorithm in words:
1. Divide n elements into groups of 5
2. Find median of each group ( How? How long? )
3. Use Select() recursively to find median x of the n/5 medians
4. Partition the n elements around x. Let k = rank(x)
5. if (i == k) then return x
if (i < k) then use Select() recursively to find ith smallest
element in first partition
else (i > k) use Select() recursively to find (i-k)th smallest
element in last partition
A: x1 x2 x3 xn/5
xxk – 1 elements n - k elements
![Page 53: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/53.jpg)
Intelligent Networking Laboratory H.CHOO 53/62
Order Statistics Analysis
One group of 5 elements
Median
GreaterElements
LesserElements
![Page 54: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/54.jpg)
Intelligent Networking Laboratory H.CHOO 54/62
Order Statistics Analysis
All groups of 5 elements(And at most one smaller group)
Median ofMedians
GreaterMedians
LesserMedians
![Page 55: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/55.jpg)
Intelligent Networking Laboratory H.CHOO 55/62
Order Statistics Analysis
Definitely LesserElements
Definitely GreaterElements
![Page 56: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/56.jpg)
Intelligent Networking Laboratory H.CHOO 56/62
Example
Find the –11th smallest element in array:
A = {12, 34, 0, 3, 22, 4, 17, 32, 3, 28, 43, 82, 25, 27, 34,
2 ,19 ,12 ,5 ,18 ,20 ,33, 16, 33, 21, 30, 3, 47}
Divide the array into groups of 5 elements
4
17
32
3
28
12
34
0
3
22
43
82
25
27
34
2
19
12
5
18
20
33
16
33
21
30
3
47
![Page 57: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/57.jpg)
Intelligent Networking Laboratory H.CHOO 57/62
Example
3
4
17
28
32
0
3
12
22
34
25
27
34
43
82
2
5
12
18
19
16
20
21
33
33
3
30
47
Sort the groups and find their medians
![Page 58: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/58.jpg)
Intelligent Networking Laboratory H.CHOO 58/62
Example
3
4
17
28
32
0
3
12
22
34
25
27
34
43
82
2
5
12
18
19
16
20
21
33
33
3
30
47
Find the median of the medians
![Page 59: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/59.jpg)
Intelligent Networking Laboratory H.CHOO 59/62
Example
Partition the array around the median of medians (17)
First partition:
{12, 0, 3, 4, 3, 2, 12, 5, 16, 3}
Pivot:
17 (position of the pivot is q = 11)
Second partition:
{34, 22, 32, 28, 43, 82, 25, 27, 34, 19, 18, 20, 33, 33, 21, 30,
47}
To find the 6-th smallest element we would have to recur
our search in the first partition
![Page 60: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/60.jpg)
Intelligent Networking Laboratory H.CHOO 60/62
Analysis of Running Time
At least half of the medians found in step 2 are ≥
x
All but two of these groups contribute 3
elements > x
groups with 3 elements > x252
1−
n
N
e
t
w
o
r
k
i
n
g
L
a
b
o
r
a
t
o
r
y
6
0
/
6
2 610
32
52
13 −
−
nn At least elements greater than x
SELECT is called on at most elements610
76
10
3+=
−−
nnn
![Page 61: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/61.jpg)
Intelligent Networking Laboratory H.CHOO 61/62
Analysis of Running Time
Step 1: making groups of 5 elements takes
Step 2: sorting n/5 groups in O(1) time each takes
Step 3: calling SELECT on n/5 medians takes time
Step 4: partitioning the n-element array around x takes
Step 5: recursing on one partition takes
T(n) = T(n/5) + T(7n/10 + 6) + O(n)
Show that T(n) = O(n)
O(n)
O(n)
T(n/5)
O(n)
≤ T(7n/10 + 6)
![Page 62: Lecture 5. Counting Sort / Radix Sort - monet.skku.edumonet.skku.edu/wp-content/uploads/2020/04/Algorithm_05.pdf · Intelligent Networking Laboratory H.CHOO 2/62 Sorting So Far Insertion](https://reader030.fdocuments.in/reader030/viewer/2022040214/5ec83a76295a8e085130f59f/html5/thumbnails/62.jpg)
Intelligent Networking Laboratory H.CHOO 62/62
Analysis of Running Time
T(n) ≤ c n/5 + c (7n/10 + 7) + an
≤ cn/5 + 7cn/10 + 7c + an
= 9cn/10 + 7c + an
= cn + (-cn/10 + 7c + an)
≤ cn if: -cn/10 + 7c + an ≤ 0
Let’s show that T(n) = O(n) by substitution