Analysis of AlgorithmsCS 477/677

Instructor: Monica Nicolescu

CS 477/677 2

Counting Sort

• Assumption: – The elements to be sorted are integers in the range 0

to k

• Idea:– Determine for each input element x, the number of

elements smaller than x– Place element x into its correct position in the output

array

30320352

1 2 3 4 5 6 7 8

A 77422

1 2 3 4 5

C 8

0

53332200

1 2 3 4 5 6 7 8

B

CS 477/677 3

Radix Sort

• Considers keys as numbers in a base-R number– A d-digit number will occupy a field of d

columns

• Sorting looks at one column at a time– For a d digit number, sort the least significant

digit first

– Continue sorting on the next least significant digit, until all digits have been sorted

– Requires only d passes through the list

CS 477/677 4

Bucket Sort

• Assumption: – the input is generated by a random process that distributes

elements uniformly over [0, 1)

• Idea:– Divide [0, 1) into n equal-sized buckets– Distribute the n input values into the buckets– Sort each bucket– Go through the buckets in order, listing elements in each one

• Input: A[1 . . n], where 0 ≤ A[i] < 1 for all i

• Output: elements ai sorted

• Auxiliary array: B[0 . . n - 1] of linked lists, each list initially empty

CS 477/677 5

BUCKET-SORT

Alg.: BUCKET-SORT(A, n)

for i ← 1 to n

do insert A[i] into list B[nA[i]]

for i ← 0 to n - 1

do sort list B[i] with insertion sort

concatenate lists B[0], B[1], . . . , B[n -1] together in order

return the concatenated lists

CS 477/677 6

Example - Bucket Sort

.78

.17

.39

.26

.72

.94

.21

.12

.23

.68

0

1

2

3

4

5

6

7

8

9

1

2

3

4

5

6

7

8

9

10

.21

.12 /

.72 /

.23 /

.78

.94 /

.68 /

.39 /

.26

.17

/

/

/

/

CS 477/677 7

Example - Bucket Sort

0

1

2

3

4

5

6

7

8

9

.23

.17 /

.78 /

.26 /

.72

.94 /

.68 /

.39 /

.21

.12

/

/

/

/

.17.12 .23 .26.21 .39 .68 .78.72 .94 /

Concatenate the lists from 0 to n – 1 together, in order

CS 477/677 8

Correctness of Bucket Sort

• Consider two elements A[i], A[ j]

• Assume without loss of generality that A[i] ≤ A[j]

• Then nA[i] ≤ nA[j]– A[i] belongs to the same group as A[j] or to a group

with a lower index than that of A[j]

• If A[i], A[j] belong to the same bucket:

– insertion sort puts them in the proper order

• If A[i], A[j] are put in different buckets:

– concatenation of the lists puts them in the proper order

CS 477/677 9

Analysis of Bucket Sort

Alg.: BUCKET-SORT(A, n)

for i ← 1 to n

do insert A[i] into list B[nA[i]]

for i ← 0 to n - 1

do sort list B[i] with insertion sort

concatenate lists B[0], B[1], . . . , B[n -1]

together in order

return the concatenated lists

O(n)

(n)

O(n)

(n)

CS 477/677 10

Conclusion

• Any comparison sort will take at least nlgn to sort an

array of n numbers

• We can achieve a better running time for sorting if we

can make certain assumptions on the input data:

– Counting sort: each of the n input elements is an integer in the

range 0 to k

– Radix sort: the elements in the input are integers represented

with d digits

– Bucket sort: the numbers in the input are uniformly distributed

over the interval [0, 1)

CS 477/677 11

A Job Scheduling Application

• Job scheduling

– The key is the priority of the jobs in the queue

– The job with the highest priority needs to be executed

next

• Operations

– Insert, remove maximum

• Data structures

– Priority queues

– Ordered array/list, unordered array/list

CS 477/677 12

Example

CS 477/677 13

PQ Implementations & Cost

Worst-case asymptotic costs for a PQ with N items

Insert Remove max

ordered array

ordered list

unordered array

unordered list

N

N

N

N

1

1

1

1

Can we implement both operations efficiently?

CS 477/677 14

Background on Trees

• Def: Binary tree = structure composed of a finite set of nodes that either:– Contains no nodes, or– Is composed of three disjoint sets of nodes: a root

node, a left subtree and a right subtree

2

14 8

1

16

4

3

9 10

root

Right subtreeLeft subtree

CS 477/677 15

Special Types of Trees

• Def: Full binary tree = a binary tree in which each node is either a leaf or has degree exactly 2.

• Def: Complete binary tree = a binary tree in which all leaves have the same depth and all internal nodes have degree 2.

Full binary tree

2

14 8

1

16

7

4

3

9 10

12

Complete binary tree

2

1

16

4

3

9 10

CS 477/677 16

The Heap Data Structure

• Def: A heap is a nearly complete binary tree with the following two properties:– Structural property: all levels are full, except

possibly the last one, which is filled from left to right– Order (heap) property: for any node x

Parent(x) ≥ x

Heap

5

7

8

4It doesn’t matter that 4 in

level 1 is smaller than 5 in

level 22

CS 477/677 17

Definitions

• Height of a node = the number of edges on a longest simple path from the node down to a leaf

• Depth of a node = the length of a path from the root to the node

• Height of tree = height of root node

= lgn, for a heap of n elements

2

14 8

1

16

4

3

9 10

Height of root = 3

Height of (2)= 1 Depth of (10)= 2

CS 477/677 18

Array Representation of Heaps

• A heap can be stored as an

array A.– Root of tree is A[1]

– Left child of A[i] = A[2i]

– Right child of A[i] = A[2i + 1]

– Parent of A[i] = A[ i/2 ]

– Heapsize[A] ≤ length[A]

• The elements in the subarray

A[(n/2+1) .. n] are leaves

• The root is the maximum

element of the heapA heap is a binary tree that is filled in order

CS 477/677 19

Heap Types

• Max-heaps (largest element at root), have the

max-heap property:

– for all nodes i, excluding the root:

A[PARENT(i)] ≥ A[i]

• Min-heaps (smallest element at root), have the

min-heap property:

– for all nodes i, excluding the root:

A[PARENT(i)] ≤ A[i]

CS 477/677 20

Operations on Heaps

• Maintain the max-heap property

– MAX-HEAPIFY

• Create a max-heap from an unordered array

– BUILD-MAX-HEAP

• Sort an array in place

– HEAPSORT

• Priority queue operations

CS 477/677 21

Operations on Priority Queues

• Max-priority queues support the following

operations:

– INSERT(S, x): inserts element x into set S

– EXTRACT-MAX(S): removes and returns element of

S with largest key

– MAXIMUM(S): returns element of S with largest key

– INCREASE-KEY(S, x, k): increases value of element

x’s key to k (Assume k ≥ x’s current key value)

CS 477/677 22

Maintaining the Heap Property

• Suppose a node is smaller than a child– Left and Right subtrees of i are max-heaps

• Invariant: – the heap condition is violated only at that

node

• To eliminate the violation:– Exchange with larger child– Move down the tree– Continue until node is not smaller than

children

CS 477/677 23

Maintaining the Heap Property

• Assumptions:– Left and Right

subtrees of i are max-heaps

– A[i] may be smaller than its children

Alg: MAX-HEAPIFY(A, i, n)

1. l ← LEFT(i)2. r ← RIGHT(i)3. if l ≤ n and A[l] > A[i]4. then largest ←l5. else largest ←i6. if r ≤ n and A[r] > A[largest]7. then largest ←r8. if largest i9. then exchange A[i] ↔ A[largest]10. MAX-HEAPIFY(A, largest,

n)

CS 477/677 24

Example

MAX-HEAPIFY(A, 2, 10)

A[2] violates the heap property

A[2] A[4]

A[4] violates the heap property

A[4] A[9]

Heap property restored

CS 477/677 25

MAX-HEAPIFY Running Time

• Intuitively:

– A heap is an almost complete binary tree must

process O(lgn) levels, with constant work at each

level

• Running time of MAX-HEAPIFY is O(lgn)

• Can be written in terms of the height of the heap,

as being O(h)

– Since the height of the heap is lgn

CS 477/677 26

Building a Heap

Alg: BUILD-MAX-HEAP(A)

1. n = length[A]

2. for i ← n/2 downto 1

3. do MAX-HEAPIFY(A, i, n)

• Convert an array A[1 … n] into a max-heap (n = length[A])

• The elements in the subarray A[(n/2+1) .. n] are leaves

• Apply MAX-HEAPIFY on elements between 1 and n/2

2

14 8

1

16

7

4

3

9 10

1

2 3

4 5 6 7

8 9 10

4 1 3 2 16 9 10 14 8 7A:

CS 477/677 27

Example: A 4 1 3 2 16 9 10 14 8 7

2

14 8

1

16

7

4

3

9 10

1

2 3

4 5 6 7

8 9 10

14

2 8

1

16

7

4

10

9 3

1

2 3

4 5 6 7

8 9 10

2

14 8

1

16

7

4

3

9 10

1

2 3

4 5 6 7

8 9 1014

2 8

1

16

7

4

3

9 10

1

2 3

4 5 6 7

8 9 10

14

2 8

16

7

1

4

10

9 3

1

2 3

4 5 6 7

8 9 108

2 4

14

7

1

16

10

9 3

1

2 3

4 5 6 7

8 9 10

i = 5 i = 4 i = 3

i = 2 i = 1

CS 477/677 28

Correctness of BUILD-MAX-HEAP• Loop invariant:

– At the start of each iteration of the for loop, each node i + 1, i + 2,…, n is the root of a max-heap

• Initialization:– i = n/2: Nodes n/2+1, n/2+2, …, n are leaves

they are the root of trivial max-heaps

2

14 8

1

16

7

4

3

9 10

1

2 3

4 5 6 7

8 9 10

CS 477/677 29

Correctness of BUILD-MAX-HEAP• Maintenance:

– MAX-HEAPIFY makes node i a max-heap root and preserves the property that nodes i + 1, i + 2, …, n are roots of max-heaps

– Decrementing i in the for loop reestablishes the loop invariant

• Termination:– i = 0 each node 1, 2, …, n is the root of

a max-heap (by the loop invariant)

2

14 8

1

16

7

4

3

9 10

1

2 3

4 5 6 7

8 9 10

CS 477/677 30

Running Time of BUILD MAX HEAP

Running time: O(nlgn)

• This is not an asymptotically tight upper bound

Alg: BUILD-MAX-HEAP(A)

1. n = length[A]

2. for i ← n/2 downto 1

3. do MAX-HEAPIFY(A, i, n) O(lgn)O(n)

CS 477/677 31

Running Time of BUILD MAX HEAP

• HEAPIFY takes O(h) the cost of HEAPIFY on a node i is proportional to the height of the node i in the tree

Height Level

h0 = 3 (lgn)

h1 = 2

h2 = 1

h3 = 0

i = 0

i = 1

i = 2

i = 3 (lgn)

No. of nodes

20

21

22

23

hi = h – i height of the heap rooted at level ini = 2i number of nodes at level i

i

h

iihnnT

0

)( ihh

i

i 0

2 )(nO

CS 477/677 33

Readings

• Chapter 6• Chapter 8