Post on 29-Jan-2021
Algorithm Design and Analysis
Yuxi Fu
BASICS, Department of Computer ScienceShanghai Jiaotong University
December 29, 2010
Part A. Introduction
I. Basic Concepts in Algorithmic Analysis
Algorithm
An algorithm is a procedure that consists of a finite set ofinstructions which, given an input from some set of possible inputs,enables us to obtain an output through a systematic execution ofthe instructions that terminates in a finite number of steps.
Theorem proving is in general not algorithmic.
Theorem verification is often algorithmic.
Quotation from Donald E. Knuth
Computer Science is the study of algorithms.
Remark on Algorithm
The word ‘algorithm’ is derived from the name of Muhamma ibnMūsā al-Khwārizm̄i (780?-850?), a Muslim mathematician whoseworks introduced Arabic numerals and algebraic concepts toWestern mathematics. The word ‘algebra’ stems from the title ofhis book Kitab al jahr wa’l-muqābala. (American HeritageDictionary)
Algorithm vs. Program
A program is an implementation of an algorithm, or algorithms.
A program does not necessarily terminate.
What is Computer Science?
Computer Science is the study of problem solving usingcomputing machines. The computing machines must be physicallyfeasible.
What is Computer Science?
I. Theory of Computation is to understand the notion ofcomputation in a formal framework.
Some well known models are: the general recursive functionmodel of Gödel and Church, Church’s λ-calculus, Post systemmodel, Turing machine model, RAM, etc.
II. Computability Theory studies what problems can be solved bycomputers.
III. Computational Complexity studies how much resource isnecessary in order to solve a problem.
IV. Theory of Algorithm studies how problems can be solved.
What is Computer Science?
I. Theory of Computation is to understand the notion ofcomputation in a formal framework.
Some well known models are: the general recursive functionmodel of Gödel and Church, Church’s λ-calculus, Post systemmodel, Turing machine model, RAM, etc.
II. Computability Theory studies what problems can be solved bycomputers.
III. Computational Complexity studies how much resource isnecessary in order to solve a problem.
IV. Theory of Algorithm studies how problems can be solved.
What is Computer Science?
I. Theory of Computation is to understand the notion ofcomputation in a formal framework.
Some well known models are: the general recursive functionmodel of Gödel and Church, Church’s λ-calculus, Post systemmodel, Turing machine model, RAM, etc.
II. Computability Theory studies what problems can be solved bycomputers.
III. Computational Complexity studies how much resource isnecessary in order to solve a problem.
IV. Theory of Algorithm studies how problems can be solved.
What is Computer Science?
I. Theory of Computation is to understand the notion ofcomputation in a formal framework.
Some well known models are: the general recursive functionmodel of Gödel and Church, Church’s λ-calculus, Post systemmodel, Turing machine model, RAM, etc.
II. Computability Theory studies what problems can be solved bycomputers.
III. Computational Complexity studies how much resource isnecessary in order to solve a problem.
IV. Theory of Algorithm studies how problems can be solved.
The problem to start with: Search and Ordering.
Linear Search, First Example of an Algorithm
Algorithm 1.1 LinearSearchInput: An array A[1..n] of n elements and an element x .Output: j if x = A[j ], 1 ≤ j ≤ n, and 0 otherwise.
1. j ← 12. while j < n and x 6= A[j ]3. j ← j + 14. end while5. if x = A[j ] then return j else return 0
Binary Search
Algorithm 1.2 BinarySearchInput: An array A[1..n] of n elements sorted in nondecreasingorder and an element x .Output: j if x = A[j ], 1 ≤ j ≤ n, and 0 otherwise.
1. low ← 1; high← n; j ← 02. while low ≤ high and j = 03. mid ← b(low + high)/2c4. if x = A[mid ] then j ← mid5. else if x < A[mid ] then high← mid − 16. else low ← mid + 17. end while8. return j
Analysis of BinarySearch
Suppose x ≥ 35. A run of BinarySearch on A[1..14] (see below) is
1 4 5 7 8 9 10 12 15 22 23 27 32 35
↓
12 15 22 23 27 32 35
↓
27 32 35
↓
35
Analysis of BinarySearch
The complexity of the algorithm is the number of comparison.
The number of comparison is maximum if x ≥ A[n].The number of comparisons is the same as the number ofiterations.
In the second iteration, the number of elements inA[mid + 1..n] is exactly bn/2c.In the j-th iteration, the number of elements in A[mid + 1..n]is exactly bn/2j−1c.The maximum number of iteration is the j such thatbn/2j−1c = 1, which is equivalent toj − 1 ≤ log n < j .Hence j = blog nc+ 1.
Merging Two Sorted Lists
Algorithm 1.3 MergeInput: An array A[1..m] of elements and three indices p, q and r .with 1 ≤ p ≤ q < r ≤ m, such that both the subarray A[p..q] andA[q + 1..r ] are sorted individually in nondecreasing order.Output: A[p..r ] contains the result of merging the two subarraysA[p..q] and A[q + 1..r ].Comment: B[p..r ] is an auxiliary array
Merging Two Sorted Lists
1. s ← p; t ← q + 1; k ← p2. while s ≤ q and t ≤ r3. if A[s] ≤ A[t] then4. B[k]← A[s]5. s ← s + 16. else7. B[k]← A[t]8. t ← t + 19. end if
10. k ← k + 111. end while12. if s = q + 1 then B[k ..r ]← A[t..r ]13. else B[k ..r ]← A[s..q]13. end if13. A[p..r ]← B[p..r ]
Analysis of Merge
Suppose A[p..q] has m elements and A[q + 1..r ] has n elements.The number of comparisons done by Algorithm Merge is
at least min{m, n}; andat most m + n − 1.
Selection Sort
Algorithm 1.4 SelectionSortInput: An array A[1..n] of n elements.Output: A[1..n] sorted in nondecreasing order.
1. for i ← 1 to n − 12. k ← i3. for j ← i + 1 to n4. if A[j ] < A[k] then k ← j5. end for6. if k 6= i then interchange A[i ] and A[k]7. end for
Analysis of SelectionSort
The number of comparisons carried out by Algorithm SelectionSortis precisely
n−1∑i=1
(n − i) = n(n − 1)2
Insertion Sort
Algorithm 1.5 InsertionSortInput: An array A[1..n] of n elements.Output: A[1..n] sorted in nondecreasing order.
1. for i ← 2 to n2. x ← A[i ]3. j ← i − 14. while j > 0 and A[j ] > x5. A[j + 1]← A[j ]6. j ← j − 17. end while8. A[j + 1]← x9. end for
Analysis of InsertionSort
The number of comparisons carried out by Algorithm InsertionSortis at least
n − 1
and at mostn∑
i=2
(i − 1) = n(n − 1)2
Bottom-Up Merge Sort
Algorithm 1.6 BottomUpSortInput: An array A[1..n] of n elements.Output: A[1..n] sorted in nondecreasing order.
1. t ← 12. while t < n3. s ← t; t ← 2s; i ← 04. while i + t ≤ n5. Merge(A, i + 1, i + s, i + t)6. i ← i + t7. end while8. if i + s < n then Merge(A, i + 1, i + s, n)9. end while
Analysis of BottomUpSort
Suppose that n is a power of 2, say n = 2k .
The outer while loop is executed k = log n times.
Step 8 is never invoked.
In the j-th iteration of the outer while loop, there are2k−j = n/2j pairs of arrays of size 2j−1.
The number of comparisons needed in the merge of two sortedarrays in the j-th iteration is at least 2j−1 and at most 2j − 1.The number of comparisons in BottomUpSort is at least
k∑j=1
(n
2j)2j−1 =
k∑j=1
n
2=
n log n
2
The number of comparisons in BottomUpSort is at most
k∑j=1
(n
2j)(2j − 1) =
k∑j=1
(n − n2j
) = n log n − n + 1
Time Complexity
Computational Complexity evolved from 1960’s, flourished in1970’s and 1980’s.
Time is the most precious resource.
Important to human.
Order of Growth
Our main concern is about the order of growth.
Our estimates of time are relative rather than absolute.
Our estimates of time are machine independent.
Our estimates of time are about the behavior of the algorithmunder investigation on large input instances.
So we are measuring the asymptotic running time of thealgorithms.
Order of Growth
Our main concern is about the order of growth.
Our estimates of time are relative rather than absolute.
Our estimates of time are machine independent.
Our estimates of time are about the behavior of the algorithmunder investigation on large input instances.
So we are measuring the asymptotic running time of thealgorithms.
The O-Notation
The O-notation provides an upper bound of the running time; itmay not be indicative of the actual running time of an algorithm.
Definition (O-Notation)
Let f (n) and g(n) be functions from the set of natural numbers tothe set of nonnegative real numbers. f (n) is said to be O(g(n)),written f (n) = O(g(n)), if
∃c .∃n0.∀n ≥ n0.f (n) ≤ cg(n)
Intuitively, f grows no faster than some constant times g .
The Ω-Notation
The Ω-notation provides a lower bound of the running time; it maynot be indicative of the actual running time of an algorithm.
Definition (Ω-Notation)
Let f (n) and g(n) be functions from the set of natural numbers tothe set of nonnegative real numbers. f (n) is said to be Ω(g(n)),written f (n) = Ω(g(n)), if
∃c .∃n0.∀n ≥ n0.f (n) ≥ cg(n)
Clearly f (n) = O(g(n)) if and only if g(n) = Ω(f (n)).
The Θ-Notation
The Θ-notation provides an exact picture of the growth rate of therunning time of an algorithm.
Definition (Θ-Notation)
Let f (n) and g(n) be functions from the set of natural numbers tothe set of nonnegative real numbers. f (n) is said to be Θ(g(n)),written f (n) = Θ(g(n)), if both f (n) = O(g(n)) andf (n) = Ω(g(n)).
Clearly f (n) = Θ(g(n)) if and only if g(n) = Θ(f (n)).
Examples
10n2 + 20n = O(n2).
log n2 = O(n).
log nk = Ω(log n).
n! = O((n + 1)!).
Examples
Consider the series∑n
j=1 log j . Clearly,
n∑j=1
log j ≤n∑
j=1
log n = n log n
On the other hand,
n∑j=1
log j ≥bn/2c∑j=1
log(n
2) = bn/2c log(n
2) = bn/2c log n − bn/2c
That isn∑
j=1
log j = Ω(n log n)
Examples
log n! =∑n
j=1 log j = Ω(n log n).
2n = O(n!).
The o-Notation
Definition (o-Notation)
Let f (n) and g(n) be functions from the set of natural numbers tothe set of nonnegative real numbers. f (n) is said to be o(g(n)),written f (n) = o(g(n)), if
∀c .∃n0.∀n ≥ n0.f (n) < cg(n)
The ω-Notation
Definition (ω-Notation)
Let f (n) and g(n) be functions from the set of natural numbers tothe set of nonnegative real numbers. f (n) is said to be ω(g(n)),written f (n) = ω(g(n)), if
∀c .∃n0.∀n ≥ n0.f (n) > cg(n)
Definition in Terms of Limits
Suppose limn→∞ f (n)/g(n) exists.
limn→∞ f (n)/g(n) 6=∞ implies f (n) = O(g(n)).limn→∞ f (n)/g(n) 6= 0 implies f (n) = Ω(g(n)).limn→∞ f (n)/g(n) = c implies f (n) = Θ(g(n)).
limn→∞ f (n)/g(n) = 0 implies f (n) = o(g(n)).
limn→∞ f (n)/g(n) =∞ implies f (n) = ω(g(n)).
A Helpful Analogy
f (n) = O(g(n)) is similar to f (n) ≤ g(n).f (n) = o(g(n)) is similar to f (n) < g(n).
f (n) = Θ(g(n)) is similar to f (n) = g(n).
f (n) = Ω(g(n)) is similar to f (n) ≥ g(n).f (n) = ω(g(n)) is similar to f (n) > g(n).
Complexity Classes
An equivalence relation R on the set of complexity functions isdefined as follows: fRg if and only if f (n) = Θ(g(n)).
A complexity class is an equivalence class of R.
The equivalence classes can be ordered by ≺ defined as follows:f ≺ g iff f (n) = o(g(n)).
Complexity Classes
An equivalence relation R on the set of complexity functions isdefined as follows: fRg if and only if f (n) = Θ(g(n)).
A complexity class is an equivalence class of R.
The equivalence classes can be ordered by ≺ defined as follows:f ≺ g iff f (n) = o(g(n)).
Complexity Classes
An equivalence relation R on the set of complexity functions isdefined as follows: fRg if and only if f (n) = Θ(g(n)).
A complexity class is an equivalence class of R.
The equivalence classes can be ordered by ≺ defined as follows:f ≺ g iff f (n) = o(g(n)).
Space Complexity
The space complexity is defined to be the number of cells (workspace)) needed to carry out an algorithm, excluding the spaceallocated to hold the input.
The exclusion of the input space is to make sense the sublinearspace complexity.
Space Complexity
The space complexity is defined to be the number of cells (workspace)) needed to carry out an algorithm, excluding the spaceallocated to hold the input.
The exclusion of the input space is to make sense the sublinearspace complexity.
Space Complexity
It is clear that the work space of an algorithm can not exceed therunning time of the algorithm. That is S(n) = O(T (n)).
Trade-off between time complexity and space complexity.
Space Complexity
It is clear that the work space of an algorithm can not exceed therunning time of the algorithm. That is S(n) = O(T (n)).
Trade-off between time complexity and space complexity.
Optimal Algorithm
In general, if we can prove that any algorithm to solve problem Πmust be Ω(f (n)), then we call any algorithm to solve problem Π intime O(f (n)) an optimal algorithm for problem π.
HOW do we estimate time complexity?
Counting the Iterations
Algorithm 1.7 Count1Input: n = 2k , for some positive integer k .Output: count = number of times Step 4 is executed.
1. count ← 0;2. while n ≥ 13. for j ← 1 to n4. count ← count + 15. end for6. n← n/27. end while8. return count
Counting the Iterations
Algorithm 1.7 Count1Input: n = 2k , for some positive integer k .Output: count = number of times Step 4 is executed.
1. count ← 0;2. while n ≥ 13. for j ← 1 to n4. count ← count + 15. end for6. n← n/27. end while8. return count
while is executed k + 1 times; for is executed n, n/2, . . . , 1 times
k∑j=0
n
2j= n
k∑j=0
1
2j= n(2− 1
2k) = 2n − 1 = Θ(n)
Counting the Iterations
Algorithm 1.8 Count2Input: A positive integer n.Output: count = number of times Step 5 is executed.
1. count ← 0;2. for i ← 1 to n3. m← bn/ic4. for j ← 1 to m5. count ← count + 16. end for7. end for8. return count
Counting the Iterations
Algorithm 1.8 Count2Input: A positive integer n.Output: count = number of times Step 5 is executed.
1. count ← 0;2. for i ← 1 to n3. m← bn/ic4. for j ← 1 to m5. count ← count + 16. end for7. end for8. return count
The inner for is executed n, bn/2c, bn/4c, . . . , bn/nc times
Θ(n log n) =n∑
i=1
(n
i− 1) ≤
n∑i=1
bnic ≤
n∑i=1
n
i= Θ(n log n)
Counting the Frequency of Basic Operations
Definition
An elementary operation in an algorithm is called a basic operationif it is of highest frequency to within a constant factor among allother elementary operations.
Worst Case Analysis
Consider the following algorithm:
1. if n is odd then k ← BinarySearch(A, x)2. else k ← LinearSearch(A, x)
In the worst case, the running time is Ω(log(n)) and O(n).
Average Case Analysis
Take Algorithm InsertionSort for instance. Two assumptions:
A[1..n] contains the numbers 1 through n.
All n! permutations are equally likely.
The number of comparisons for inserting element A[i ] in its properposition, say j , is on average the following
i − 1i
+i∑
j=2
i − j + 1i
=i − 1
i+
i−1∑j=1
j
i=
i
2− 1
i+
1
2
The average number of comparisons performed by AlgorithmInsertionSort is
n∑i=2
(i
2− 1
i+
1
2) =
n2
4+
3n
4−
n∑i=1
1
i
Amortized Analysis
In amortized analysis, we average out the time taken by theoperation throughout the execution of the algorithm, and refer tothis average as the amortized running time of that operation.
Amortized analysis guarantees the average cost of the operation,and thus the algorithm, in the worst case.
Amortized Analysis
In amortized analysis, we average out the time taken by theoperation throughout the execution of the algorithm, and refer tothis average as the amortized running time of that operation.
Amortized analysis guarantees the average cost of the operation,and thus the algorithm, in the worst case.
Amortized Analysis
Consider the following algorithm:
1. for j ← 1 to n2. x ← A[j ]3. Append x to the list4. if x is even then5. while pred(x) is odd do delete pred(x)6. end if7. end for
The total number of elementary operations of insertions anddeletions is between n and 2n− 1. So the time complexity is Θ(n).
It follows that the time used to delete each element is O(1)amortized time.
Amortized Analysis
Consider the following algorithm:
1. for j ← 1 to n2. x ← A[j ]3. Append x to the list4. if x is even then5. while pred(x) is odd do delete pred(x)6. end if7. end for
The total number of elementary operations of insertions anddeletions is between n and 2n− 1. So the time complexity is Θ(n).
It follows that the time used to delete each element is O(1)amortized time.
Amortized Analysis
Consider the following algorithm:
1. for j ← 1 to n2. x ← A[j ]3. Append x to the list4. if x is even then5. while pred(x) is odd do delete pred(x)6. end if7. end for
The total number of elementary operations of insertions anddeletions is between n and 2n− 1. So the time complexity is Θ(n).
It follows that the time used to delete each element is O(1)amortized time.
Input Size and Problem Instance
Suppose that the following integer
21024 − 1
is a legitimate input of an algorithm. What is the size of the input?
Input Size and Problem Instance
Algorithm 1.9 FIRSTInput: A positive integer n and an array A[1..n] with A[j ] = j for1 ≤ j ≤ n.Output:
∑nj=1 A[j ].
1. sum← 0;2. for j ← 1 to n3. sum← sum + A[j ]4. end for5. return sum
The input size is n. The time complexity is O(n). It is linear time.
Input Size and Problem Instance
Algorithm 1.10 SECONDInput: A positive integer n.Output:
∑nj=1 j .
1. sum← 0;2. for j ← 1 to n3. sum← sum + j4. end for5. return sum
The input size is k = blog nc+ 1. The time complexity is O(2k). Itis exponential time.
Exercise
1.5, 1.9, 1.13, 1.16, 1.17, 1.25, 1.31, 1.32, 1.33, 1.37
II. Mathematical Preliminaries
Mathematical Induction
Consider the fibonacci sequence 1, 1, 2, 3, 5, 8, . . ., defined by
f (n)def=
1, if n = 11, if n = 2f (n − 1) + f (n − 2), if n > 2
Let φ = 1+√
52 . Prove that f (n) ≤ φ
n−1 for all n.
Notice that φ2 = φ+ 1. So the induction may go as follows:
f (n) = f (n − 1) + f (n − 2) ≤ φn−2 + φn−3 = φn−3(φ+ 1) = φn−1
Mathematical Induction
Consider the fibonacci sequence 1, 1, 2, 3, 5, 8, . . ., defined by
f (n)def=
1, if n = 11, if n = 2f (n − 1) + f (n − 2), if n > 2
Let φ = 1+√
52 . Prove that f (n) ≤ φ
n−1 for all n.
Notice that φ2 = φ+ 1. So the induction may go as follows:
f (n) = f (n − 1) + f (n − 2) ≤ φn−2 + φn−3 = φn−3(φ+ 1) = φn−1
The Pigeonhole Principle
This principle is easy, powerful and indispensable.
Example. If n balls are distributed into m boxes, then(i) one box must contain at least dn/me balls, and(ii) one box must contain at most bn/mc balls.
Proof. If no box contains dn/me balls or more, then the total ballsin all the boxes are at most
m(d nme − 1) ≤ m( n
m+
m − 1m
− 1) < n
If all boxes have greater than bn/mc balls, then the total balls inall the boxes are at least
m(b nmc+ 1) ≥ m( n
m− m − 1
m+ 1) > n
The Pigeonhole Principle
This principle is easy, powerful and indispensable.
Example. If n balls are distributed into m boxes, then(i) one box must contain at least dn/me balls, and(ii) one box must contain at most bn/mc balls.
Proof. If no box contains dn/me balls or more, then the total ballsin all the boxes are at most
m(d nme − 1) ≤ m( n
m+
m − 1m
− 1) < n
If all boxes have greater than bn/mc balls, then the total balls inall the boxes are at least
m(b nmc+ 1) ≥ m( n
m− m − 1
m+ 1) > n
Logarithms
Suppose that b is a positive real number, x a real number and y apositive real number such that y = bx . Then x is called thelogarithm of y to the base b, denoted by
x = logb y
Here b is referred to as the base of the logarithm. We shall writelog x for log2 x and ln x for loge x , the natural logarithm, where
e = limn→∞
(1 +1
n)n = 1 +
1
1!+
1
2!+
1
3!+ · · · = 2.7182818 . . .
It is well known that
ln x =
∫ x1
1
tdt
Logarithms
Suppose that b is a positive real number, x a real number and y apositive real number such that y = bx . Then x is called thelogarithm of y to the base b, denoted by
x = logb y
Here b is referred to as the base of the logarithm. We shall writelog x for log2 x and ln x for loge x , the natural logarithm, where
e = limn→∞
(1 +1
n)n = 1 +
1
1!+
1
2!+
1
3!+ · · · = 2.7182818 . . .
It is well known that
ln x =
∫ x1
1
tdt
Logarithms
Facts:
logb xy = logb x + logb y
logb cy = y logb c
logb x =loga x
loga b
x loga y = y loga x
Floor and Ceiling Functions
The floor of x , denoted by bxc, is the greatest integer less than orequal to x .
The ceiling of x , denoted by dxe, is the least integer greater thanor equal to x .
Facts
bx/2c+ dx/2e = xb−xc = −dxed−xe = −bxc
Floor and Ceiling Functions
The floor of x , denoted by bxc, is the greatest integer less than orequal to x .
The ceiling of x , denoted by dxe, is the least integer greater thanor equal to x .
Facts
bx/2c+ dx/2e = xb−xc = −dxed−xe = −bxc
Floor and Ceiling Functions
The floor of x , denoted by bxc, is the greatest integer less than orequal to x .
The ceiling of x , denoted by dxe, is the least integer greater thanor equal to x .
Facts
bx/2c+ dx/2e = xb−xc = −dxed−xe = −bxc
Floor and Ceiling Functions
Theorem
Let f (x) be a continuous and monotonically increasing functionsuch that x is an integer whenever f (x) is an integer. Then
bf (bxc)c = bf (x)cdf (dxe)e = df (x)e
Proof: bf (x)c = f (x ′) for some integer x ′. Hence x ′ ≤ bxc. Thusbf (x)c = f (x ′) ≤ f (bxc). It follows that bf (x)c ≤ bf (bxc)c.
Floor and Ceiling Functions
Theorem
Let f (x) be a continuous and monotonically increasing functionsuch that x is an integer whenever f (x) is an integer. Then
bf (bxc)c = bf (x)cdf (dxe)e = df (x)e
Proof: bf (x)c = f (x ′) for some integer x ′. Hence x ′ ≤ bxc. Thusbf (x)c = f (x ′) ≤ f (bxc). It follows that bf (x)c ≤ bf (bxc)c.
Floor and Ceiling Functions
For example
d√dxee = d
√xe
blogbxcc = blog xcddxe/ne = dx/nebbxc/nc = bx/nc
Summation
Arithmetic Series:
n∑j=1
j =n(n + 1)
2= Θ(n2)
Geometric Series:
n∑j=0
c j =cn+1 − 1
c − 1= Θ(cn)
Summation
Arithmetic Series:
n∑j=1
j =n(n + 1)
2= Θ(n2)
Geometric Series:
n∑j=0
c j =cn+1 − 1
c − 1= Θ(cn)
Approximation of Summations by Integration
Suppose that f (x) is continuous and monotonically increasing.Then ∫ n
m−1f (x)dx ≤
n∑j=m
f (j) ≤∫ n+1m
f (x)dx
Suppose that f (x) is continuous and monotonically decreasing.Then ∫ n+1
mf (x)dx ≤
n∑j=m
f (j) ≤∫ nm−1
f (x)dx
Approximation of Summations by Integration
Consider the summation∑n
j=1 jk . Since jk is increasing, we have∫ n
0xkdx ≤
n∑j=1
xk ≤∫ n+1
1xkdx
That isnk+1
k + 1≤
n∑j=1
xk ≤ (n + 1)k+1 − 1
k + 1
Hencen∑
j=1
xk = Θ(nk+1)
Approximation of Summations by Integration
Consider the Harmonic Series
Hn =n∑
j=1
1
j
Since 1x is monotonically decreasing, we have∫ n+11
1
xdx ≤
n∑j=1
1
j= 1 +
n∑j=2
1
j≤ 1 +
∫ n1
1
xdx
That is
ln(n + 1) ≤n∑
j=1
1
j≤ 1 + ln n
Hence
Hn =n∑
j=1
1
j= Θ(ln n) = Θ(log n)
Approximation of Summations by Integration
Consider the seriesn∑
j=1
log j
Since log x is monotonically increasing, we have∫ n1
log(x)dx ≤n∑
j=2
log j =n∑
j=1
log j ≤ log n +∫ n
1log(x)dx
That is
n log n− n log e + log e ≤n∑
j=1
log j ≤ log n + n log n− n log e + log e
Hencen∑
j=1
log j = Θ(n log n)
Recursively Defined Functions
The complexity functions for recursive algorithms are oftenrecursively defined. This is why solving recursive functionequations are of crucial importance to algorithm analysis.
Solution of Linear Homogeneous Recurrence
Consider the following recursively defined function
f (n) = a1f (n − 1) + a2f (n − 2) + . . .+ ak f (n − k) (1)
Substituting xn for f (n) in (1) yields
xn = a1xn−1 + a2x
n−2 + . . .+ akxn−k
Dividing both sides by xn−k yields
xk = a1xk−1 + a2x
k−2 + . . .+ ak
or equivalently
xk − a1xk−1 − a2xk−2 − . . .− ak = 0 (2)
(2) is called the characteristic function of (1).
Solution of Linear Homogeneous Recurrence
Consider the following recursively defined function
f (n) = a1f (n − 1) + a2f (n − 2) + . . .+ ak f (n − k) (1)
Substituting xn for f (n) in (1) yields
xn = a1xn−1 + a2x
n−2 + . . .+ akxn−k
Dividing both sides by xn−k yields
xk = a1xk−1 + a2x
k−2 + . . .+ ak
or equivalently
xk − a1xk−1 − a2xk−2 − . . .− ak = 0 (2)
(2) is called the characteristic function of (1).
Solution of Linear Homogeneous Recurrence
Consider the following recursively defined function
f (n) = a1f (n − 1) + a2f (n − 2) + . . .+ ak f (n − k) (1)
Substituting xn for f (n) in (1) yields
xn = a1xn−1 + a2x
n−2 + . . .+ akxn−k
Dividing both sides by xn−k yields
xk = a1xk−1 + a2x
k−2 + . . .+ ak
or equivalently
xk − a1xk−1 − a2xk−2 − . . .− ak = 0 (2)
(2) is called the characteristic function of (1).
Solution of Linear Homogeneous Recurrence
Consider the following recursively defined function
f (n) = a1f (n − 1) + a2f (n − 2) + . . .+ ak f (n − k) (1)
Substituting xn for f (n) in (1) yields
xn = a1xn−1 + a2x
n−2 + . . .+ akxn−k
Dividing both sides by xn−k yields
xk = a1xk−1 + a2x
k−2 + . . .+ ak
or equivalently
xk − a1xk−1 − a2xk−2 − . . .− ak = 0 (2)
(2) is called the characteristic function of (1).
Solution of Linear Homogeneous Recurrence
Let x1, . . . , xk be the solutions to (2). Then f (n) = xni is a solution
to (1) when n ≥ k .
Suppose the general solution to (1) is
f (n) = c1xn1 + . . .+ ckx
nk .
In order for f (n) to be a solution when n < k, we solve thefollowing linear equation
x1c1 + . . .+ xkck = f (0),...
xk−11 c1 + . . .+ xk−1k ck = f (k − 1).
Solution of Linear Homogeneous Recurrence
Let x1, . . . , xk be the solutions to (2). Then f (n) = xni is a solution
to (1) when n ≥ k .
Suppose the general solution to (1) is
f (n) = c1xn1 + . . .+ ckx
nk .
In order for f (n) to be a solution when n < k, we solve thefollowing linear equation
x1c1 + . . .+ xkck = f (0),...
xk−11 c1 + . . .+ xk−1k ck = f (k − 1).
Solution of Linear Homogeneous Recurrence
Let x1, . . . , xk be the solutions to (2). Then f (n) = xni is a solution
to (1) when n ≥ k .
Suppose the general solution to (1) is
f (n) = c1xn1 + . . .+ ckx
nk .
In order for f (n) to be a solution when n < k, we solve thefollowing linear equation
x1c1 + . . .+ xkck = f (0),...
xk−11 c1 + . . .+ xk−1k ck = f (k − 1).
Solution of Linear Homogeneous Recurrence
Example 1. The sequence 1, 4, 16, 64, 256, . . . is generated by
f (n) = 3f (n − 1) + 4f (n − 2)
The solution is f (n) = 4n.
Solution of Linear Homogeneous Recurrence
Example 2. Consider the Fibonacci sequence 0, 1, 1, 2, 3, 5, 8, . . .which can be computed by the recurrence
f (n)def=
0, if n = 01, if n = 1f (n − 1) + f (n − 2), if n ≥ 2
The characteristic function is x2 − x − 1 = 0, the two roots ofwhich are
1 +√
5
2and
1−√
5
2
Solution of Linear Homogeneous Recurrence
Example 2. Consider the Fibonacci sequence 0, 1, 1, 2, 3, 5, 8, . . .which can be computed by the recurrence
f (n)def=
0, if n = 01, if n = 1f (n − 1) + f (n − 2), if n ≥ 2
The characteristic function is x2 − x − 1 = 0, the two roots ofwhich are
1 +√
5
2and
1−√
5
2
Solution of Linear Homogeneous Recurrence
Let the solution be
f (n) = c1
(1 +√
5
2
)n+ c2
(1−√
5
2
)n
Then
c1 + c2 = 0
c11 +√
5
2+ c2
1−√
5
2= 1
One has that
c1 =1√5
and c2 = −1√5
So the solution is
f (n) =1√5
(1 +√
5
2
)n− 1√
5
(1−√
5
2
)n
Solution of Linear Homogeneous Recurrence
Let the solution be
f (n) = c1
(1 +√
5
2
)n+ c2
(1−√
5
2
)nThen
c1 + c2 = 0
c11 +√
5
2+ c2
1−√
5
2= 1
One has that
c1 =1√5
and c2 = −1√5
So the solution is
f (n) =1√5
(1 +√
5
2
)n− 1√
5
(1−√
5
2
)n
Solution of Linear Homogeneous Recurrence
Let the solution be
f (n) = c1
(1 +√
5
2
)n+ c2
(1−√
5
2
)nThen
c1 + c2 = 0
c11 +√
5
2+ c2
1−√
5
2= 1
One has that
c1 =1√5
and c2 = −1√5
So the solution is
f (n) =1√5
(1 +√
5
2
)n− 1√
5
(1−√
5
2
)n
Solution of Linear Homogeneous Recurrence
Let the solution be
f (n) = c1
(1 +√
5
2
)n+ c2
(1−√
5
2
)nThen
c1 + c2 = 0
c11 +√
5
2+ c2
1−√
5
2= 1
One has that
c1 =1√5
and c2 = −1√5
So the solution is
f (n) =1√5
(1 +√
5
2
)n− 1√
5
(1−√
5
2
)n
Solution of Inhomogeneous Recurrence, I
Consider the inhomogeneous recurrence
f (n) = f (n − 1) + g(n) (3)
It is easy to see that the solution to (3) is
f (n) = f (0) +n∑
i=1
g(i)
Solution of Inhomogeneous Recurrence, I
Consider the inhomogeneous recurrence
f (n) = f (n − 1) + g(n) (3)
It is easy to see that the solution to (3) is
f (n) = f (0) +n∑
i=1
g(i)
Solution of Inhomogeneous Recurrence, II
Consider the inhomogeneous recurrence
f (n) = g(n)f (n − 1) (4)
It is easy to see that the solution to (4) is
f (n) = g(n)g(n − 1) . . . g(1)f (0)
Solution of Inhomogeneous Recurrence, II
Consider the inhomogeneous recurrence
f (n) = g(n)f (n − 1) (4)
It is easy to see that the solution to (4) is
f (n) = g(n)g(n − 1) . . . g(1)f (0)
Solution of Inhomogeneous Recurrence, III
Consider the inhomogeneous recurrence
f (n) = g(n)f (n − 1) + h(n) (5)
where g(n) > 0 for all n > 0.
Let f ′(0) satisfy the following equations:
f (0) = f ′(0)
f (n) = g(n)g(n − 1) . . . g(1)f ′(n) if n > 0
It follows from (5) that
f ′(n) = f ′(n − 1) + h(n)g(n)g(n − 1) . . . g(1)
for n > 1.
Solution of Inhomogeneous Recurrence, III
Consider the inhomogeneous recurrence
f (n) = g(n)f (n − 1) + h(n) (5)
where g(n) > 0 for all n > 0.
Let f ′(0) satisfy the following equations:
f (0) = f ′(0)
f (n) = g(n)g(n − 1) . . . g(1)f ′(n) if n > 0
It follows from (5) that
f ′(n) = f ′(n − 1) + h(n)g(n)g(n − 1) . . . g(1)
for n > 1.
Solution of Inhomogeneous Recurrence, IV
Consider the inhomogeneous recurrence
f (n) = nf (n − 1) + n! (6)
Let f (n) = n!f ′(n), assuming that f ′(0) = 0. It follows from (6)that
n!f ′(n) = n(n − 1)!f ′(n − 1) + n!
Thereforef ′(n) = f ′(n − 1) + 1
So f ′(n) = n. Hence
f (n) = n!f ′(n) = nn!
Solution of Inhomogeneous Recurrence, IV
Consider the inhomogeneous recurrence
f (n) = nf (n − 1) + n! (6)
Let f (n) = n!f ′(n), assuming that f ′(0) = 0. It follows from (6)that
n!f ′(n) = n(n − 1)!f ′(n − 1) + n!
Thereforef ′(n) = f ′(n − 1) + 1
So f ′(n) = n. Hence
f (n) = n!f ′(n) = nn!
Solution of Inhomogeneous Recurrence, V
Consider the inhomogeneous recurrence
f (0) = 0
f (n) = 2f (n − 1) + n, if n > 0
Let f (0) = f ′(0) and f (n) = 2nf ′(n). Then
2nf ′(n) = 2(2n−1f ′(n − 1)) + n
which simplifies to
f ′(n) = f ′(n − 1) + n2n
The rest should now be routine.
Solution of Inhomogeneous Recurrence, V
Consider the inhomogeneous recurrence
f (0) = 0
f (n) = 2f (n − 1) + n, if n > 0
Let f (0) = f ′(0) and f (n) = 2nf ′(n). Then
2nf ′(n) = 2(2n−1f ′(n − 1)) + n
which simplifies to
f ′(n) = f ′(n − 1) + n2n
The rest should now be routine.
Solution of Divide-and-Conquer Recurrence
The recurrences appeared in the divide-and-conquer algorithm takethe following form
f (n) =
{d if n ≤ n0a1f (n/c1) + . . .+ apf (n/cp) + g(n) if n > n0
We shall discuss the most common techniques for solving suchrecurrences.
Solution of Divide-and-Conquer Recurrence
The recurrences appeared in the divide-and-conquer algorithm takethe following form
f (n) =
{d if n ≤ n0a1f (n/c1) + . . .+ apf (n/cp) + g(n) if n > n0
We shall discuss the most common techniques for solving suchrecurrences.
Expanding the Recurrence
Consider
f (n) =
{d if n = 1af (n/c) + bnx if n > 1
where a, c are nonnegative integers, b, d , x are nonnegativeconstants, and n = ck . Then
f (n) =
{bnx logc(n) + dn
x if a = cx(d + bc
x
a−cx)
nlogc (a) −(
bcx
a−cx)
nx if a 6= cx
Expanding the Recurrence
f (n) = af (n/c) + bnx
= a(af (n/c2) + b(n/c)x) + bnx
= a2f (n/c2) + ab(n/c)x + bnx
...
= ak f (n/ck) + (a/cx)k−1bnx + . . .+ (a/cx)1bnx + bnx
= dalogc n + bnxk−1∑j=0
(a/cx)j
= dnlogc a + bnxk−1∑j=0
(a/cx)j
Substitution
In this method, we guess a solution and try to prove it byappealing to mathematical induction. Unlike the common naturalinduction, here the induction is carried out with some unknownconstants. The constants will be chosen in such a way that theinduction is not affected.
Substitution
Consider
f (n) =
{d if n = 1f (bn/2c) + f (dn/2e) + bn if n > 1
where b, d are nonnegative constants.
When n is a power of 2, we already know that
f (n) = bn log n + dn
We are led to guess that
f (n) = cbn log n + dn
for some c > 0.
Substitution
Consider
f (n) =
{d if n = 1f (bn/2c) + f (dn/2e) + bn if n > 1
where b, d are nonnegative constants.
When n is a power of 2, we already know that
f (n) = bn log n + dn
We are led to guess that
f (n) = cbn log n + dn
for some c > 0.
Substitution
Now
f (n) = f (bn/2c) + f (dn/2e) + bn≤ cbbn/2c logbn/2c+dbn/2c+ cbdn/2e logdn/2e+ddn/2e+ bn≤ cbbn/2c logdn/2e+ cbdn/2e logdn/2e+ dn + bn≤ cbn logdn/2e+ dn + bn≤ cbn log((n + 1)/2) + dn + bn= cbn log(n + 1)− cbn + dn + bn
It is sufficient to require that
cbn log(n + 1)− cbn + dn + bn ≤ cbn log n + dn
That is c log(n + 1)− c + 1 ≤ c log n. We can then fix some valuefor c .
Substitution
Now
f (n) = f (bn/2c) + f (dn/2e) + bn≤ cbbn/2c logbn/2c+dbn/2c+ cbdn/2e logdn/2e+ddn/2e+ bn≤ cbbn/2c logdn/2e+ cbdn/2e logdn/2e+ dn + bn≤ cbn logdn/2e+ dn + bn≤ cbn log((n + 1)/2) + dn + bn= cbn log(n + 1)− cbn + dn + bn
It is sufficient to require that
cbn log(n + 1)− cbn + dn + bn ≤ cbn log n + dn
That is c log(n + 1)− c + 1 ≤ c log n. We can then fix some valuefor c .
Exercise
2.10, 2.21
III. Data Structures
Graph
A graph G = (V ,E ) consists of a set of vertices V = {v1, . . . , vn}and a set of edges E . G is either directed or undirected.
An undirected graph is said to be complete if there is an edgebetween each pair of its vertices. A complete undirected graphwith n vertices is denoted by Kn.
An undirected graph G = (V ,E ) is said to be bipartite if V can bepartitioned into two disjoint subsets X and Y such that each edgehas one end in X and the other end in Y . Let m = |X | andn = |Y |. If there is an edge between each vertex x ∈ X and y ∈ Y ,then it is called a complete bipartite graph, and is denoted by Km,n.
Graph
A graph G = (V ,E ) consists of a set of vertices V = {v1, . . . , vn}and a set of edges E . G is either directed or undirected.
An undirected graph is said to be complete if there is an edgebetween each pair of its vertices. A complete undirected graphwith n vertices is denoted by Kn.
An undirected graph G = (V ,E ) is said to be bipartite if V can bepartitioned into two disjoint subsets X and Y such that each edgehas one end in X and the other end in Y . Let m = |X | andn = |Y |. If there is an edge between each vertex x ∈ X and y ∈ Y ,then it is called a complete bipartite graph, and is denoted by Km,n.
Graph
A graph G = (V ,E ) consists of a set of vertices V = {v1, . . . , vn}and a set of edges E . G is either directed or undirected.
An undirected graph is said to be complete if there is an edgebetween each pair of its vertices. A complete undirected graphwith n vertices is denoted by Kn.
An undirected graph G = (V ,E ) is said to be bipartite if V can bepartitioned into two disjoint subsets X and Y such that each edgehas one end in X and the other end in Y . Let m = |X | andn = |Y |. If there is an edge between each vertex x ∈ X and y ∈ Y ,then it is called a complete bipartite graph, and is denoted by Km,n.
Two Representations of Graph
A graph G = (V ,E ) can be represented by a boolean matrix M,called the adjacency matrix of G , defined as M[i , j ] = 1 if and onlyif (vi , vj) is an edge of G . The space complexity of such arepresentation is Θ(|V |2).
A graph G = (V ,E ) can be represented by an adjacency list. Thespace complexity of such a representation is Θ(|E |+ |V |).
Two Representations of Graph
A graph G = (V ,E ) can be represented by a boolean matrix M,called the adjacency matrix of G , defined as M[i , j ] = 1 if and onlyif (vi , vj) is an edge of G . The space complexity of such arepresentation is Θ(|V |2).
A graph G = (V ,E ) can be represented by an adjacency list. Thespace complexity of such a representation is Θ(|E |+ |V |).
Planar Graph
A graph G = (V ,E ) is planar if it can be embedded in the planewithout edge crossing. The Euler’s formula for the planar graphssays that v − e + r = 2,
n is the set of vertices,
e is the set of edges, and
r is the set of regions.
Moreover e ≤ 3v − 6.
Tree
A tree is a connected undirected graph that contains no cycles. IfT is a tree with n vertices, then
Any two vertices of T are connected by a unique path;
T has exactly n − 1 edges;The addition of one more edge to T creates a cycle.
When analyzing the time and space complexity in the context oftrees, the number of edges is insignificant.
Rooted Tree
A rooted tree T is a tree with a distinguished vertex r called theroot of T .
parent, child, sibling, leaf, internal vertex, ancestor, descendant,subtree, depth, height.
Rooted Tree
A rooted tree T is a tree with a distinguished vertex r called theroot of T .
parent, child, sibling, leaf, internal vertex, ancestor, descendant,subtree, depth, height.
Binary Tree
A binary tree is full if all its internal vertices have exactly twochildren.
A full binary tree is complete if all its leaves have the same depth.
A tree is almost complete if it is complete except that possibly oneor more leaves that occupy the rightmost positions may be missing.
Binary Tree
A binary tree is full if all its internal vertices have exactly twochildren.
A full binary tree is complete if all its leaves have the same depth.
A tree is almost complete if it is complete except that possibly oneor more leaves that occupy the rightmost positions may be missing.
Binary Tree
A binary tree is full if all its internal vertices have exactly twochildren.
A full binary tree is complete if all its leaves have the same depth.
A tree is almost complete if it is complete except that possibly oneor more leaves that occupy the rightmost positions may be missing.
Facts about Binary Tree
In a binary tree, the number of vertices at level j is at most 2j .
Let n be the number of vertices in a binary tree T of height k .Then
n ≤k∑
j=0
2j = 2k+1 − 1
If T is almost complete, then
2k ≤ n ≤ 2k+1 − 1
The height of an almost complete tree is blog nc.
In a full binary tree, the number of leaves is equal to the numberof internal vertices plus one.
Facts about Binary Tree
In a binary tree, the number of vertices at level j is at most 2j .
Let n be the number of vertices in a binary tree T of height k .Then
n ≤k∑
j=0
2j = 2k+1 − 1
If T is almost complete, then
2k ≤ n ≤ 2k+1 − 1
The height of an almost complete tree is blog nc.
In a full binary tree, the number of leaves is equal to the numberof internal vertices plus one.
Facts about Binary Tree
In a binary tree, the number of vertices at level j is at most 2j .
Let n be the number of vertices in a binary tree T of height k .Then
n ≤k∑
j=0
2j = 2k+1 − 1
If T is almost complete, then
2k ≤ n ≤ 2k+1 − 1
The height of an almost complete tree is blog nc.
In a full binary tree, the number of leaves is equal to the numberof internal vertices plus one.
Binary Tree
A (almost) complete tree with n vertices can be representedefficiently by an array A[1..n] that lists its vertices according to thefollowing relationship:
The left and the right children, if any, of a vertex stored inA[j ] are stored in A[2j ] and A[2j + 1] respectively.
The parent of a vertex stored in A[j ] is stored in A[bj/2c].
Binary Search Tree
A binary search tree is a binary tree in which the vertices arelabeled with elements from a linearly ordered set in the followingmanner:
All elements stored in the left subtree of a vertex v are lessthan the element stored in v .
All elements stored in the right subtree of a vertex v aregreater than the element stored in v .
IV. Heaps and the Disjoint Sets Data Structures
Heap
A heap is a data structure that supports two operations:
inserting an element, and
finding the element of maximum value.
Formally, a heap is an almost complete binary tree with verticessatisfying the heap property: If v and p(v) are a vertex and itsparent respectively, then the key of the item stored in p(v) is notless than the key of the item stored in v .
Heap
A heap is a data structure that supports two operations:
inserting an element, and
finding the element of maximum value.
Formally, a heap is an almost complete binary tree with verticessatisfying the heap property: If v and p(v) are a vertex and itsparent respectively, then the key of the item stored in p(v) is notless than the key of the item stored in v .
Operations on Heap
delete-ma[H]: Delete and return an item of maximum key from anonempty heap H.
insert[H, x ]: Insert item x into heap H.
delete[H, i ]: Delete the i-th item from heap H.
makeheap[A]: Transform array A into a heap.
Operations on Heap
delete-ma[H]: Delete and return an item of maximum key from anonempty heap H.
insert[H, x ]: Insert item x into heap H.
delete[H, i ]: Delete the i-th item from heap H.
makeheap[A]: Transform array A into a heap.
Operations on Heap
delete-ma[H]: Delete and return an item of maximum key from anonempty heap H.
insert[H, x ]: Insert item x into heap H.
delete[H, i ]: Delete the i-th item from heap H.
makeheap[A]: Transform array A into a heap.
Operations on Heap
delete-ma[H]: Delete and return an item of maximum key from anonempty heap H.
insert[H, x ]: Insert item x into heap H.
delete[H, i ]: Delete the i-th item from heap H.
makeheap[A]: Transform array A into a heap.
Sift-Up
Algorithm 3.1 SiftUpInput: An array H[1..n] and an index i between 1 and n.Output: H[i ] is moved up, if necessary, so that it is not lager thanits parent.
1. done ← false2. if i = 1 then exit3. repeat4. if key(H[i ]) > key(H[bi/2c])5. then interchange H[i ] and H[bi/2c]6. else done ← true7. i ← bi/2c8. until i = 1 or done
Sift-Down
Algorithm 3.2 SiftDownInput: An array H[1..n] and an index i between 1 and n.Output: H[i ] is percolated down, if necessary, so that it is notsmaller than its children.
1. done ← false2. if 2i > n then exit3. repeat4. i ← 2i5. if i + 1 ≤ n and key(H[i + 1]) > key(H[i ])6. then i ← i + 17. if key(H[bi/2c]) < key(H[i ])8. then interchange H[i ] and H[bi/2c]9. else done ← true
10. until 2i > n or done
Heap Insertion
Algorithm 3.3 InsertInput: A heap H[1..n] and a heap element x .Output: A new heap H[1..n + 1] with x being one of its elements.
1. n← n + 12. H[n]← x3. SiftUp(H, n)
The time required to insert one element into a heap of size n isO(log n).
Heap Deletion
Algorithm 3.4 DeleteInput: A nonempty heap H[1..n] and an index i between 1 and n.Output: A new heap H[1..n − 1] after H[i ] is removed.
1. x ← H[i ]2. y ← H[n]3. n← n − 14. if i = n + 1 then exit5. H[i ]← y6. if key(y) ≥ key(x) then SiftUp(H, i)7. else SiftDown(H, i)8. end if
The time required to insert one element into a heap of size n isO(log n).
DeleteMax
Algorithm 3.5 DeleteMaxInput: A heap H[1..n].Output: An element x of maximum key is retutrned and deletedfrom the heap.
1. x ← H[1]2. Delete(H, 1)3. return x
Creating a Heap
Algorithm 3.6 MakeHeapInput: An array A[1..n] of n elements.Output: A[1..n] is transformed into a heap.
1. for i ← bn/2c downto 12. SiftDown(A, i)3. end for
Complexity of MakeHeap
Let T be the almost complete tree corresponding to A[1..n].
The height of T is k = logbnc.The total number of iterations is bounded by
k−1∑i=0
= 20(k) + 21(k − 1) + . . .+ 2k−1(1)
=k∑
i=1
i2k−i
= 2kk∑
i=1
i/2i
≤ 2n
Heapsort
Algorithm 3.7 HeapsortInput: An array A[1..n] of n elements.Output: Array A[1..n] sorted in nondecreasing order.
1. MakeHeap(A)2. for j ← n downto 23. interchange A[1] and A[j ]4. SiftDown(A[1..j − 1], 1)5. end for
Disjoint Sets Data Structures
A set S of n distinct elements, partitioned into disjoint sets.Initially all disjoint sets are singleton sets.
Each set has a distinguished representative.
There are two operations on the distinct sets
Find(x): Find and return the representative of the setcontaining x .
Union(x , y): Replace the two sets containing x and y by theirunion. The representative of the union is either that of x orthat of y , to be determined later.
The goal is to design efficient algorithms for the two operations.
Disjoint Sets Data Structures
A set S of n distinct elements, partitioned into disjoint sets.Initially all disjoint sets are singleton sets.
Each set has a distinguished representative.
There are two operations on the distinct sets
Find(x): Find and return the representative of the setcontaining x .
Union(x , y): Replace the two sets containing x and y by theirunion. The representative of the union is either that of x orthat of y , to be determined later.
The goal is to design efficient algorithms for the two operations.
Disjoint Sets Data Structures
A set S of n distinct elements, partitioned into disjoint sets.Initially all disjoint sets are singleton sets.
Each set has a distinguished representative.
There are two operations on the distinct sets
Find(x): Find and return the representative of the setcontaining x .
Union(x , y): Replace the two sets containing x and y by theirunion. The representative of the union is either that of x orthat of y , to be determined later.
The goal is to design efficient algorithms for the two operations.
Disjoint Sets Data Structures
A set S of n distinct elements, partitioned into disjoint sets.Initially all disjoint sets are singleton sets.
Each set has a distinguished representative.
There are two operations on the distinct sets
Find(x): Find and return the representative of the setcontaining x .
Union(x , y): Replace the two sets containing x and y by theirunion. The representative of the union is either that of x orthat of y , to be determined later.
The goal is to design efficient algorithms for the two operations.
Representations of the Disjoint Sets Data Structures
Tree representation: The representatives are the roots; a childpoints to its parent; roots have null pointers.
Array representation: array representation of the trees. Thefollowing is an array representation. What is its treerepresentation?
1 2 3 4 5 6 7 8 9 10 11
0 3 0 8 2 2 1 0 0 1 1
Representations of the Disjoint Sets Data Structures
Tree representation: The representatives are the roots; a childpoints to its parent; roots have null pointers.
Array representation: array representation of the trees. Thefollowing is an array representation. What is its treerepresentation?
1 2 3 4 5 6 7 8 9 10 11
0 3 0 8 2 2 1 0 0 1 1
Algorithm
Straightforward algorithm
Find(x): Follow the path to the root.
Union(x , y): Let the pointer of root(x) points to root(y).
There is a problem:
A tree may degenerate. Suppose we start with{1, }, {2}, . . . , {n} and perform the following sequence ofactions: Union(1, 2),Union(2, 3), . . . ,Union(n − 1, n) followedby Find(1),Find(2), . . . ,Find(n). The cost of n Findoperations is Θ(n2).
Algorithm
Straightforward algorithm
Find(x): Follow the path to the root.
Union(x , y): Let the pointer of root(x) points to root(y).
There is a problem:
A tree may degenerate. Suppose we start with{1, }, {2}, . . . , {n} and perform the following sequence ofactions: Union(1, 2),Union(2, 3), . . . ,Union(n − 1, n) followedby Find(1),Find(2), . . . ,Find(n). The cost of n Findoperations is Θ(n2).
The Union by Rank Heuristic
Each vertex of a tree is attached with a rank (the height of thevertex). When performing Union(x , y), we do
If rank(x) < rank(y), we make y the parent of x .
If rank(x) > rank(y), we make x the parent of y .
If rank(x) = rank(y), we make y the parent of x and increasethe rank of y by 1.
Fact: The number of vertices in a tree with root x is at least2rank(x).Fact: The time complexity of a sequence of m interspersed Unionand Find operations is O(m log n).
The Union by Rank Heuristic
Each vertex of a tree is attached with a rank (the height of thevertex). When performing Union(x , y), we do
If rank(x) < rank(y), we make y the parent of x .
If rank(x) > rank(y), we make x the parent of y .
If rank(x) = rank(y), we make y the parent of x and increasethe rank of y by 1.
Fact: The number of vertices in a tree with root x is at least2rank(x).Fact: The time complexity of a sequence of m interspersed Unionand Find operations is O(m log n).
Path Compression
In a Find(x) operation, after the root y is found, we traverse thepath from x to y once again and change the parent pointers of allvertices along the path to point directly to y .
Find
Algorithm 3.8 FindInput: A vertex x .Output: root(x), the root of the tree containing x .
1. y ← x2. while p(y) 6= null do y ← p(y) end while3. root ← y4. y ← x5. while p(y) 6= null6. w ← p(y)7. p(y)← root8. y ← w9. end while
10. return root
Union
Algorithm 3.9 UnionInput: Two elements x , y .Output: The union of the two trees containing x and yrespectively.
1. u ← Find(x); v ← Find(y)2. if rank(u) ≤ rank(v) then3. p(u)← v4. if rank(u) = rank(v) then rank(v)← rank(v) + 15. else p(v)← u6. end if
Example
Let S be {1, 2, 3, 4, 5, 6, 7, 8, 9}. Demonstrate the procedure ofperforming the following operations:Union(1, 2),Union(3, 4),Union(5, 6),Union(7, 8),Union(2, 4),Union(8, 9),Union(6, 8),Find(5),Union(4, 8),Find(1).
Part B. Techniques Based on Recursion
This part is concerned with a particular class of algorithm, calledrecursive algorithms.
Recursion is the process of dividing a problem into subproblemsthat are identical in structure to the original problem. Combiningthe solutions of these subproblems gives rise to the solution of theoriginal problem.
This part is concerned with a particular class of algorithm, calledrecursive algorithms.
Recursion is the process of dividing a problem into subproblemsthat are identical in structure to the original problem. Combiningthe solutions of these subproblems gives rise to the solution of theoriginal problem.
V. Induction
Induction is based on the well known proof technique ofmathematical induction.
Given a problem with parameter n, designing an algorithm byinduction is based on the fact that if we know how to solve theproblem when presented with a parameter less than n, called theinduction hypothesis, then our task reduces to extending thatsolution to include those instances with parameter n.
Induction is based on the well known proof technique ofmathematical induction.
Given a problem with parameter n, designing an algorithm byinduction is based on the fact that if we know how to solve theproblem when presented with a parameter less than n, called theinduction hypothesis, then our task reduces to extending thatsolution to include those instances with parameter n.
Selection Sort by Induction
Algorithm 5.1 SelectionSortRecInput: An array A[1..n] of n elements.Output: A[1..n] sorted in nondecreasing order.
1. sort(1)
Procedure sort(i)
1. if i < n then2. k ← i3. for j ← i + 1 to n4. if A[j ] < A[k] then k ← j5. end for6. if k 6= i then interchange A[i ] and A[k]7. sort(i + 1)8. end if
Analysis of SelectionSortRec
Let C (n) denote the number of element comparisons performed bysort. Then
C (n) =
{0 if n = 1C (n − 1) + (n − 1) if n ≥ 2
The running time is proportional to the number of elementcomparisons. Therefore
C (n) =n(n − 1)
2= Θ(n2)
Analysis of SelectionSortRec
Let C (n) denote the number of element comparisons performed bysort. Then
C (n) =
{0 if n = 1C (n − 1) + (n − 1) if n ≥ 2
The running time is proportional to the number of elementcomparisons. Therefore
C (n) =n(n − 1)
2= Θ(n2)
Insertion Sort by Induction
Algorithm 5.2 InsertionSortRecInput: An array A[1..n] of n elements.Output: A[1..n] sorted in nondecreasing order.
1. sort(n)
Procedure sort(i)
1. if i > 1 then2. x ← A[i ]3. sort(i − 1)3. j ← i − 14. while j > 0 and A[j ] > x5. A[j + 1]← A[j ]6. j ← j − 17. end while8. A[j + 1]← x9. end if
Radix Sort
Let L = {a1, . . . , an} be a list of numbers each consisting ofexactly k digits. For instance the left most column is a list of 8numbers of 4 digits.
7467 6792 9134 9134 12391247 9134 1239 9187 12473275 3275 1247 1239 32756792 4675 7467 1247 46759187 7467 3275 3275 67929134 1247 4675 7467 74674675 9187 9187 4675 91341239 1239 6792 6792 9187
Radix Sort
Algorithm 5.3 RadixSortInput: A linked list of numbers L = {a1, . . . , an} and k , thenumber of digits.Output: L sorted in nondecreasing order.
1. for j ← 1 to k2. Prepare 10 empty lists L0, . . . , L93. while L is not empty4. a← next element in L; Delete a from L5. i ← j-th digit in a; Append a to list Li6. end while7. L← L08. for i ← 1 to 99. L← L, Li
10. end for11. end for12. return L
Radix Sort
The time complexity is Θ(kn).
Integer Exponentiation
Algorithm 5.4 ExpRecInput: A real number x and a nonnegative integer n.Output: xn.
1. power(x , n)
Procedure power(x ,m)
1. if m = 0 then y ← 12. else3. y ← power(x , bm/2c)4. y ← y 25. if m is odd then y ← xy6. end if7. return y
The Number of Multiplications
Let C (x , n) be the number of multiplications performed byExpRec . Then clearly
C (x , 0) = 0
C (x , bn/2c) + 1 ≤ C (x , n) ≤ C (x , bn/2c) + 2
It follows that blog nc ≤ C (x , n) ≤ 2blog nc. So the timecomplexity is Θ(log n).
The Number of Multiplications
Let C (x , n) be the number of multiplications performed byExpRec . Then clearly
C (x , 0) = 0
C (x , bn/2c) + 1 ≤ C (x , n) ≤ C (x , bn/2c) + 2
It follows that blog nc ≤ C (x , n) ≤ 2blog nc. So the timecomplexity is Θ(log n).
Integer Exponentiation
Algorithm 5.5 ExpInput: A real number x and a nonnegative integer n.Output: xn.
1. y ← 12. Let n be dkdk−1 . . . d0 in binary notation.3. for j ← k downto 04. y ← y 25. if dj = 1 then y ← xy6. end for7. return y
Evaluating Polynomials (Horner’s Rule)
Algorithm 5.6 HornerInput: A sequence of n + 2 real number a0, a1, . . . , an and x .Output: Pn(x) = anxn + an−1xn−1 + . . . a1x + a0.
1. p ← an2. for j ← 1 to n3. p ← xp + an−j4. end for5. return p
Horner costs n multiplications and n additions.
Generating Permutations
Algorithm 5.7 Permutations1Input: A positive integer n.Output: All permutations of the numbers 1, 2, . . . , n.
1. for j ← 1 to n2. P[j ]← j3. end for4. perm1(1)
Generating Permutations
Algorithm 5.7 Permutations1Procedure perm1(m)
1. if m = n then output P[1..n]2. else3. for j ← m to n4. interchange P[j ] and P[m]5. perm1(m + 1)6. interchange P[j ] and P[m]7. comment: At this point P[m..n] = m,m + 1, . . . , n.8. end for9. end if
Analysis of Permutations1
Consider perm1(1).
1 Step 1 is executed n! times. Altogether it takes nn! to output.
2 The number of the iterations of the for loop in Step 3 is
f (n) =
{0 if n = 1nf (n − 1) + n if n ≥ 2
Let h(n) be such that n!h(n) = f (n). Therefore
h(n) = h(n − 1) + 1(n − 1)!
=n−1∑j=1
1
j!<∞∑j=1
1
j!= e − 1
Hence f (n) = n!h(n) = Θ(n!).
Generating Permutations
Algorithm 5.8 Permutations2Input: A positive integer n.Output: All permutations of the numbers 1, 2, . . . , n.
1. for j ← 1 to n2. P[j ]← 03. end for4. perm2(n)
Generating Permutations
Algorithm 5.8 Permutations2Procedure perm2(m)
1. if m = 0 then output P[1..n]2. else3. for j ← 1 to n4. if P[j ] = 0 then5. P[j ]← m6. perm2(m − 1)7. P[j ]← 08. end if8. end for9. end if
Analysis of Permutations2
The number of the iterations of the for loop is
f (m) =
{0 if m = 0mf (m − 1) + n if m ≥ 1
Let m!h(m) = f (m). Then
h(m) = h(m − 1) + n(m − 1)!
= nm−1∑j=1
1
j!< n
∞∑j=1
1
j!= (e − 1)n
Hence f (n) = Θ(nn!).
Analysis of Permutations2
The number of the iterations of the for loop is
f (m) =
{0 if m = 0mf (m − 1) + n if m ≥ 1
Let m!h(m) = f (m). Then
h(m) = h(m − 1) + n(m − 1)!
= nm−1∑j=1
1
j!< n
∞∑j=1
1
j!= (e − 1)n
Hence f (n) = Θ(nn!).
Finding the Majority Element
Let A[1..n] be a sequence of integers. An integer a in A is calledthe majority of A if it appears more than bn/2c times in A.
Observation: If two different elements in the original sequence areremoved, then the majority in the original sequence remains themajority in the new sequence.
Finding the Majority Element
Let A[1..n] be a sequence of integers. An integer a in A is calledthe majority of A if it appears more than bn/2c times in A.
Observation: If two different elements in the original sequence areremoved, then the majority in the original sequence remains themajority in the new sequence.
Finding the Majority Element
Algorithm 5.9 MajorityInput: An array A[1..n] of n elements.Output: The majority element if it exists; otherwise none.
1. c ← candidate(1)2. count ← 03. for j ← 1 to n4. if A[j ] = c then count ← count + 15. end for6. if count > bn/2c then return c7. else return none
Finding the Majority Element
Procedure candidate(m)
1. j ← m; c ← A[m]; count ← 12. while j < n and count > 03. j ← j + 14. if A[j ] = c then count ← count + 15. else count ← count − 16. end while7. if j = n then return c8. else return candidate(j + 1)
Exercise
5.8, 5.12, 5.14, 5.33
VI. Divide and Conquer
The terminology “Divide and Conquer” has been given to apowerful algorithm design technique that is used to solve a varietyof problems.
In its simplest form, a divide-and-conquer algorithm divides theproblem instance into a number of subinstances (in most cases 2),recursively solves each subinstance separately, and then combinesthe solutions to the subinstances to obtain the solution to theoriginal problem instance.
The terminology “Divide and Conquer” has been given to apowerful algorithm design technique that is used to solve a varietyof problems.
In its simplest form, a divide-and-conquer algorithm divides theproblem instance into a number of subinstances (in most cases 2),recursively solves each subinstance separately, and then combinesthe solutions to the subinstances to obtain the solution to theoriginal problem instance.
A Simple Example
The following algorithm finds the minimum and the maximumelements in the array A[1..n].
1. x ← A[1]; y ← A[1]2. for i ← 2 to n3. if A[i ] < x then x ← A[i ]4. if A[i ] > y then y ← A[j ]5. end for6. return (x , y)
The number of element comparisons is 2n − 2.
A Divide and Conquer Solution
Using the divide-and-conquer approach we may have a solution asfollows:
Algorithm 6.1 MinMaxInput: An array A[1..n] of n integers, where n is a power of 2.Output: (x , y): the minimum and the maximum integers in A.
1. minmax(1, n)
A Divide and Conquer Solution
Procedure minmax(low , high)
1. if high − low = 1 then2. if A[low ] < A[high] then return (A[low ],A[high])3. else return (A[high],A[low ])4. end if5. else6. mid ← b(low + high)/2c7. (x1, y1)← minmax(low ,mid)8. (x2, y2)← minmax(mid + 1, high)9. x ← min{x1, x2}
10. y ← max{y1, y2}11. return (x , y)12. end if
Analysis of MinMax
Assuming n = 2k , the number of comparisons is calculated asfollows:
C (n) =
{1 if n = 22C (n/2) + 2 if n > 2
Hence
C (n) = 2C (n/2) + 2
= 2(2C (n/4) + 2) + 2
= 2k−1C (n/2k−1) + 2k−1 + 2k−2 + . . .+ 2
= n/2 + 2k − 2= 3n/2− 2
Analysis of MinMax
Assuming n = 2k , the number of comparisons is calculated asfollows:
C (n) =
{1 if n = 22C (n/2) + 2 if n > 2
Hence
C (n) = 2C (n/2) + 2
= 2(2C (n/4) + 2) + 2
= 2k−1C (n/2k−1) + 2k−1 + 2k−2 + . . .+ 2
= n/2 + 2k − 2= 3n/2− 2
Binary Search via Recursion
Algorithm 6.2 BinarySearchRecInput: An array A[1..n] of n elements sorted in nondecreasingorder and an element x .Output: j if x = A[j ] for some j ∈ {1..n}, and 0 otherwise.
1. binarysearch(1, n)
Procedure binarysearch(low , high)
1. if low > high then return 02. else3. mid ← b(low + high)/2c4. if x = A[mid ] then return mid5. else if x < A[mid ] then return
binarysearch(low ,mid − 1)6. else return binarysearch(mid + 1, high)7. end if
Analysis of BinarySearchRec
The number of comparisons is calculated as follows:
C (n) ≤{
1 if n = 11 + C (bn/2c) if n ≥ 2
Let 2k−1 ≤ n < 2k for some k . Then
C (n) ≤ (k − 1) + C (bn/2k−1c) = k = blog nc+ 1
Analysis of BinarySearchRec
The number of comparisons is calculated as follows:
C (n) ≤{
1 if n = 11 + C (bn/2c) if n ≥ 2
Let 2k−1 ≤ n < 2k for some k . Then
C (n) ≤ (k − 1) + C (bn/2k−1c) = k = blog nc+ 1
Mergesort
Algorithm 6.3 MergesortInput: An array A[1..n] of n elements.Output: A[1..n] sorted in nondecreasing order.
1. mergesort(A, 1, n)
Procedure mergesort(A, low , high)
1. if low < high then2. mid ← b(low + high)/2c3. mergesort(A, low ,mid)4. mergesort(A,mid + 1, high)5. MERGE (A, low ,mid , high)6. end if
Analysis of Mergesort
The number of comparisons needed to merge two sorted subarraysis between n/2 and n − 1.
The minimum number of comparisons:
C (n) ≤{
0 if n = 12C (n/2) + n/2 if n ≥ 2
The maximum number of comparisons:
C (n) ≤{
0 if n = 12C (n/2) + n − 1 if n ≥ 2
Analysis of Mergesort
The number of comparisons needed to merge two sorted subarraysis between n/2 and n − 1.
The minimum number of comparisons:
C (n) ≤{
0 if n = 12C (n/2) + n/2 if n ≥ 2
The maximum number of comparisons:
C (n) ≤{
0 if n = 12C (n/2) + n − 1 if n ≥ 2
Analysis of Mergesort
The number of comparisons needed to merge two sorted subarraysis between n/2 and n − 1.
The minimum number of comparisons:
C (n) ≤{
0 if n = 12C (n/2) + n/2 if n ≥ 2
The maximum number of comparisons:
C (n) ≤{
0 if n = 12C (n/2) + n − 1 if n ≥ 2
The Divide and Conquer Paradigm
Divide Step: In this step of the algorithm, the input is partitionedinto p ≥ 1 parts, each of which is strictly less than n, the size ofthe original instance.
Conquer Step: This step consists of performing p recursive call(s)if the problem size is greater than some predefined threshold n0.
Combine Step: In this step, the solution to the p recursive call(s)are combined to obtain the desired output.
The Divide and Conquer Paradigm
Divide Step: In this step of the algorithm, the input is partitionedinto p ≥ 1 parts, each of which is strictly less than n, the size ofthe original instance.
Conquer Step: This step consists of performing p recursive call(s)if the problem size is greater than some predefined threshold n0.
Combine Step: In this step, the solution to the p recursive call(s)are combined to obtain the desired output.
The Divide and Conquer Paradigm
Divide Step: In this step of the algorithm, the input is partitionedinto p ≥ 1 parts, each of which is strictly less than n, the size ofthe original instance.
Conquer Step: This step consists of performing p recursive call(s)if the problem size is greater than some predefined threshold n0.
Combine Step: In this step, the solution to the p recursive call(s)are combined to obtain the desired output.
The Divide and Conquer Algorithm
If the size of the instance I is “small”, then solve the problemusing a straightforward method and return the answer. Otherwise,continue to the next step.
Divide the instance I into p subinstances I1, . . . , Ip ofapproximately the same size.
Recursively call the algorithm on each subinstance Ij , 1 ≤ j ≤ p, toobtain p partial solutions.
Combine the results of the p partial solutions to obtain the solutionto the original instance I . Return the solution of instance I .
The Divide and Conquer Algorithm
If the size of the instance I is “small”, then solve the problemusing a straightforward method and return the answer. Otherwise,continue to the next step.
Divide the instance I into p subinstances I1, . . . , Ip ofapproximately the same size.
Recursively call the algorithm on each subinstance Ij , 1 ≤ j ≤ p, toobtain p partial solutions.
Combine the results of the p partial solutions to obtain the solutionto the original instance I . Return the solution of instance I .
The Divide and Conquer Algorithm
If the size of the instance I is “small”, then solve the problemusing a straightforward method and return the answer. Otherwise,continue to the next step.
Divide the instance I into p subinstances I1, . . . , Ip ofapproximately the same size.
Recursively call the algorithm on each subinstance Ij , 1 ≤ j ≤ p, toobtain p partial solutions.
Combine the results of the p partial solutions to obtain the solutionto the original instance I . Return the solution of instance I .
The Divide and Conquer Algorithm
If the size of the instance I is “small”, then solve the problemusing a straightforward method and return the answer. Otherwise,continue to the next step.
Divide the instance I into p subinstances I1, . . . , Ip ofapproximately the same size.
Recursively call the algorithm on each subinstance Ij , 1 ≤ j ≤ p, toobtain p partial solutions.
Combine the results of the p partial solutions to obtain the solutionto the original instance I . Return the solution of instance I .
Selection: Finding the Median and the k-th Element
The median of a sequence of n numbers A[1..n] is the middleelement, or the dn/2e-th smallest element.
A straightforward method of finding the median, or the k-thelement, is to sort all elements and pick the k-th element. Thistakes Ω(n log n) times.
Selection: Finding the Median and the k-th Element
The median of a sequence of n numbers A[1..n] is the middleelement, or the dn/2e-th smallest element.
A straightforward method of finding the median, or the k-thelement, is to sort all elements and pick the k-th element. Thistakes Ω(n log n) times.
The Idea of Selection
1. The input array A[1..n] is broken into two sets A1 and A3.
2. In order for the Conquer step to be possible, A1 and A3 mustsatisfy the condition ∀x ∈ A1.∀y ∈ A3.x < y .
3. To guarantee the good performance of the algorithm, both A1and A3 should have sufficient number of elements.
The Idea of Selection
1. The input array A[1..n] is broken into two sets A1 and A3.
2. In order for the Conquer step to be possible, A1 and A3 mustsatisfy the condition ∀x ∈ A1.∀y ∈ A3.x < y .
3. To guarantee the good performance of the algorithm, both A1and A3 should have sufficient number of elements.
The Idea of Selection
1. The input array A[1..n] is broken into two sets A1 and A3.
2. In order for the Conquer step to be possible, A1 and A3 mustsatisfy the condition ∀x ∈ A1.∀y ∈ A3.x < y .
3. To guarantee the good performance of the algorithm, both A1and A3 should have sufficient number of elements.
Selection
Algorithm 6.4 SelectInput: An array A[1..n] of n elements and an integer k , 1 ≤ k ≤ n.Output: The k-th smallest element in A.
1. select(A, 1, n, k)
Selection
Procedure select(A, low , high, k)
1. p ← high − low + 12. if p < 44 then sort A and return (A[k])3. Let q = bp/5c. Divide A into q groups of 5 elements each. If
5 does not divide p, then discard the remaining elements.4. Sort each of the q groups individually and extract its median.
Let the set of the medians be M.5. mm← select(M, 1, q, dq/2e)6. Partition A[low ..high] into three arrays:
A1 = {a | a < mm}; A2 = {a | a = mm}A3 = {a | a > mm}
7. case|A1| ≥ k : return select(A1, 1, |A1|, k)|A1| < k and |A1|+ |A2| ≥ k : return mm|A1|+ |A2| < k : return select(A3, 1, |A3|, k − |A1| − |A2|)
8. end case
Selection
Steps 1-5 make sure that A1 and A3 are not too small.
Analysis of Select
Let A1 be defined by {a | a ≥ mm}. Clearly |A1|+ |A1| = n.Observe that
|A1| ≥ 3dbn/5c/2e ≥3
2bn/5c
Hence
|A1| ≤ n −3
2bn/5c ≤ n − 3
2(
n − 45
) = 0.7n + 1.2
Similarly |A3| ≤ 0.7n + 1.2.
Analysis of Select
Let A1 be defined by {a | a ≥ mm}. Clearly |A1|+ |A1| = n.Observe that
|A1| ≥ 3dbn/5c/2e ≥3
2bn/5c
Hence
|A1| ≤ n −3
2bn/5c ≤ n − 3
2(
n − 45
) = 0.7n + 1.2
Similarly |A3| ≤ 0.7n + 1.2.
Analysis of Select
Steps 1 and 2 cost Θ(1) time each.
Step 3 costs Θ(n) time.
Step 4 costs Θ(n) time.
Step 5 costs T (bn/5c) time.Step 6 costs Θ(n) time.
Step 7 costs at most T (0.7n + 1.2) time. Assume that0.7n + 1.2 ≤ b0.75nc, which is subsumed by0.7n + 1.2 ≤ 0.75n + 1, which is in turn subsumed by n ≥ 44.It follows that
T (n) ≤{
c if n < 44T (bn/5c) + T (b3n/4c) + cn if n ≥ 44
for some c . Hence T (n) = Θ(n).
Example
For the sake of illustration, let’s temporarily change the thresholdfrom 44 to, say 6. Suppose we want to find the median of thefollowing 25 numbers:
8, 33, 17, 51, 57, 49, 35, 11, 25, 37, 14, 3, 2, 13, 52, 12, 6, 29, 32,54, 5, 16, 22, 23, 7
Selection
What if each group consists of 7 elements, or 9 elements?
The Quick Sort Algorithm
Quick sort is a popular and efficient sorting algorithm. Oneadvantage of the algorithm is that it does not need additionalstorage.
The Quick Sort Algorithm
We say that an element A[j ] is in its proper position or correctposition if it is neither smaller than the elements in A[1..j − 1] norlarger than the elements in A[j + 1..n].
An Auxiliary Algorithm
Algorithm 6.5 SplitInput: An array of elements A[low ..high].Output: A with its elements rearranged and w the new position ofthe splitting element A[low ].
1. i ← low2. x ← A[low ]3. for j ← low + 1 to high4. if A[j ] ≤ x then5. i ← i + 16. if i 6= j then interchange A[i ] and A[j ]7. end if8. end for9. interchange A[low ] and A[i ]10. w ← i11. return A and w
Time Complexity of Split
The number of comparisons is precisely n − 1.
Quicksort
Algorithm 6.6 QuickSortInput: An array A[1..n] of n elements.Output: The elements in A sorted in nondecreasing order.
1. QuickSort(A, 1, n)
Procedure QuickSort(A, low , high)
1. if low < high then2. Split(A[low ..high],w)3. QuickSort(A, low ,w − 1)4. QuickSort(A,w + 1, high)5. end if
Analysis of QuickSort
The worst case complexity:
f (n) = (n − 1) + (n − 2) + . . .+ 1 + 0 = Θ(n2)
If we always select the median as the pivot element, then roughly
C (n) ≤{
0 if n = 12C (n/2) + Θ(n) if n ≥ 2
whose solution is C (n) = Θ(n log n).
Analysis of QuickSort
The worst case complexity:
f (n) = (n − 1) + (n − 2) + . . .+ 1 + 0 = Θ(n2)
If we always select the median as the pivot element, then roughly
C (n) ≤{
0 if n = 12C (n/2) + Θ(n) if n ≥ 2
whose solution is C (n) = Θ(n log n).
Practical Performance of QuickSort
If the input is ordered randomly, QuickSort is fast
Otherwise we could choose the pivot randomly.
Practical Performance of QuickSort
If the input is ordered randomly, QuickSort is fast
Otherwise we could choose the pivot randomly.
Average Case Behavior of QuickSort
Assume that the input numbers are 1, 2, . . . , n and that eachnumber is equally likely to be the first of the input.
Now
C (n) = (n − 1) + 1n
n∑w=1
(C (w − 1) + C (n − w))
= (n − 1) + 2n
n∑w=1
C (w − 1)
Average Case Behavior of QuickSort
Then
nC (n) = n(n − 1) + 2n∑
w=1
C (w − 1)
Replacing n by n − 1
(n − 1)C (n − 1) = (n − 1)(n − 2) + 2n−1∑w=1
C (w − 1)
Subtracting and rearranging
C (n)
n + 1=
C (n − 1)n
+2(n − 1)n(n + 1)
Average Case Behavior of QuickSort
Let
D(n) =C (n)
n + 1
Then
D(1) = 0
D(n) = D(n − 1) + 2(n − 1)n(n + 1)
Clearly
D(n) = 2n∑
j=1
j − 1j(j + 1)
Average Case Behavior of QuickSort
2n∑
j=1
j − 1j(j + 1)
= 2n∑
j=1
2
(j + 1)− 2
n∑j=1
1
j
= 4n+1∑j=2
1
j− 2
n∑j=1
1
j
= 2n∑
j=1
1
j− 4n
n + 1
= 2 ln n −Θ(1)
=2
log elog n −Θ(1)
≈ 1.44 log n
So C (n) = (n + 1)D(n) ≈ 1.44n log n.
Multiplication of Large Integers
Assume that u, v are two n digits integers, where for simplicity it isassumed that n is a power of 2. Then
u = w2n/2 + x
v = y2n/2 + z
The product of u and v can be computed as
uv = wy2n + (wz + xy)2n/2 + xz
The time complexity is
T (n) =
{d if n = 14T (n/2) + bn if n > 1
So T (n) = Θ(n2).
Multiplication of Large Integers
Assume that u, v are two n digits integers, where for simplicity it isassumed that n is a power of 2. Then
u = w2n/2 + x
v = y2n/2 + z
The product of u and v can be computed as
uv = wy2n + (wz + xy)2n/2 + xz
The time complexity is
T (n) =
{d if n = 14T (n/2) + bn if n > 1
So T (n) = Θ(n2).
Multiplication of Large Integers
Assume that u, v are two n digits integers, where for simplicity it isassumed that n is a power of 2. Then
u = w2n/2 + x
v = y2n/2 + z
The product of u and v can be computed as
uv = wy2n + (wz + xy)2n/2 + xz
The time complexity is
T (n) =
{d if n = 14T (n/2) + bn if n > 1
So T (n) = Θ(n2).
Multiplication of Large Integers
But observe that
wz + xy = (w + x)(y + z)− wy − xz
Namely
uv = wy2n + ((w + x)(y + z)− wy − xz)2n/2 + xz
The time complexity is
T (n) =
{d if n = 13T (n/2) + bn if n > 1
So T (n) = Θ(nlog 3) = Θ(n1.59).
Multiplication of Large Integers
But observe that
wz + xy = (w + x)(y + z)− wy − xz
Namely
uv = wy2n + ((w + x)(y + z)− wy − xz)2n/2 + xz
The time complexity is
T (n) =
{d if n = 13T (n/2) + bn if n > 1
So T (n) = Θ(nlog 3) = Θ(n1.59).
Multiplication of Large Integers
But observe that
wz + xy = (w + x)(y + z)− wy − xz
Namely
uv = wy2n + ((w + x)(y + z)− wy − xz)2n/2 + xz
The time complexity is
T (n) =
{d if n = 13T (n/2) + bn if n > 1
So T (n) = Θ(nlog 3) = Θ(n1.59).
Matrix Multiplication: The Traditional Algorithm
Matrix multiplication: a11 . . . a1n...an1 . . . ann
b11 . . . b1n...
bn1 . . . bnn
= c11 . . . c1n...
cn1 . . . cnn
where
cij =n∑
k=1
aikbkj
Clearly the complexity of multiplying two n × n matrices isT (n) = Θ(n3).
Matrix Multiplication: The Recursive Version
Assume that n = 2k . If n ≥ 2 then the matrix can be partitionedinto four matrices of dimensions n/2× n/2(
A11 A12A21 A22
)(B11 B12B21 B22
)=
(C11 C12C21 C22
)where(
C11 C12C21 C22
)=
(A11B11 + A12B21 A11B12 + A12B22A21B11 + A22B21 A21B12 + A22B22
)
Matrix Multiplication: The Recursive Version
Time Complexity:
T (n) =
{m if n = 18T (n/2) + 4(n/2)2a if n ≥ 2
Therefore T (n) = Θ(n3).
Matrix Multiplication: The Recursive Version
Time Complexity:
T (n) =
{m if n = 18T (n/2) + 4(n/2)2a if n ≥ 2
Therefore T (n) = Θ(n3).
Matrix Multiplication: Strassen’s Algorithm
Let
D1 = (A11 + A22)(B11 + B22)
D2 = (A21 + A22)B11
D3 = A11(B12 − B22)D4 = A22(B21 − B11)D5 = (A11 + A22)B22
D6 = (A21 − A11)(B11 + B12)D7 = (A12 − A22)(B21 + B22)
Then(C11 C12C21 C22
)=
(D1 + D4 − D5 + D7 D3 + D5D2 + D4 D1 + D3 − D2 + D6
)
Matrix Multiplication: Strassen’s Algorithm
Time Complexity:
T (n) =
{m if n = 17T (n/2) + 18(n/2)2a if n ≥ 2
Therefore T (n) = Θ(nlog 7) = Θ(n2.81).
Matrix Multiplication: Strassen’s Algorithm
Time Complexity:
T (n) =
{m if n = 17T (n/2) + 18(n/2)2a if n ≥ 2
Therefore T (n) = Θ(nlog 7) = Θ(n2.81).
The Closest Pair Problem
Given a set S of n points in the plane, find the pair p1 = (x1, y1)and p2 = (x2, y2) such that the Euclidean distance between p1 andp2 is minimal among all the pairs of points.
The Divide Step
1. Divide the points of S into two sets Sl , Sr .
2. This is achieved by sorting the points of S according to theirx-coordinates, and then divide S by the median x0.
The Conquer Step
1. Apply the algorithm to Sl , Sr .
2. Let δl , δr be the values returned by the recursive calls.
The Combine Step
1. Let δ be min{δl , δr}.
2. Find the distance δ′ between δl and δr .
3. Return min{δ, δ′}.
The Combine Step: How to Calculate δ′?
1. Sort the elements of S according to their y -coordinates.
2. Construct T whose elements are within δ-distance to thevertical line passing x0.
3. Find the closest pair in T using a gliding method.
ClosestPair
Algorithm 6.7 ClosestPairInput: A set of n points in the plane.Output: The minimum separation realized by two points in S .
1. Sort the points in S in nondecreasing order of theirx-coordinates.