IT 301: Algorithm Analysis Lecture-00
description
Transcript of IT 301: Algorithm Analysis Lecture-00
IT 301: Algorithm Analysis
Lecture-00
Jesmin Akhter Lecturer,IIT
Jahangirnagar University, Savar, DhakaBangladesh
04/21/23
Course Overview Algorithmic Basics:
Introduction(L00) Algorithm analysis(L01) Techniques for analysis of Algorithm(L02+L03).
Methodes for the design of efficient algorithms: 3 sets Divide and Conquer (L04) Greedy Algorithms (L05) Dynamic Programming (L06)
Back tracking,Branch and bound : (L07) Binary search and Traversal Techniques: (L8) Sorting and Selection :
Heapsort : L9 3 sets Quicksort : L10 Bucket Sort, Radix Sort: (L11+L12)
Spanning Trees, Shortest paths: (L13) Lower Bound Theory: (L14) NP-Completeness :
NP-Completeness: (L15) 2 sets NP-hard: (L16) NP-Complete problems: (L17)
Tutorial: (L18+L19+L20)
Course Policy
No late homework is acceptable. Cheating directly affects the reputation of
Department and the University and lowers the morale of other students.
Cheating in homework and exam will not be tolerated.
An automatic grade of 0 will be assigned for any student caught cheating.
Presenting another person’s work as your own constitutes cheating.
Primary Focus
Develop thinking ability.
problem solving skills.
(algorithm design and application) formal thinking.
(proof techniques & analysis)
What Will We Do? Examine interesting problems. Devise algorithms to solve them. Prove these algorithms correct. Analyze their resource usage (time, memory). Understand data structures & core algorithms. Learn problem-solving techniques. See applications in real-world situations.
Goals
Intro 6
Be very familiar with a collection of core algorithms. Be fluent in algorithm design paradigms: divide & conquer,
greedy algorithms, dynamic programming. Be able to analyze the correctness and runtime performance
of a given algorithm. Be familiar with the inherent complexity (lower bounds &
intractability) of some problems. Be intimately familiar with basic data structures. Be able to apply techniques in practical problems.
Textbook & References Introduction to Algorithms, 2nd Ed. by Cormen, Leiserson, Rivest,
& Stein (CLRS), McGraw Hill, 2002.
OTHER REFERENCES: Algorithmics: The Spirit of Computing, Harel How to Solve It, Polya. The Design and Analysis of Computer Algorithms,
Aho, Hopcroft and Ullman. Algorithms, Sedgewick. Algorithmics: Theory & Practice, Brassard & Bratley. Writing Efficient Programs & Programming Pearls, Bentley. The Science of Programming, by Gries.
The Craft of Programming, by Reynolds.
How to Succeed in this Course
Start early on all assignments. DON'T procrastinate.
Complete all reading before class. Participate in class. Think in class. Review after each class. Be formal and precise on all problem sets
and in-class exams.
IT 301: Algorithm Analysis
Lecture-01
Jesmin Akhter Lecturer,IIT
Jahangirnagar University, Savar, DhakaBangladesh
04/21/23
What is an Algorithm? (And how do we analyze one?)
What is an Algorithm
An algorithm is a finite sequence of unambiguous instructions for solving a well-specified computational problem.
All algorithm must satisfy the following criteria: Input. Output. Definiteness. Effectiveness. Finiteness.
A technique for solving a well-specified computational problem.
Example: sortinginput: A sequence of numbers.output: An ordered permutation of the input.
DefinitenessEffectivenessFiniteness
Input Output
What is an Algorithm
Definiteness: Each instruction is clear and unambiguous.
Effectiveness: Each instruction is executable; in other words, feasibility.
Finiteness: The algorithm terminates after a finite number of steps.
What is an Algorithm
Algorithm Analysis Determining performance characteristics.
(Predicting the resource requirements.) Time, memory, communication bandwidth etc.
• Computation time (running time) is of primary concern.
Why analyze algorithms? Choose the most efficient of several possible
algorithms for the same problem. Is the best possible running time for a problem
reasonably finite for practical purposes? Is the algorithm optimal (best in some sense)? –
Is something better possible?
Running Time
Run time expression should be machine-independent. Use a model of computation or “hypothetical”
computer. Our choice – RAM model (most commonly-used).
Model should be Simple. Applicable.
RAM Model Generic single-processor model. Supports simple constant-time instructions found in
real computers. Arithmetic (+, –, *, /, %, floor, ceiling). Data Movement (load, store, copy). Control (conditional and unconditional branch, subroutine
call). Data types: Integer and floating point. Run time (cost) is uniform (1 time unit) for all simple
instructions. Memory is unlimited. Flat memory model – no hierarchy. Access to a word of memory takes 1 time unit. Sequential execution – no concurrent operations.
Running Time – Definition
Call each simple instruction and access to a word of memory a “primitive operation” or “step.”
Running time of an algorithm for a given input is The number of steps executed by the algorithm
on that input.
Often referred to as the complexity of the algorithm.
Complexity and Input
Complexity of an algorithm generally depends on Size of input.
• Input size depends on the problem.– Examples: No. of items to be sorted.– No. of vertices and edges in a graph.
Other characteristics of the input data.• Are the items already sorted? • Are there cycles in the graph?
Worst, Average, and Best-case Complexity
Worst-case Complexity Maximum steps the algorithm takes for any possible
input. Most tractable measure.
Average-case Complexity Average of the running times of all possible inputs. Demands a definition of probability of each input, which is
usually difficult to provide and to analyze. Best-case Complexity
Minimum number of steps for any possible input. Not a useful measure. Why?
A Simple Example – Linear Search INPUT: a sequence of n numbers, key to search for.OUTPUT: true if key occurs in the sequence, false otherwise.
n
i 21
LinearSearch(A, key) cost times1 i 1 c1 12 while i ≤ n and A[i] != key c2 x3 do i++ c3 x-14 if i n c4 15 then return true c5 16 else return false c6 1x ranges between 1 and n+1.So, the running time ranges between c1+ c2+ c4 + c5 – best caseand c1+ c2(n+1)+ c3n + c4 + c6 – worst case
A Simple Example – Linear Search INPUT: a sequence of n numbers, key to search for.OUTPUT: true if key occurs in the sequence, false otherwise.
n
i 21
Assign a cost of 1 to all statement executions.Now, the running time ranges between 1+ 1+ 1 + 1 = 4 – best caseand 1+ (n+1)+ n + 1 + 1 = 2n+4 – worst case
LinearSearch(A, key) cost times1 i 1 1 12 while i ≤ n and A[i] != key 1 x3 do i++ 1 x-14 if i n 1 15 then return true 1 16 else return false 1 1
A Simple Example – Linear Search INPUT: a sequence of n numbers, key to search for.OUTPUT: true if key occurs in the sequence, false otherwise.
n
i 21
If we assume that we search for a random item in the list,on an average, Statements 2 and 3 will be executed n/2 times.Running times of other statements are independent of input.Hence, average-case complexity is 1+ n/2+ n/2 + 1 + 1 = n+3
LinearSearch(A, key) cost times1 i 1 1 12 while i ≤ n and A[i] != key 1 x3 do i++ 1 x-14 if i n 1 15 then return true 1 16 else return false 1 1
Order of growth
Principal interest is to determine how running time grows with input size – Order of growth. the running time for large inputs – Asymptotic complexity.
In determining the above, Lower-order terms and coefficient of the highest-order term are
insignificant. Ex: In insertion sort the worst case running time : an2+bn+c,
which term dominates the running time for very large n? Complexity of an algorithm is denoted by the highest-order
term in the expression for running time. Ex: Θ(n2) Constant complexity when running time is independent of the input
size – denoted Ο(1). Linear Search: Best case Θ(1), Worst and Average cases: Θ(n).
Order of growth One algorithm to be more efficient than other
if its worst case running time has a lower order of growth. Due to constant factors and lower order terms
an algorithm whose running time has a higher order of growth might take less time for small inputs than an algorithm whose running time has a lower order of growth.
But for large enough inputs, a Θ(n2) algorithm will run more quickly than a Θ(n3) algorithm .
Comparison of Algorithms
Complexity function can be used to compare the performance of algorithms.
Algorithm A is more efficient than Algorithm B for solving a problem, if the complexity function of A is of lower order than that of B.
Examples: Linear Search – (n) vs. Binary Search – (lg
n) Insertion Sort – (n2) vs. Quick Sort – (n lg n)
Comparisons of Algorithms Sorting For 106 numbers,
insertion sort: (n2): it took 5.56 hrs on a supercomputer using machine language
merge sort: (n lg n): It took 16.67 min on a PC using C/C++.
Efficiency of Algorithms
Computer speed is not infinite and memory is not free so we need efficient algorithms.
Algorithms efficiency can be determined by time and space it need.
For example, insertion sort takes time equal to C1n2 to sort n items while merge sort takes C2nlogn.
Which one would work faster depends on size of n.
Example of Efficiency
Computer A is 1000 times faster than B. A takes 2n2
time to sort n items using insertion sort
B takes 50logn time to sort n items using merge sort
)5.5seconds(020,00ns/secondinstructio10
nsinstructio)2.(1010
27
hours
mins) 20second(1163ns/secondinstructio10
nsinstructiolog1050.107
77
Correctness Proofs
Proving (beyond “any” doubt) that an algorithm is correct. Prove that the algorithm produces correct output
when it terminates. Partial Correctness. Prove that the algorithm will necessarily
terminate. Total Correctness. Techniques
Proof by Construction. Proof by Induction. Proof by Contradiction.
Loop Invariant
Logical expression with the following properties. Holds true before the first iteration of the loop –
Initialization. If it is true before an iteration of the loop, it remains true
before the next iteration – Maintenance. When the loop terminates, the invariant gives a useful
property that helps show that the algorithm is correct – Termination.
Similar to mathematical induction.
Correctness Proof of Linear Search
Use Loop Invariant for the while loop: At the start of each iteration of the while loop, the
search key is not in the subarray A[1..i-1].
LinearSearch(A, key)1 i 12 while i ≤ n and A[i] != key3 do i++4 if i n5 then return true6 else return false
If the algm. terminates, then it produces correct result.
Initialization.Maintenance.Termination.
Argue that it terminates.
Go through correctness proof of insertion sort in the text.(2.1)(18)
The problem of sorting
Input: sequence a1, a2, …, an of numbers.
Example:
Input: 8 2 4 9 3 6
Output: 2 3 4 6 8 9
Output: permutation a'1, a'2, …, a'n such
that a'1 a'2 … a'n .
Insertion sort
INSERTION-SORT (A, n) ⊳ A[1 . . n]for j ← 2 to n
do key ← A[ j]i ← j – 1while i > 0 and A[i] > key
do A[i+1] ← A[i]i ← i – 1
A[i+1] = key
“pseudocode”
i j
keysorted
A:1 n
Example of insertion sort
8 2 4 9 3 6
Example of insertion sort
8 2 4 9 3 6
Example of insertion sort
8 2 4 9 3 6
2 8 4 9 3 6
Example of insertion sort
8 2 4 9 3 6
2 8 4 9 3 6
Example of insertion sort
8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
Example of insertion sort
8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
Example of insertion sort
8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
2 4 8 9 3 6
Example of insertion sort
8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
2 4 8 9 3 6
Example of insertion sort
8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
2 4 8 9 3 6
2 3 4 8 9 6
Example of insertion sort
8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
2 4 8 9 3 6
2 3 4 8 9 6
Example of insertion sort
8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
2 4 8 9 3 6
2 3 4 8 9 6
2 3 4 6 8 9 done