Algorithm Complexity lalit bansal

7/29/2019 Algorithm Complexity lalit bansal

1/52

Chapter 2

Fundamentals of the Analysis

of Algorithm Efficiency

Copyright 2007 Pearson Addison-Wesley. All rights reserved.


2/52

Analysis of algorithms

Issues: correctness

time efficiency

space efficiency

optimality

Approaches:

theoretical analysis

empirical analysis


3/52

Theoretical analysis of time efficiency

Time efficiency is analyzed by determining the number ofrepetitions of the basic operationas a function ofinput size

Basic operation: the operation that contributes most

towards the running time of the algorithm

T(n) copC(n)

running time execution time

for basic operation

Number of times

basic operation is

executed

input size


4/52

Input size and basic operation examples

Problem I nput size measure Basic operation

Searching for key in a

list ofnitems

Number of lists items,

i.e. nKey comparison

Multiplication of two

matrices

Matrix dimensions or

total number of elements

Multiplication of two

numbers

Checking primality of

a given integer nnsize = number of digits

(in binary representation) Division

Typical graph problem #vertices and/or edgesVisiting a vertex or

traversing an edge


5/52

Empirical analysis of time efficiency

Select a specific (typical) sample of inputs

Use physical unit of time (e.g., milliseconds)

or

Count actual number of basic operations executions

Analyze the empirical data


6/52

Best-case, average-case, worst-case

For some algorithms efficiency depends on form of input:

Worst case: Cworst(n)maximum over inputs of size n

Best case: Cbest(n) minimum over inputs of size n

Average case: Cavg(n)average over inputs of size n

Number of times the basic operation will be executed on typical input

NOT the average of worst and best case

Expected number of basic operations considered as a random variable

under some assumption about the probability distribution of all

possible inputs


7/52

Example: Sequential search

Worst case

Best case

Average case


8/52

Types of formulas for basic operations count

Exact formulae.g., C(n) = n(n-1)/2

Formula indicating order of growth with specific

multiplicative constante.g., C(n) 0.5 n2

Formula indicating order of growth with unknown

multiplicative constant

e.g., C(n) cn2


9/52

The Big-Oh Notation:

given functions f(n) and g(n), we

say that f(n) is O(g(n) ) if and only

if there are positive constants c

and n0such that f(n) c g(n)for

n n0


10/52

Example

(n) =2n +6For functionsf(n)and g(n) (to the

right) there are

positive constants c

and n0

such that:

f(n) c g(n)

forn n0

conclusion:

2n+6 is O(n).


11/52

Another Example

On the other hand

n2 is not O(n) because

there is no c and n0 suchthat: n2 c n fornn0

(As the graph to the right

illustrates, no matter howlarge a c is chosen there is

an n big enough that

N2 > c n ) .


12/52

EXAMPLE

Consider 1/3 n2 5n

The dominating term is n

2

Therefore it should be of O(n2)

Given a positive constant c , a positive integern0 to be found such that

1/3 n2 - 5 n c n2


13/52

Dividing the inequality throughout by n2

1/3 - 5/n c

Therefore if n0 1 , choosing c 1/3 theinequality will never be violated

Hence the expression is indeed of O(n2)


14/52

Relatives of the Big-Oh

(f(n)): Big Omega--asymptoticlowerbound

(f(n)): Big Theta--asymptotic t ightbound


15/52

Categories of algorithm efficiency

Efficiency

Constant

Big O

O(1)

Logarithmic O(log n)

Linear O(n)

Linear logarithmic O(n log n)

Quadratic O(n2)

Polynomial O(nk)

Exponential O(cn)

Factorial O(n!)


16/52

-notation

(g(n)) = {f(n): +veconstantsc

1

, c2

, and n0

such

that 0 c1g(n) f(n)c2g(n),

nn0}

For functiong(n), (g(n)) is

given by:

g(n) is an asymptotically tight boundfor f(n).

Intuitively: Set of all functions that

have the samerate of growth asg(n).


17/52

O -notat ion

O(g(n)) = {f(n): +veconstantsc and n0 such that 0

f(n)cg(n), n n0}

For functiong(n), O(g(n)) is

given by:

g(n) is an asymptotic upper boundfor f(n).

Intuitively: Set of all functions

whose rate of growthis the same as

or lower than that ofg(n).

f(n) = (g(n)) f(n) = O(g(n)).(g(n)) O(g(n)).


18/52

Relat ions Between, O,


19/52

Practical Complexity

0

250

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

f(n) = n

f(n) = log(n)

f(n) = n log(n)

f(n) = n^2

f(n) = n^3

f(n) = 2^n


20/52

Problem size

10 102 103 104

log2n 3.3 6.6 10 13.3

n 10 102 103 104

nlog2n 0.33x10 0.7x103 104 1.3x105

n2 102 104 106 108

2n 1024 1.3x103 >10100 >10100

n! 3x106 >10100 >10100 >10100


21/52

Linear loop

1 i=12 Loop(I


22/52

Logarithm Loops

Multiply loops Divide loops

1 I = 1 1 I = n

2 Loop (I < n) 2 loop( i>= 1)

1 application code 1 application code

2 I=I/2

2 I = i*2

F(n) = [log n] F(n) = [log n]


23/52

Nested loop- linear logarithmic

1 I = 1 2 loop(I


24/52

Dependent Quadratic

1 I = 12 loop ( I < = n)

1 j = 1

2 loop( j < = i)

1 application code2 j = j + 1

3 I = I + 1

no of iterations in the body of the inner loop is

1 + 2 + 3 + 4 + + 10 = 55 I.e. n(n +1)/2

On an average = ( n+1/)2

thus total no of iterations = n (n+1)/2


25/52

Quadratic

1 i=12 Loop (I < = n)

1 j = 1

2 Loop( j < = n)

1 application code

2 j = j+1

3 I = i+1

F(n) = n2


26/52

Order of growth

Most important: Order of growth within a constant multipleas n

Example:

How much faster will algorithm run on computer that istwice as fast?

How much longer does it take to solve problem of double

input size?


27/52

Values of some important functions as n


28/52

Asymptotic order of growth

A way of comparing functions that ignores constant factors andsmall input sizes

O(g(n)): class of functions f(n) that grow no faster than g(n)

(g(n)): class of functions f(n) that grow at same rate as g(n)

(g(n)): class of functions f(n) that grow at least as fast as g(n)


29/52

Establishing order of growth using the definition

Definition: f(n) is in O(g(n)) if order of growth of f(n) orderof growth ofg(n) (within constant multiple),

i.e., there exist positive constant cand non-negative integer

n0 such that

f(n) c g(n) for every nn0

Examples:

10nis O(n2)

5n+20 is O(n)


30/52

Some properties of asymptotic order of growth

f(n)

O(f(n))

f(n) O(g(n)) iffg(n) (f(n))

Iff(n)

O(g(n)) and g(n)

O(h(n)) , then f(n)

O(h(n))

Note similarity with ab

Iff1

(n) O(g1

(n)) and f2

(n) O(g2

(n)) , then

f1(n) + f2(n) O(max{g1(n), g2(n)})

E bli hi d f h i li i


31/52

Establishing order of growth using limits

limT(n)/g(n) =

0 order of growth ofT(n)< order of growth ofg(n)

c> 0 order of growth ofT(n)= order of growth ofg(n)

order of growth ofT(n)> order of growth ofg(n)

Examples:

10n vs. n2

n(n+1)/2 vs. n2

n


32/52

LHpitals rule and Stirlings formula

LHpitals rule: Ifl im

nf(n

) =l im

ng(n

) =

andthe derivatives f, gexist, then

Stirlings formula: n! (2n)1/2 (n/e)n

f(n)

g(n)l imn

=f (n)

g(n)l imn

Example: log nvs. n

Example: 2nvs. n!


33/52

Orders of growth of some important functions

All logarithmic functions loga

nbelong to the same class(log n) no matter what the logarithms base a> 1 is

All polynomials of the same degree kbelong to the same class:akn

k+ ak-1nk-1+ + a0(nk)

Exponential functions anhave different orders of growth fordifferent as

order log n 0) < order an < order n! < order nn


34/52

Basic asymptotic efficiency classes

1 constant

log n logarithmic

n linear

nlog n n-log-n

n2 quadratic

n3 cubic

2n exponential

n! factorial


35/52

Time efficiency of nonrecursive algorithms

General Plan for Analysis

Decide on parameter nindicating input size

Identify algorithms basic operation

Determine worst, average, and bestcases for input of size n

Set up a sum for the number of times the basic operation isexecuted

Simplify the sum using standard formulas and rules (seeAppendix A)


36/52

Useful summation formulas and rules

liu1 = 1+1++1 = u- l+ 1In particular, liu1 = n- 1 + 1 = n(n)

1ini= 1+2++n= n(n+1)/2 n2/2 (n2)

1ini2 = 12+22++n2 = n(n+1)(2n+1)/6 n3/3 (n3)

0inai = 1+ a++ an = (an+1 - 1)/(a- 1) for any a 1In particular, 0in2i = 20 + 21 ++ 2n = 2n+1 - 1 (2n)

(aibi) = aibi cai = cai liuai = limai+ m+1iuai


37/52


38/52

Example 2: Element uniqueness problem


39/52

Example 3: Matrix multiplication


40/52

Example 4: Gaussian elimination

AlgorithmGaussianElimination(A[0..n-1,0..n])

//Implements Gaussian elimination of an n-by-(n+1) matrixA

fori 0 ton -2 do

forj i + 1 to n - 1 do

fork i ton do

A[j,k] A[j,k] -A[i,k] A[j,i] /A[i,i]

Find the efficiency class and a constant factor improvement.


41/52

Pl f A l i f R i Al ith


42/52

Plan for Analysis of Recursive Algorithms

Decide on a parameter indicating an inputs size.

Identify the algorithms basic operation.

Check whether the number of times the basic op. is executedmay vary on different inputs of the same size. (If it may, the

worst, average, and best cases must be investigatedseparately.)

Set up a recurrence relation with an appropriate initialcondition expressing the number of times the basic op. is

executed.

Solve the recurrence (or, at the very least, establish itssolutions order of growth) by backward substitutions oranother method.


43/52

Example 1: Recursive evaluation ofn!

Definition: n! = 1 2 (n-1) nfor n1 and 0! = 1

Recursive definition ofn!: F(n) = F(n-1) nfor n1 and

F(0) = 1

Size:

Basic operation:

Recurrence relation:

S l i th f M( )


44/52

Solving the recurrence for M(n)

M(n) = M(n-1) + 1, M(0) = 0


45/52

Example 2: The Tower of Hanoi Puzzle

1

2

3

Recurrence for number of moves:

S l i f b f


46/52

Solving recurrence for number of moves

M(n) = 2M(n-1) + 1, M(1) = 1


47/52


48/52

Example 3: Counting #bits


49/52

Fibonacci numbers

The Fibonacci numbers:

0, 1, 1, 2, 3, 5, 8, 13, 21,

The Fibonacci recurrence:

F(n) = F(n-1) + F(n-2)F(0) = 0

F(1) = 1

General 2nd order linear homogeneous recurrence with

constant coefficients:

aX(n) + bX(n-1) + cX(n-2) =0

S l i X( ) bX( 1) X( 2) 0


50/52

Solving aX(n) + bX(n-1) + cX(n-2) =0

Set up the characteristic equation (quadratic)

ar2 + br+ c=0

Solve to obtain roots r1 and r2

General solution to the recurrence

ifr1 and r2 are two distinct real roots: X(n) = r1n+r2

n

ifr1 =r2 = rare two equal real roots: X(n) = rn+nr

n

Particular solution can be found by using initial conditions

Application to the Fibonacci numbers


51/52

Application to the Fibonacci numbers

F(n) = F(n-1) + F(n-2) or F(n) - F(n-1) - F(n-2) = 0

Characteristic equation:

Roots of the characteristic equation:

General solution to the recurrence:

Particular solution for F(0) =0, F(1)=1:


52/52

Computing Fibonacci numbers

1. Definition-based recursive algorithm

2. Nonrecursive definition-based algorithm

3. Explicit formula algorithm

4. Logarithmic algorithm based on formula:

F(n-1) F(n)

F(n) F(n+1)

0 1

1 1=

n

for n1, assuming an efficient way of computing matrix powers.

Algorithm Complexity lalit bansal

Documents

Transcript of Algorithm Complexity lalit bansal