Asymptotic Complexity (Big-O Notation) CS 1037 Fundamentals of Computer Science II.

22
Asymptotic Complexity (Big-O Notation) CS 1037 Fundamentals of Computer Science II

Transcript of Asymptotic Complexity (Big-O Notation) CS 1037 Fundamentals of Computer Science II.

Page 1: Asymptotic Complexity (Big-O Notation) CS 1037 Fundamentals of Computer Science II.

Asymptotic Complexity(Big-O Notation)

CS 1037 Fundamentals of Computer Science II

Page 2: Asymptotic Complexity (Big-O Notation) CS 1037 Fundamentals of Computer Science II.

2

What is a “Fast” Algorithm?

Best case

Average case

Worst case

Small problems

Large problems

Best case

Average case

Worst case

Small problems

Large problems

Takes the fewest seconds? Takes the fewest steps?

In practice, these matter. Choice depends on: • application requirements• computer architecture• expected input

P PP P

In theoretical comp. sci., algorithms ranked by:• fewest # of steps,• in the worst case,• on large problems

T

Page 3: Asymptotic Complexity (Big-O Notation) CS 1037 Fundamentals of Computer Science II.

3

Ranking Running-Time

• Suppose algorithm A takes an2+b steps on problem of size n in worst case

• Suppose algorithm B takes cnlgn+d steps• Which algorithm is faster in practice?– depends on a,b,c,d and typical input

• Which algorithm is faster in theory?– for large enough n coefficients a,b,c,d don’t matter– algorithm B therefore fundamentally faster than A

Page 4: Asymptotic Complexity (Big-O Notation) CS 1037 Fundamentals of Computer Science II.

4

Example: Algorithm A vs B

• Force cnlgn+d to have large c...

B faster than A for large enough

nA

B

Page 5: Asymptotic Complexity (Big-O Notation) CS 1037 Fundamentals of Computer Science II.

5

When is Algorithm A “as fast as” B?

• Let A(n) be # steps for A on problem size n• Let B(n) be # steps for algorithm B– e.g. define functions A(n)= 4n+2, B(n)= n2

A as fast as B if A(n) · kB(n) for all large enough n and some k > 0

4n+2 · n2 for all n ¸ 54n+2 · 2n2 for all n ¸ 320nlgn · n2 for all n ¸ 144

n

tim

e

# items 5

Page 6: Asymptotic Complexity (Big-O Notation) CS 1037 Fundamentals of Computer Science II.

6

Asymptotic Notation (“Big-O”)

• Let g(n) be some function of n• Define O(g(n)) to be set of all functions

as fast as g(n) in theoretical sense:

O(g(n)) = f(n) j there exist positive k, n0

such that 0 · f(n) · kg(n) for all n ¸n0

f

g

n

tim

e

# items

n2

f(n) 2 O(n2)n0

n 2 O(n) n 2 O(n2)n2 2 O(n)

e.g.

Page 7: Asymptotic Complexity (Big-O Notation) CS 1037 Fundamentals of Computer Science II.

7

Asymptotic Notation (“Big-O”)

• Intuition:

n+100 2 O(n) because n+100 · 50n for n ¸ 3 or because n+100 · 2n for n ¸ 100

n2 2 O(n) because for any k>0 eventually n2 > kn

for large n (i.e. “n2 slower than n”)

any constant c 2 O(n) because c · n for n ¸ c

f(n) 2 O(g(n))

“f(n) no slower than kg(n) for large n”

Page 8: Asymptotic Complexity (Big-O Notation) CS 1037 Fundamentals of Computer Science II.

8

Asymptotic Notation (“Big-O”)

• Claim: any function of the form an+b with a ¸0 is a member of the set O(n)

Proof: choose k = a+j b j, then an+b · (a+j b j)n 8 n ¸ 1

• Claim: any an2+bn+c 2 O(n2) if a ¸0 Proof: choose k = a+j b j+j c j, then an2+bn+c · (a+j b j+j c j)n2 8 n ¸ 1

Page 9: Asymptotic Complexity (Big-O Notation) CS 1037 Fundamentals of Computer Science II.

9

Asymptotic Notation (“Big-O”)

• Can say Insertion Sort runs in...– “quadratic time in the worst-case”, or– O(n2) time ... “oh of n squared time”

• Can say Binary Search runs in...– “logarithmic time in the worst-case”, or– O(lgn) time ... “oh of log n time”

• Can say Counting Sort runs in...– “time linear in n and k”, or– O(n+k) time ... “oh of n plus k time”

this k means “max range of items”

Page 10: Asymptotic Complexity (Big-O Notation) CS 1037 Fundamentals of Computer Science II.

10

Asymptotic Notation (“Big-O”)

• Claim: if f(n) 2 O(g(n)) and g(n) 2

O(h(n)) then f(n) 2 O(h(n)) Proof: We know for some n1,n2,k1,k2 > 0

0·f(n)·k1g(n) 8 n ¸ n1

0·g(n)·k2h(n) 8 n ¸ n2

and therefore 0·f(n)·k1k2h(n) 8 n ¸ maxfn1,n2g

Page 11: Asymptotic Complexity (Big-O Notation) CS 1037 Fundamentals of Computer Science II.

11

Big-O Notation Used Everywhere

DOCUMENTATION

COMP. SCI. COURSES

ALGORITHMS RESEARCH

Page 12: Asymptotic Complexity (Big-O Notation) CS 1037 Fundamentals of Computer Science II.

12

More Examples of Big-O Membership

5050

n+506n2+50

n2+10n+50n3¡10n

2lgn+50n+lgn+50

2nlgn+n+lgn

2n+500n3

cn+nc

22222222222

O(1)O(n)O(n)O(n2)O(n2)O(n3)O(lgn)O(n)O(nlgn)O(2n)O(cn) if c >

1

We say any constant c 2 O(1), i.e. O(1) = set of all constants

Such functions are “exponential in n” and grow extremely fast! Such algorithms take longer than age of universe to compute medium-sized problems

Page 13: Asymptotic Complexity (Big-O Notation) CS 1037 Fundamentals of Computer Science II.

13

History of Big-O

• Invented by mathematician Paul Bachmann in 1894 to characterize error bounds

• Don Knuth popularized use for running time bounds in 1970s

The Art of Computer Programming, Volume 1

Page 14: Asymptotic Complexity (Big-O Notation) CS 1037 Fundamentals of Computer Science II.

14

Knuth on Importance of Big-O“Students will be motivated to use O(n) notation for two important reasons. First, it significantly simplifies calculations because it allows us to be sloppy — but in a satisfactorily controlled way. Second, it appears in the power series

calculations of symbolic algebra systems like Maple and Mathematica, which today’s students will surely be using...”

—Don Knuth advocating Big-O in high school, Letter to The American Mathematical Society March 1998

http://micromath.wordpress.com/2008/04/14/donald-knuth-calculus-via-o-notation/

Page 15: Asymptotic Complexity (Big-O Notation) CS 1037 Fundamentals of Computer Science II.

15

END OF COURSE MATERIAL

hooray!!!

Page 16: Asymptotic Complexity (Big-O Notation) CS 1037 Fundamentals of Computer Science II.

16

variablesif-elseloops

arrayspointersfunctions

CS1036

Programming Skills Take Time!eff

ectiv

enes

s of

you

stuff in your brain

structsclasses

templates

sortinglists/queues

trees

CS1037

dictionarieshashingmore…

CS2210SE2205

Page 17: Asymptotic Complexity (Big-O Notation) CS 1037 Fundamentals of Computer Science II.

Big Universe of Ideas!

17

variablesloops if-else

arrayspointersfunctions

exceptions

structs

classes

templates

sortinglistsqueuestrees

big-O

recursion

polymorphism

inheritance

debuggingassertions

iterators

threads

networking

security

optimization

encapsulationsearching

hashing

clusteringheaps

balanced trees

caching

randomized algorithms

graphs

matching

NP-completeness

computability theory

finite automata

approximation algorithms

multisets

dynamic programming

compression

float arithmetic

symbolic computation

GPUs

distributed computing

compilers

operating systems

greedy algorithms

interpretersparsing

databases

dictionaries

stacks

user interfaces

JavaPython

virtual machinesC#

backtracking

profiling

reducibility

Cdata mining

assembly

file I/O

proof techniques

portabilitylinear algebra

rasterization

references

physics simulation

version control

concurrency

combinatorics

UML

Page 18: Asymptotic Complexity (Big-O Notation) CS 1037 Fundamentals of Computer Science II.

18

Language Popularity in 2010source: http://www.langpop.com/

Page 19: Asymptotic Complexity (Big-O Notation) CS 1037 Fundamentals of Computer Science II.

19

Goals of CS 1037

• Understand how to…– use data structures– implement data structures– characterize an algorithm’s performance– express computation in C++

• Develop intuition about…– best data structure / algorithm for situation– good code, poor code, and dangerous code

• Programming experience, confidence

Page 20: Asymptotic Complexity (Big-O Notation) CS 1037 Fundamentals of Computer Science II.

20

Pearls of Wisdom“Anybody who comes to you and says he has a perfect language is either naive or a salesman.” —Bjarne Stroustrup

at Waterloo Comp Sci Club Meeting

“A program that has not been tested does not work.” —Bjarne Stroustrup

Page 21: Asymptotic Complexity (Big-O Notation) CS 1037 Fundamentals of Computer Science II.

21

Pearls of Wisdom“The psychological profiling [of a programmer] is mostly the ability to shift levels of abstraction, from low level to high level. To see something in the small and to see something in the large.” —Don Knuth

“Premature optimization is the root of all evil.”

—Don Knuth

remember that one!!

Page 22: Asymptotic Complexity (Big-O Notation) CS 1037 Fundamentals of Computer Science II.

22

EXAM REVIEW TIME!

hooray?