3101 3.0A Lecture 1 - 1 10/24/2015 COSC3101A: Design and Analysis of Algorithms Tianying Ji Lecture...

3101 3.0A Lecture 1 - 1 04/21/23

COSC3101A: Design and Analysis of Algorithms

Tianying Ji

Lecture 1

3101 3.0A Lecture 1 - 2 04/21/23

Part 1

• The course General information

• Introduction to algorithms

• Algorithm analysis basics

• Asymptotic notation

3101 3.0A Lecture 1 - 3 04/21/23

The Course

• Purpose: The theoretical study of design and analysis of algorithms Not a programming course Not a math course, either

• Textbook: Introduction to Algorithms, Cormen, Leiserson, Rivest, Stein Second edition An excellent reference you should own

3101 3.0A Lecture 1 - 4 04/21/23

The Course

• Instructor: Tianying Ji [email protected] Office: CSEB 2022 Office hours: 5:30pm-7:00pm, Tuesday TA: TBA TA Office hours and location TBA

3101 3.0A Lecture 1 - 5 04/21/23

The Course

• Grading policy (Probably, TBD): Assignments: 20% Mid-term Exam: 30% Final Exam: 50%

3101 3.0A Lecture 1 - 6 04/21/23

The Course

• Format One lecture/week About four Assignments

o Problem sets

o Maybe occasional programming assignments

Mid-term + final exam

3101 3.0A Lecture 1 - 7 04/21/23

The Course

• Course overview Mathematical background (asymptotic notation, recurrence relations,

bounding sums, induction, etc.) [Ch 1, 2.2, 3, 4] Proofs of correctness, including loop invariants [Ch 2.1] Recursive algorithms; divide and conquer technique [Ch 2.3, 4] Sorting algorithms and lower bounds [Ch 6,7,8] Selection [Ch 9] Greedy algorithms [Ch 16] Dynamic programming [Ch 15] Graph algorithms (searching, spanning trees, shortest path, etc.) [Ch

22,23,24,25,26] (If time permits) A brief introduction to NP-completeness. [Ch 34]

3101 3.0A Lecture 1 - 8 04/21/23

Part 1 Questions?

3101 3.0A Lecture 1 - 9 04/21/23

Part 2

• The course ☻

• Introduction to algorithms [Ch 1] What are Algorithms? Analysis vs. Design RAM

• Algorithm analysis basics


3101 3.0A Lecture 1 - 10 04/21/23

What are algorithms?

• Webster definition: a procedure for solving a mathematical problem (as of finding the greatest common divisor) in a finite number of steps that frequently involves repetition of an operation;

• broadly : a step-by-step procedure for solving a problem or accomplishing some end especially by a computer

3101 3.0A Lecture 1 - 11 04/21/23

Basics of Algorithms

• Well-defined computational procedure

• Takes input values

• Produces output values

• Results are always correct for any input event

• Always terminates

3101 3.0A Lecture 1 - 12 04/21/23

Musical analogy

Computing Musical

Software Music

Hardware Musical instruments

Algorithms Chord progressions

Source code Musical notation

Programming Composing

3101 3.0A Lecture 1 - 13 04/21/23

Basic goals for an algorithm

• Always correct

• Always terminates

• This class: performance Performance often draws the line between what is

possible and what is impossible.

3101 3.0A Lecture 1 - 14 04/21/23

Analysis vs. Design

• Analysis: predict the cost of an algorithm in terms of resources and performance

• Design: design algorithms which minimize the cost

3101 3.0A Lecture 1 - 15 04/21/23

Machine model: Generic Random Access Machine (RAM)

• Executes operations sequentially

• Set of primitive operations: Arithmetic. Logical, Comparisons, Function calls

• Simplifying assumption: all ops cost 1 unit Eliminates dependence on the speed of our computer,

otherwise impossible to verify and to compare

3101 3.0A Lecture 1 - 16 04/21/23

Part 2 Questions?

3101 3.0A Lecture 1 - 17 04/21/23

Part 3

• The course ☻

• Introduction to algorithms ☻

• Algorithm analysis basics [Ch 2.1, 2.2] Sorting Insertion sort


3101 3.0A Lecture 1 - 18 04/21/23

The problem of sorting

Input: sequence a1, a2, …, an of numbers.

Output: permutation a'1, a'2, …, a'n such

that a'1 a'2 … a'n .

Example:

Input: 8 2 4 9 3 6

Output: 2 3 4 6 8 9

3101 3.0A Lecture 1 - 19 04/21/23

Insertion sort

INSERTION-SORT (A, n) ⊳ A[1 . . n]for i ← 2 to n

do key ← A[i]j ← i – 1while j > 0 and A[j] > key

do A[j+1] ← A[j]j ← j – 1

A[j+1] ← key

“pseudocode”

j i

keysorted

A:1 n

3101 3.0A Lecture 1 - 20 04/21/23

An Example: Insertion Sort

InsertionSort(A, n)for i ← 2 to n

do key ← A[i] j ← i - 1while j > 0 and A[j] > key

do A[j+1] ← A[j] j ← j - 1

A[j+1] ← key

30 10 40 20

1 2 3 4

i = j = key = A[j] = A[j+1] =

3101 3.0A Lecture 1 - 21 04/21/23




do A[j+1] ← A[j] j ← j - 1

A[j+1] ← key

30 10 40 20

1 2 3 4

i = 2 j = 1 key = 10A[j] = 30 A[j+1] = 10

3101 3.0A Lecture 1 - 22 04/21/23




do A[j+1] ← A[j] j ← j - 1

A[j+1] ← key

30 30 40 20

1 2 3 4

i = 2 j = 1 key = 10A[j] = 30 A[j+1] = 30

3101 3.0A Lecture 1 - 23 04/21/23




do A[j+1] ← A[j] j ← j - 1

A[j+1] ← key

30 30 40 20

1 2 3 4

i = 2 j = 1 key = 10A[j] = 30 A[j+1] = 30

3101 3.0A Lecture 1 - 24 04/21/23




do A[j+1] ← A[j] j ← j - 1

A[j+1] ← key

30 30 40 20

1 2 3 4

i = 2 j = 0 key = 10A[j] = A[j+1] = 30

3101 3.0A Lecture 1 - 25 04/21/23




do A[j+1] ← A[j] j ← j - 1

A[j+1] ← key

10 30 40 20

1 2 3 4

i = 2 j = 0 key = 10A[j] = A[j+1] = 10

3101 3.0A Lecture 1 - 26 04/21/23




do A[j+1] ← A[j] j ← j - 1

A[j+1] ← key

10 30 40 20

1 2 3 4

i = 3 j = 0 key = 10A[j] = A[j+1] = 10

3101 3.0A Lecture 1 - 27 04/21/23




do A[j+1] ← A[j] j ← j - 1

A[j+1] ← key

10 30 40 20

1 2 3 4

i = 3 j = 0 key = 40A[j] = A[j+1] = 10

3101 3.0A Lecture 1 - 28 04/21/23




do A[j+1] ← A[j] j ← j - 1

A[j+1] ← key

10 30 40 20

1 2 3 4

i = 3 j = 0 key = 40A[j] = A[j+1] = 10

3101 3.0A Lecture 1 - 29 04/21/23




do A[j+1] ← A[j] j ← j - 1

A[j+1] ← key

10 30 40 20

1 2 3 4

i = 3 j = 2 key = 40A[j] = 30 A[j+1] = 40

3101 3.0A Lecture 1 - 30 04/21/23




do A[j+1] ← A[j] j ← j - 1

A[j+1] ← key

10 30 40 20

1 2 3 4

i = 3 j = 2 key = 40A[j] = 30 A[j+1] = 40

3101 3.0A Lecture 1 - 31 04/21/23




do A[j+1] ← A[j] j ← j - 1

A[j+1] ← key

10 30 40 20

1 2 3 4

i = 3 j = 2 key = 40A[j] = 30 A[j+1] = 40

3101 3.0A Lecture 1 - 32 04/21/23




do A[j+1] ← A[j] j ← j - 1

A[j+1] ← key

10 30 40 20

1 2 3 4

i = 4 j = 2 key = 40A[j] = 30 A[j+1] = 40

3101 3.0A Lecture 1 - 33 04/21/23




do A[j+1] ← A[j] j ← j - 1

A[j+1] ← key

10 30 40 20

1 2 3 4

i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40

3101 3.0A Lecture 1 - 34 04/21/23




do A[j+1] ← A[j] j ← j - 1

A[j+1] ← key

10 30 40 20

1 2 3 4

i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40

3101 3.0A Lecture 1 - 35 04/21/23




do A[j+1] ← A[j] j ← j - 1

A[j+1] ← key

10 30 40 20

1 2 3 4

i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 20

3101 3.0A Lecture 1 - 36 04/21/23




do A[j+1] ← A[j] j ← j - 1

A[j+1] ← key

10 30 40 20

1 2 3 4

i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 20

3101 3.0A Lecture 1 - 37 04/21/23




do A[j+1] ← A[j] j ← j - 1

A[j+1] ← key

10 30 40 40

1 2 3 4

i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 40

3101 3.0A Lecture 1 - 38 04/21/23




do A[j+1] ← A[j] j ← j - 1

A[j+1] ← key

10 30 40 40

1 2 3 4

i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 40

3101 3.0A Lecture 1 - 39 04/21/23




do A[j+1] ← A[j] j ← j - 1

A[j+1] ← key

10 30 40 40

1 2 3 4

i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 40

3101 3.0A Lecture 1 - 40 04/21/23




do A[j+1] ← A[j] j ← j - 1

A[j+1] ← key

10 30 40 40

1 2 3 4

i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40

3101 3.0A Lecture 1 - 41 04/21/23




do A[j+1] ← A[j] j ← j - 1

A[j+1] ← key

10 30 40 40

1 2 3 4

i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40

3101 3.0A Lecture 1 - 42 04/21/23




do A[j+1] ← A[j] j ← j - 1

A[j+1] ← key

10 30 30 40

1 2 3 4

i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 30

3101 3.0A Lecture 1 - 43 04/21/23




do A[j+1] ← A[j] j ← j - 1

A[j+1] ← key

10 30 30 40

1 2 3 4

i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 30

3101 3.0A Lecture 1 - 44 04/21/23




do A[j+1] ← A[j] j ← j - 1

A[j+1] ← key

10 30 30 40

1 2 3 4

i = 4 j = 1 key = 20A[j] = 10 A[j+1] = 30

3101 3.0A Lecture 1 - 45 04/21/23




do A[j+1] ← A[j] j ← j - 1

A[j+1] ← key

10 30 30 40

1 2 3 4

i = 4 j = 1 key = 20A[j] = 10 A[j+1] = 30

3101 3.0A Lecture 1 - 46 04/21/23




do A[j+1] ← A[j] j ← j - 1

A[j+1] ← key

10 20 30 40

1 2 3 4

i = 4 j = 1 key = 20A[j] = 10 A[j+1] = 20

3101 3.0A Lecture 1 - 47 04/21/23




do A[j+1] ← A[j] j ← j - 1

A[j+1] ← key

10 20 30 40

1 2 3 4

i = 4 j = 1 key = 20A[j] = 10 A[j+1] = 20

Done!

3101 3.0A Lecture 1 - 48 04/21/23

Example of insertion sort

8 2 4 9 3 6

2 8 4 9 3 6

2 4 8 9 3 6

2 4 8 9 3 6

2 3 4 8 9 6

2 3 4 6 8 9 done

3101 3.0A Lecture 1 - 49 04/21/23

Kinds of analyses

Worst-case: (usually)• T(n) = maximum time of algorithm

on any input of size n.Average-case: (sometimes)

• T(n) = expected time of algorithm over all inputs of size n.

• Need assumption of statistical distribution of inputs.

Best-case: (NEVER)• Cheat with a slow algorithm that

works fast on some input.

3101 3.0A Lecture 1 - 50 04/21/23

Insertion SortStatement Cost Times

InsertionSort(A, n)

for i ← 2 to n c1 n

do key ← A[i] c2 n-1

j ← i - 1 c3 n-1

while j > 0 and A[j] > key c4 T

do A[j+1] ← A[j] c5 T-(n-1)

j ← j - 1 c6 T-(n-1)

0

A[j+1] ← key c7 n-1

T = t2 + t3 + … + tn = where ti is number of while expression evaluations for the ith for loop iteration

n

i it2

3101 3.0A Lecture 1 - 51 04/21/23

Analyzing Insertion Sort

• T(n) = c1n + c2(n-1) + c3(n-1) + c4T + c5(T - (n-1)) + c6(T - (n-1)) + c7(n-1)

= c8T + c9n + c10

• What can T be? Best case -- inner loop body never executed

o ti = 1 T(n) is a linear function Worst case -- inner loop body executed for all

previous elementso ti = i T(n) is a quadratic function

Average caseo ???

3101 3.0A Lecture 1 - 52 04/21/23

Analysis

• Simplifications Ignore actual and abstract statement costs Order of growth is the interesting measure:

o Highest-order term is what counts Remember, we are doing asymptotic analysis As the input size grows larger it is the high order term that

dominates

3101 3.0A Lecture 1 - 53 04/21/23

Part 3 Questions?

3101 3.0A Lecture 1 - 54 04/21/23

Part 4

• The course ☻

• Introduction to algorithms ☻

• Algorithm analysis basics [Ch 2.1, 2.2] ☻

• Asymptotic notation[Ch3]

3101 3.0A Lecture 1 - 55 04/21/23

Asymptotic Complexity

• Running time of an algorithm for large input sizes.

• Expressed using only the highest-order term in the expression for the exact running time. Instead of exact running time, say (n2).

• Describe behavior of functions in the limit.

• Described using Asymptotic Notation.

3101 3.0A Lecture 1 - 56 04/21/23

Asymptotic Notation• , O, , o, • Defined in terms of functions whose domain is the set

of natural numbers. Ex: f(n) = (n2). Describes how f(n) grows in comparison to n2.

• Determine sets of functions, in practice used to compare two function sizes.

• The notations differ in the rate-of-growth relation they describe between the defining function and the defined set of functions.

3101 3.0A Lecture 1 - 57 04/21/23

-notation

(g(n)) = {f(n): +ve constants c1, c2, and n0 such that 0 c1g(n) f(n) c2g(n),

n n0 }

For function g(n), (g(n)) is given by:

g(n) is an asymptotically tight bound for f(n).

Intuitively: Set of all functions thathave the same rate of growth as g(n).

3101 3.0A Lecture 1 - 58 04/21/23

O -notation

O(g(n)) = {f(n): +ve constants c and n0 such that 0 f(n) cg(n), n n0 }

For function g(n), O(g(n)) is given by:

g(n) is an asymptotic upper bound for f(n).

Intuitively: Set of all functions whose rate of growth is the same as or lower than that of g(n).

f(n) = (g(n)) f(n) = O(g(n)).(g(n)) O(g(n)).

3101 3.0A Lecture 1 - 59 04/21/23

-notation

(g(n)) = {f(n): +ve constants c and n0 such that 0 cg(n) f(n), n n0 }

For function g(n), (g(n)) is given by:

g(n) is an asymptotic lower bound for f(n).

Intuitively: Set of all functions whose rate of growth is the same as or higher than that of g(n).

f(n) = (g(n)) f(n) = (g(n)).(g(n)) (g(n)).

3101 3.0A Lecture 1 - 60 04/21/23

Relations Between , O,

3101 3.0A Lecture 1 - 61 04/21/23

Upper Bound Notation

• We say InsertionSort’s run time is O(n2) Properly we should say run time is in O(n2) Read O as “Big-O” (you’ll also hear it as “order”)

• In general a function f(n) is O(g(n)) if there exist positive constants c

and n0 such that f(n) c g(n) for all n n0

• Formally O(g(n)) = { f(n): positive constants c and n0 such

that f(n) c g(n) n n0}

3101 3.0A Lecture 1 - 62 04/21/23

Insertion Sort Is O(n2)

• Proof Suppose runtime is an2 + bn + c

o If any of a, b, and c are less than 0 replace the constant with its absolute value

an2 + bn + c (a + b + c)n2 + (a + b + c)n + (a + b + c)

3(a + b + c)n2 for n 1

Let c’ = 3(a + b + c) and let n0 = 1

• Question Is InsertionSort O(n3)? Is InsertionSort O(n)?

3101 3.0A Lecture 1 - 63 04/21/23

Big O Fact

• A polynomial of degree k is O(nk)

• Proof: Suppose f(n) = bknk + bk-1nk-1 + … + b1n + b0

o Let ai = | bi |

f(n) aknk + ak-1nk-1 + … + a1n + a0

ki

kk

i

ik cnan

n

nan

3101 3.0A Lecture 1 - 64 04/21/23

Lower Bound Notation

• We say InsertionSort’s run time is (n)

• In general a function f(n) is (g(n)) if positive constants c and n0 such

that 0 cg(n) f(n) n n0

• Proof: Suppose run time is an + b

o Assume a and b are positive (what if b is negative?)

an an + b

3101 3.0A Lecture 1 - 65 04/21/23

Asymptotic Tight Bound

• A function f(n) is (g(n)) if positive constants c1, c2, and n0 such that

c1 g(n) f(n) c2 g(n) n n0

• Theorem f(n) is (g(n)) iff f(n) is both O(g(n)) and (g(n))

3101 3.0A Lecture 1 - 66 04/21/23

Practical Complexity

0

250

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

f(n) = n

f(n) = log(n)

f(n) = n log(n)

f(n) = n 2̂

f(n) = n 3̂

f(n) = 2 n̂

3101 3.0A Lecture 1 - 67 04/21/23


0

500

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

f(n) = n

f(n) = log(n)

f(n) = n log(n)

f(n) = n 2̂

f(n) = n 3̂

f(n) = 2 n̂

3101 3.0A Lecture 1 - 68 04/21/23


0

1000

1 3 5 7 9 11 13 15 17 19

f(n) = n

f(n) = log(n)

f(n) = n log(n)

f(n) = n 2̂

f(n) = n 3̂

f(n) = 2 n̂

3101 3.0A Lecture 1 - 69 04/21/23


0

1000

2000

3000

4000

5000

1 3 5 7 9 11 13 15 17 19

f(n) = n

f(n) = log(n)

f(n) = n log(n)

f(n) = n 2̂

f(n) = n 3̂

f(n) = 2 n̂

3101 3.0A Lecture 1 - 70 04/21/23

Other Asymptotic Notations

• A function f(n) is o(g(n)) if positive constants c and n0 such that

f(n) < c g(n) n n0

• A function f(n) is (g(n)) if positive constants c and n0 such that

c g(n) < f(n) n n0

• Intuitively, o() is like < O() is like

() is like > () is like

() is like =

3101 3.0A Lecture 1 - 71 04/21/23

Part 4 Questions?

3101 3.0A Lecture 1 - 72 04/21/23

Review: Induction

• Suppose S(k) is true for fixed constant k

o Often k = 0

S(n) S(n+1) for all n >= k

• Then S(n) is true for all n >= k

3101 3.0A Lecture 1 - 73 04/21/23

Proof By Induction

• Claim:S(n) is true for all n >= k

• Basis: Show formula is true when n = k

• Inductive hypothesis: Assume formula is true for an arbitrary n

• Step: Show that formula is then true for n+1

3101 3.0A Lecture 1 - 74 04/21/23

Induction Example: Gaussian Closed Form

• Prove 1 + 2 + 3 + … + n = n(n+1) / 2 Basis:

o If n = 0, then 0 = 0(0+1) / 2

Inductive hypothesis:o Assume 1 + 2 + 3 + … + n = n(n+1) / 2

Step (show true for n+1):1 + 2 + … + n + n+1 = (1 + 2 + …+ n) + (n+1)

= n(n+1)/2 + n+1 = [n(n+1) + 2(n+1)]/2

= (n+1)(n+2)/2 = (n+1)(n+1 + 1) / 2

3101 3.0A Lecture 1 - 75 04/21/23

The End

• Read your text Ch1, Ch2.1, Ch2.2, Ch3!

3101 3.0A Lecture 1 - 1 10/24/2015 COSC3101A: Design and Analysis of Algorithms Tianying Ji Lecture...

Documents

Transcript of 3101 3.0A Lecture 1 - 1 10/24/2015 COSC3101A: Design and Analysis of Algorithms Tianying Ji Lecture...