461191 Discrete Mathematics Lecture 3: Algorithms, The Integers, and Matrices

461191 Discrete Mathematics

Lecture 3: Algorithms, The Integers, and Matrices

San RatanasanyaCS, KMUTNB

Adapted from several sources: Michael P. Frank, University of Florida, University of Nevada, and University of Maryland at Baltimore

Today’s Topics

Review Administrivia Algorithms Algorithmic and Problem Complexity Introduction to Number Theory Applications of Number Theory Matrices

…from last week… Sets and Set operations

Set is an unordered collection of objects Operations on Set: union, intersection, difference, complement,

symmetric difference Venn Diagram and equivalence

Functions Domain, Codomain, Range One-to-one, onto, bijection, inverse, and composite functions

Sequences Represents ordered list of elements

a1,a,2a3,…,an or {an} It is convenient to describe Sequence using formula

an = 2an-1

Summations

Example: Set

Prove that A A = U is true

A = {x | x A}A = {x | x A}A A = {x | x A x A}A A = {x | x U}A A = U

Example: Function Let f(x) = x2/3. Find f(S) if S = {x | xZ |x| 3}

f(3) = 32/3 = 3, f(2) = 22/3 = 1, f(1) = 12/3 = 0, f(0) = 0 f(S) = {0, 1, 3}

Suppose that g is a function from A to B and f is a function form B to C. Show that if both f and g are one-to-one functions, then fg is also one-to-one.

We need to show that x A y A (x y f(g(x)) f(g(y))).Or we need to show that x A y A (x y g(x) g(y)) x B y B (x y f(x) f(y)).Given that g is one-to-one, therefore g(x) g(y) in B.Given that f is one-to-one, therefore f(x) f(y) in C.We can now conclude that f(g(x)) f(g(y)).

Example: Sequence

Find a simple formula for an if the first 10 terms of the sequence {an} are 1, 7, 25, 79, 241, 727, 2185, 6559, 19681, 59047

a1 = 1

an = 3n - 2

Example: Summations

?3

1

3

0

i j

i

242

)13(*3*4

4

4

1

3

1

3

1

3

1

3

0

i

i

i j

i

i

i

0

1

2123

1201

aa

aa

aaaa

aaaa

n

nn

nn

2

2

111

2

)1)((*2

212

n

nnnnnn

nkkn

k

n

k

n

k

?)12(1

n

k

k

?1

1

n

iii aa

Administrivia Homeworks due today

Homework 1.1 – 1.4 Homework 2

Programs used in this class cannot be downloaded from class webpage

they are TOO BIG… There are only links to download those

programs on class webpage Prolog, ISETLW Python, Scheme

Information on what to download is also on the webpage

Algorithms The foundation of computer programming. Most generally, an algorithm just means a definite

procedure for performing some sort of task. A computer program is simply a description of an

algorithm, in a language precise enough for a computer to understand, requiring only operations that the computer already knows how to do.

We say that a program implements (or “is an implementation of”) its algorithm.

Algorithms You Already Know Grade-school arithmetic algorithms:

How to add any two natural numbers written in decimal on paper, using carries.

Similar: Subtraction using borrowing. Multiplication & long division.

Your favorite cooking recipe. How to register for classes at KMUTNB.

Programming Languages Some common programming languages:

Newer: Java, C, C++, C#, Visual Basic, JavaScript, Perl, Tcl, Pascal, many others…

Older: Fortran, Cobol, Lisp, Basic Assembly languages, for low-level coding.

In this class we will use an informal, Pascal-like “pseudo-code” language.

You should know at least 1 real language!

Algorithm Example (English) Task: Given a sequence {ai}=a1,…,an, aiN,

say what its largest element is. One algorithm for doing this, in English:

Set the value of a temporary variable v (largest element seen so far) to a1’s value.

Look at the next element ai in the sequence. If ai>v, then re-assign v to the number ai. Repeat then previous 2 steps until there are no

more elements in the sequence, & return v.

Executing an Algorithm When you start up a piece of software, we

say the program or its algorithm are being run or executed by the computer.

Given a description of an algorithm, you can also execute it by hand, by working through all of its steps with pencil & paper.

Before ~1940, “computer” meant a person whose job was to execute algorithms!

Executing the Max algorithm

Let {ai}=7,12,3,15,8. Find its maximum… Set v = a1 = 7. Look at next element: a2 = 12. Is a2>v? Yes, so change v to 12. Look at next element: a2 = 3. Is 3>12? No, leave v alone…. Is 15>12? Yes, v=15…

Algorithm CharacteristicsSome important general features of algorithms: Input. Information or data that comes in. Output. Information or data that goes out. Definiteness. Algorithm is precisely defined. Correctness. Outputs correctly relate to inputs. Finiteness. Won’t take forever to describe or run. Effectiveness. Individual steps are all do-able. Generality. Works for many possible inputs. Efficiency. Takes little time & memory to run.

Our Pseudocode Language

procedurename(argument: type)

variable := expressioninformal statementbegin statements end{comment}if condition then

statement [else statement]

for variable := initial value to final value statement

while condition statement

procname(arguments) Not defined in book:return expression

What is Pseudocode Pseudocode is an intermediate step between

language description, normally English, of the steps of a procedure and a specification of this procedure using an actual programming language.

เป็�นการอธิ�บายโป็รแกรมด้�วยภาษาที่��ใช้�ก�นที่��วไป็ ด้�วยลั�กษณะที่��คลั�ายภาษาที่��ใช้�ในการเขี�ยนโป็รแกรม

HumanLanguages

ProgrammingLanguages

Pseudocode

What is Program Program = Algorithm + Data Structure Each step of a program may make changes to variables Trace = a sequence of changes in variables as each step of

program is executed

Trace is a simple model of program executions State = variable

procedure procname(arg: type) Declares that the following text defines a

procedure named procname that takes inputs (arguments) named arg which are data objects of the type type. Example:

procedure maximum(L: list of integers)[statements defining maximum…]

variable := expression An assignment statement evaluates the

expression expression, then reassigns the variable variable to the value that results. Example assignment statement:

v := 3x+7 (If x is 2, changes v to 13.) In pseudocode (but not real code), the

expression might be informally stated: x := the largest integer in the list L

Informal statement Sometimes we may write a statement as an

informal English imperative, if the meaning is still clear and precise: e.g.,“swap x and y”

Keep in mind that real programming languages never allow this.

When we ask for an algorithm to do so-and-so, writing “Do so-and-so” isn’t enough! Break down algorithm into detailed steps.

begin statements end Groups a sequence of

statements together:begin statement 1 statement 2 … statement n end

Allows the sequence to be used just like a single statement.

Might be used: After a procedure

declaration. In an if statement after

then or else. In the body of a for or

while loop.Curly braces {} are used insteadin many languages.

{comment} Not executed (does nothing). Natural-language text explaining some

aspect of the procedure to human readers. Also called a remark in some real

programming languages, e.g. BASIC. Example, might appear in a max program:

{Note that v is the largest integer seen so far.}

Conditional Constructionsif condition then statement

if condition thenbegin Block of statementsend

if condition then statement 1else statement 2

if condition 1 then statement 1else if condition 2 then statement 2else if condition 2 then statement 2………………………………………………………else if condition n then statement nelse statement n+1

if condition then begin statement 1end

else begin statement 2end

if condition then statement Evaluate the propositional expression

condition. If the resulting truth value is True, then

execute the statement statement; otherwise, just skip on ahead to the next

statement after the if statement. Variant: if cond then stmt1 else stmt2

Like before, but iff truth value is False, executes stmt2.

Loop Constructionsfor variable := initial value to final

value statement

for variable := initial value to final value begin

Block of statementsend

for all element with certain property

while condition statement

while conditionbegin Block of statements 1end

sum := 0for i := 1 to n sum := sum + i

sum := 0while n > 0 begin sum := sum + n n := n - 1end

while condition statement Evaluate the propositional (Boolean)

expression condition. If the resulting value is True, then execute

statement. Continue repeating the above two actions

over and over until finally the condition evaluates to False; then proceed to the next statement.

while condition statement Also equivalent to infinite nested ifs, like so:

if condition begin statement if condition begin statement …(continue infinite nested if’s) end end

for var := initial to final stmt Initial is an integer expression. Final is another integer expression. Semantics: Repeatedly execute stmt, first with

variable var := initial, then with var := initial+1, then with var := initial+2, etc., then finally with var := final.

Question: What happens if stmt changes the value of var, or the value that initial or final evaluates to?

for var := initial to final stmt For can be exactly defined in terms of

while, like so: begin var := initial while var final begin stmt var := var + 1 endend

procedure(argument) A procedure call statement invokes the

named procedure, giving it as its input the value of the argument expression.

Various real programming languages refer to procedures as functions (since the procedure call notation works similarly to function application f(x)), or as subroutines, subprograms, or methods.

Nested Block of statements Conditional Loop Procedure – a call to procedure within procedure

Procedure statistic (L:list of integers)begin max(L) min(L) average(L) ………..end

Max procedure in pseudocode

procedure max(a1, a2, …, an: integers)

v := a1 {largest element so far}

for i := 2 to n {go thru rest of elems} if ai > v then v := ai {found bigger?}

{at this point v’s value is the same as the largest integer in the list}

return v

Inventing an Algorithm Requires a lot of creativity and intuition

Like writing proofs. Unfortunately, we can’t give you an

algorithm for inventing algorithms. Just look at lots of examples… And practice (preferably, on a computer) And look at more examples… And practice some more… etc., etc.

Algorithm-Inventing Example Suppose we ask you to write an algorithm

to compute the predicate:IsPrime:N→{T,F}

Computes whether a given natural number is a prime number.

First, start with a correct predicate-logic definition of the desired function:n: IsPrime(n) ¬1<d<n: d|n

Means d divides nevenly (without remainder)

IsPrime example, cont. Notice that the negated exponential can be

rewritten as a universal:¬1<d<n: d|n 1<d<n: d | n

2≤ d ≤ n−1: d | n This universal can then be translated directly into

a corresponding for loop:for d := 2 to n−1 { Try all potential divisors >1 & <n } if d|n then return F { n has divisor d; not prime }return T { no divisors were found; n must be prime}

Means d does not divide n evenly (the remainder is ≠0)

Optimizing IsPrime The IsPrime algorithm can be further optimized:

for d := 2 to n1/2 if d|n then return Freturn T

This works because of this theorem: If n has any (integer) divisors, it must have one less than n1/2.Proof: Suppose n’s smallest divisor >1 is a, and let b :≡ n/a. Then n = ab, but if a > n1/2 then b > n1/2 (since a is n’s smallest divisor) and so n = ab > (n1/2)2 = n, an absurdity. Further optimizations are possible:

- E.g., only try divisors that are primes less than n1/2.

Note smaller range of search.

Another example task Problem of searching an ordered list.

Given a list L of n elements that are sorted into a definite order (e.g., numeric, alphabetical),

And given a particular element x, Determine whether x appears in the list, and if so, return its index (position) in the list.

Problem occurs often in many contexts. Let’s find an efficient algorithm!

Search alg. #1: Linear Search

procedure linear search(x: integer, a1, a2, …, an: distinct integers)i := 1 {start at beginning of list}while (i n x ai) {not done, not found}

i := i + 1 {go to the next position}if i n then location := i {it was found}else location := 0 {it wasn’t found}return location {index or 0 if not found}

Search alg. #2: Binary Search Basic idea: On each step, look at the middle

element of the remaining list to eliminate half of it, and quickly zero in on the desired element.

<x >x<x <x

Search alg. #2: Binary Search

procedure binary search(x:integer, a1, a2, …, an: distinct integers) i := 1 {left endpoint of search interval}j := n {right endpoint of search interval}while i<j begin {while interval has >1 item}

m := (i+j)/2 {midpoint}if x>am then i := m+1 else j := m

endif x = ai then location := i else location := 0return location

binary search for the letter binary search for the letter ‘j’‘j’

a c d f g h j l m o p r s u v x za c d f g h j l m o p r s u v x z

center elementcenter element

search intervalsearch interval

a c d f g h j l m a c d f g h j l m o p r s u v x zo p r s u v x z




a c d f ga c d f g h j l m h j l m o p r s u v x zo p r s u v x z



a c d f ga c d f g h j h j l m o p r s u v x zl m o p r s u v x z




a c d f g ha c d f g h j j l m o p r s u v x zl m o p r s u v x z



found !found !

Practice exercises Devise an algorithm that finds the sum of

all the integers in a list. [2 min] procedure sum(a1, a2, …, an: integers)

s := 0 {sum of elems so far}for i := 1 to n {go thru all elems}

s := s + ai {add current item}{at this point s is the sum of all items}return s

Sorting Algorithms Sorting is a common operation in many

applications. E.g. spreadsheets and databases

It is also widely used as a subroutine in other data-processing algorithms.

Two sorting algorithms shown in textbook: Bubble sort Insertion sort

However, these are notvery efficient, and you shouldnot use them on large data sets!

We’ll see some more efficient algorithms later in the course.

Smallest elements “float” up to the top of the list, like bubbles in a container of liquid.

Bubble Sort

3031

13233

234

3

301

3132

233

334

13031

232

33334

130

231

3323334

12

303

31323334

123

3031323334

Insertion Sort Algorithm English description of algorithm:

For each item in the input list, “Insert” it into the correct place in the sorted

output list generated so far. Like so: Use linear or binary search to find the location where the

new item should be inserted. Then, shift the items from that position onwards down by

one position. Put the new item in the hole remaining.

Pseudocodes

Procedure insertionsort(a1,…,an: real numbers with n 2)

for j := 2 to nbegin

i := 1 while aj > ai

i := i +1 m : = aj

for k := 0 to j – i – 1aj-k := aj-k-1

ai := am

end {a1, a2,…,an are sorted}

Procedure bubblesort(a1,…,an: real numbers with n 2)

for i := 1 to n-1 for j := 1 to n-1 if aj > aj+1 then interchange aj and aj+1

{a1,…,an is in increasing order}

Greedy Algorithms

Any algorithms selecting the ‘best’ choice at each step are called Greedy

Can solve optimization problemProcedure change(c1, c2,…,cr: values of denomination of coins, where c1 > c2 >… > cr; n: a positive integer)for i := 1 to r while n ci begin add a coin with value ci to the change n := n - ci

end

Analysis of Algorithms What is the goal of analysis of algorithms?

To compare algorithms mainly in terms of running time but also in terms of other factors (e.g., memory requirements, programmer's effort etc.)

What do we mean by running time analysis? Determine how running time increases as the

size of the problem increases.

Is Binary Search more efficient? Number of iterations:

For a list of n elements, Binary Search can execute at most log2 n times!!

Linear Search, on the other hand, can execute up to n times !!

Average Number of Iterations Length Linear Search Binary Search

10 5.5 2.9100 50.5 5.8

1,000 500.5 9.010,000 5000.5 12.0

Is Binary Search more efficient? Number of computations per iteration:

Binary search does more computations than Linear Search per iteration.

Overall: If the number of components is small (say, less

than 20), then Linear Search is faster. If the number of components is large, then

Binary Search is faster.

How do we analyze algorithms?

We need to define a number of objective measures.

(1) Compare execution times? Not good: times are specific to a particular computer !!

(2) Count the number of statements executed? Not good: number of statements vary with the

programming language as well as the style of the individual programmer.

Example (# of statements)

Algorithm 1 Algorithm 2 arr[0] = 0; for(i=0; i<N; i++)arr[1] = 0; arr[i] = 0;arr[2] = 0; ...arr[N-1] = 0;

How do we analyze algorithms?

(3) Express running time as a function of the input size n (i.e., f(n)).

To compare two algorithms with running times f(n) and g(n), we need a rough measure of how fast a function grows.

Such an analysis is independent of machine time, programming style, etc.

Computing running time

Associate a "cost" with each statement and find the "total cost“ by finding the total number of times each statement is executed.

Express running time in terms of the size of the problem.

Computing running time (contd.)

Algorithm 1 Algorithm 2 Cost Cost arr[0] = 0; c1 for(i=0; i<N; i++) c2 arr[1] = 0; c1 arr[i] = 0; c1 arr[2] = 0; c1 ... arr[N-1] = 0; c1 ----------- ------------- c1+c1+...+c1 = c1 x N (N+1) x c2 + N x c1 = (c2 + c1) x N + c2

Computing running time (cont.)

Cost sum = 0; c1

for(i=0; i<N; i++) c2 for(j=0; j<N; j++) c2 sum += arr[i][j]; c3 ------------c1 + c2 x (N+1) + c2 x N x (N+1) + c3 x N x N

Comparing Functions UsingRate of Growth Consider the example of buying elephants and

goldfish:Cost: cost_of_elephants + cost_of_goldfishCost ~ cost_of_elephants (approximation)

The low order terms in a function are relatively insignificant for large n

n4 + 100n2 + 10n + 50 ~ n4

i.e., n4 + 100n2 + 10n + 50 and n4 have the same rate of growth

Rate of Growth ≡Asymptotic Analysis

Using rate of growth as a measure to compare different functions implies comparing them asymptotically.

If f(x) is faster growing than g(x), then f(x) always eventually becomes larger than g(x) in the limit (for large enough values of x).

Example Suppose you are designing a web site to process

user data (e.g., financial records). Suppose program A takes fA(n)=30n+8

microseconds to process any n records, while program B takes fB(n)=n2+1 microseconds to process the n records.

Which program would you choose, knowing you’ll want to support millions of users?

Visualizing Orders of Growth On a graph, as

you go to theright, a fastergrowingfunctioneventuallybecomeslarger...

fA(n)=30n+8

Increasing n

fB(n)=n2+1

Valu

e of

func

tion

Big-O Notation We say fA(n)=30n+8 is order n, or O(n).

It is, at most, roughly proportional to n. fB(n)=n2+1 is order n2, or O(n2). It is, at most,

roughly proportional to n2. In general, an O(n2) algorithm will be slower

than O(n) algorithm. Warning: an O(n2) function will grow faster

than an O(n) function.

More Examples …

We say that n4 + 100n2 + 10n + 50 is of the order of n4 or O(n4)

We say that 10n3 + 2n2 is O(n3) We say that n3 - n2 is O(n3) We say that 10 is O(1), We say that 1273 is O(1)

Big-O Visualization

Computing running time

Algorithm 1 Algorithm 2 Cost Cost arr[0] = 0; c1 for(i=0; i<N; i++) c2 arr[1] = 0; c1 arr[i] = 0; c1 arr[2] = 0; c1 ... arr[N-1] = 0; c1 ----------- ------------- c1+c1+...+c1 = c1 x N (N+1) x c2 + N x c1 = (c2 + c1) x N + c2

O(n)

Computing running time (cont.)

Cost

sum = 0; c1 for(i=0; i<N; i++) c2

for(j=0; j<N; j++) c2

sum += arr[i][j]; c3 ------------

c1 + c2 x (N+1) + c2 x N x (N+1) + c3 x N x N

O(n2)

Running time of various statements

while-loop for-loop

i = 0;while (i<N) {

X=X+Y; // O(1)result = mystery(X); // O(N), just an example...i++; // O(1)

} The body of the while loop: O(N) Loop is executed: N times

N x O(N) = O(N2)

Examples

if (i<j)for ( i=0; i<N; i++ )

X = X+i;else

X=0;

Max ( O(N), O(1) ) = O (N)

O(N)

O(1)

Examples (cont.’d)

The Growth of Functions

Example:Show that f(x) = x2 + 2x + 1 is O(x2).

For x > 1 we have:x2 + 2x + 1 x2 + 2x2 + x2

x2 + 2x + 1 4x2

Therefore, for C = 4 and k = 1:f(x) Cx2 whenever x > k.

f(x) is O(x2).


Question: If f(x) is O(x2), is it also O(x3)?

Yes. x3 grows faster than x2, so x3 grows also faster than f(x).

Therefore, we always have to find the smallest simple function g(x) for which f(x) is O(g(x)).


“Popular” functions g(n) aren log n, 1, 2n, n2, n!, n, n3, log n

Listed from slowest to fastest growth: 1 log n n n log n n2

n3

2n

n!

Complexity Examples

What does the following algorithm compute?

procedure who_knows(a1, a2, …, an: integers)m := 0for i := 1 to n-1

for j := i + 1 to nif |ai – aj| > m then m := |ai – aj|

{m is the maximum difference between any two numbers in the input sequence}Comparisons: n-1 + n-2 + n-3 + … + 1 = (n – 1)n/2 = 0.5n2 – 0.5n

Time complexity is O(n2).

Complexity Examples

Another algorithm solving the same problem:

procedure max_diff(a1, a2, …, an: integers)min := a1max := a1for i := 2 to n

if ai < min then min := ai

else if ai > max then max := ai

m := max - minComparisons: 2n - 2

Time complexity is O(n).

Asymptotic Notation

O notation: asymptotic “less than”:

f(n)=O(g(n)) implies: f(n) “≤” g(n)

notation: asymptotic “greater than”:

f(n)= (g(n)) implies: f(n) “≥” g(n)

notation: asymptotic “equality”:

f(n)= (g(n)) implies: f(n) “=” g(n)

Definition: O(g), at most order g

Let f,g are functions RR. We say that “f is at most order g”, if: c,k: f(x) cg(x), x>k

“Beyond some point k, function f is at most a constant c times g (i.e., proportional to g).”

“f is at most order g”, or “f is O(g)”, or “f=O(g)” all just mean that fO(g).

Sometimes the phrase “at most” is omitted.

Big-O Visualization

k

Points about the definition Note that f is O(g) as long as any values of c

and k exist that satisfy the definition. But: The particular c, k, values that make

the statement true are not unique: Any larger value of c and/or k will also work.

You are not required to find the smallest c and k values that work. (Indeed, in some cases, there may be no smallest values!)

However, you should prove that the values you choose do work.

“Big-O” Proof Examples Show that 30n+8 is O(n).

Show c,k: 30n+8 cn, n>k . Let c=31, k=8. Assume n>k=8. Then

cn = 31n = 30n + n > 30n+8, so 30n+8 < cn. Show that n2+1 is O(n2).

Show c,k: n2+1 cn2, n>k: . Let c=2, k=1. Assume n>1. Then cn2 = 2n2 = n2+n2 > n2+1, or n2+1< cn2.

Note 30n+8 isn’tless than nanywhere (n>0).

It isn’t evenless than 31neverywhere.

But it is less than31n everywhere tothe right of n=8.

n>k=8

Big-O example, graphically

Increasing n

Valu

e of

func

tion

n

30n+8cn =31n

30n+8O(n)

Common orders of magnitude

Order-of-Growth in Expressions “O(f)” can be used as a term in an arithmetic

expression .E.g.: we can write “x2+x+1” as “x2+O(x)” meaning

“x2 plus some function that is O(x)”. Formally, you can think of any such expression as

denoting a set of functions: “x2+O(x)” : {g | fO(x): g(x)= x2+f(x)}

Useful Facts about Big O Constants ... c>0, O(cf)=O(f+c)=O(fc)=O(f) Sums:

- If gO(f) and hO(f), then g+hO(f). - If gO(f1) and hO(f2), then

g+hO(f1+f2) =O(max(f1,f2))

(Very useful!)

More Big-O facts Products:

If gO(f1) and hO(f2), then ghO(f1f2)

Big O, as a relation, is transitive: fO(g) gO(h) fO(h)

More Big O facts f,g & constants a,bR, with b0,

af = O(f) (e.g. 3x2 = O(x2)) f+O(f) = O(f) (e.g. x2+x = O(x2)) |f|1-b = O(f) (e.g. x1 = O(x)) (logb |f|)a = O(f) (e.g. log x = O(x)) g=O(fg) (e.g. x = O(x log x)) fg O(g) (e.g. x log x O(x)) a=O(f) (e.g. 3 = O(x))

Definition: (g), at least order gLet f,g be any function RR. We say that “f is at least order g”, written (g), if

c,k: f(x) cg(x), x>k “Beyond some point k, function f is at least a constant c

times g (i.e., proportional to g).” Often, one deals only with positive functions and can

ignore absolute value symbols. “f is at least order g”, or “f is (g)”, or “f= (g)” all

just mean that f (g).

Big- Visualization

Definition: (g), exactly order g If fO(g) and gO(f) then we say “g and f are

of the same order” or “f is (exactly) order g” and write f(g).

Another equivalent definition: c1c2,k: c1g(x)f(x)c2g(x), x>k

“Everywhere beyond some point k, f(x) lies in between two multiples of g(x).”

(g) O(g) (g) (i.e., fO(g) and f(g) )

Big- Visualization

Rules for Mostly like rules for O( ), except: f,g>0 & constants a,bR, with b>0,

af (f) Same as with O. f (fg) unless g=(1) Unlike O.|f| 1-b (f), and Unlike with O. (logb |f|)c (f). Unlike with O.

The functions in the latter two cases we say are strictly of lower order than (f).

example Determine whether: Quick solution:

)( 2?

1

nin

i

1

2

( 1) / 2

( ) / 2

( )

( )

n

i

i n n

n n

n n

n

Other Order-of-Growth Relations o(g) = {f | c k: f(x) < cg(x), x>k}

“The functions that are strictly lower order than g.” o(g) O(g) (g).

(g) = {f | c k: cg(x) < f(x), x>k }“The functions that are strictly higher order than g.” (g) (g) (g).

Subset relations between order-of-growth sets.

Relations Between the Relations

RR( f )O( f )

( f ) ( f )o( f )• f

Strict Ordering of Functions Temporarily let’s write fg to mean fo(g),

f~g to mean f(g) Note that

Let k>1. Then the following are true:1 log log n log n ~ logk n logk n n1/k n n log n nk kn n! nn …

.0)(

)(lim

xg

xfgf

x

Common orders of magnitude

Review: Orders of Growth

Definitions of order-of-growth sets, g:RR

O(g) {f | c,k: f(x) cg(x), x>k } o(g) {f | c k: f(x) < cg(x),x>k } (g) {f| | c,k : f(x) cg(x),x>k } (g) {f | c k: f(x) >cg(x), x>k} (g) {f | c1c2,k: c1g(x)f(x)|c2g(x), x>k}

Examples

Algorithmic Complexity The algorithmic complexity of a

computation is some measure of how difficult it is to perform the computation.

Measures some aspect of cost of computation (in a general sense of cost).

Time complexity Best-case time = minimum time needed to

execute the algorithm for inputs of size n Worst-case time = maximum time needed t

o execute the algorithm for inputs of size n Average-case time = average time needed

focus on best and worst time

Problem Complexity The complexity of a computational problem

or task is the complexity of the algorithm with the lowest order of growth of complexity for solving that problem or performing that task.

E.g. the problem of searching an ordered list has at most logarithmic time complexity. (Complexity is O(log n).)

Tractable vs. Intractable Problems A problem or algorithm with at most polynomial

time complexity is considered tractable (or feasible). P is the set of all tractable problems.

A problem or algorithm that has more than polynomial complexity is considered intractable (or infeasible).

Note n1,000,000 is technically tractable, but really impossible.

nlog log log n is technically intractable, but easy. Such cases are rare though.

Dealing with Intractable Problems Many times, a problem is intractable for a small

number of input cases that do not arise in practice very often.

Average running time is a better measure of problem complexity in this case.

Find approximate solutions instead of exact solutions.

Unsolvable problems It can be shown that there exist problems that no

algorithm exists for solving them. Turing discovered in the 1930’s that there are

problems unsolvable by any algorithm. Example: the halting problem (see page 176)

Given an arbitrary algorithm and its input, will that algorithm eventually halt, or will it continue forever in an “infinite loop?”

NP and NP-complete NP is the set of problems for which there exists a

tractable algorithm for checking solutions to see if they are correct.

NP-complete is a class of problems with the property that if any one of them can be solved by a polynomial worst-case algorithm, then all of them can be solved by polynomial worst-case algorithms.

Satisfiability problem: find an assignment of truth values that makes a compound proposition true.

P vs. NP We know PNP, but the most famous

unproven conjecture in computer science is that this inclusion is proper (i.e., that PNP rather than P=NP).

It is generally accepted that no NP-complete problem can be solved in polynomial time.

Whoever first proves it will be famous!

Questions Find the best big-O notation to describe the

complexity of following algorithms: A binary search of n elements A linear search to find the smallest number in a

list of n numbers An algorithm that lists all ways to put the

numbers 1,2,3,…,n in a row.

Questions (cont’d) An algorithm that prints all bit strings of length n An iterative algorithm to compute n! An algorithm that finds the average of n

numbers by adding them and dividing by n An algorithm that prints all subsets of size three

of the set {1,2,3,…,n} The best case analysis of a linear search of a list

of size n.

Questions (cont’d) The wors-case analysis of a linear search of a list

of size n The number of print statements in the following

while n>1 {print “hello”n=n/2

}

Questions (cont’d) The number of print statements in the following for (i=1, i≤n; i++)

for (j=1, j ≤n; j++) print “hello” The number of print statements in the following for (i=1, i≤n; i++)

for (j=1, j ≤i; j++) print “hello”

Introduction to Number Theory

Introduction to Number Theory

Number theory is about integers and their properties.

We will start with the basic principles of divisibility, greatest common divisors, least common multiples, and modular arithmetic

and look at some relevant algorithms.

Division

If a and b are integers with a 0, we say that a divides b if there is an integer c so that b = ac.

When a divides b we say that a is a factor of b and that b is a multiple of a.

The notation a | b means that a divides b.

We write a χ b when a does not divide b(see book for correct symbol).

Divisibility Theorems

For integers a, b, and c it is true that

if a | b and a | c, then a | (b + c) Example: 3 | 6 and 3 | 9, so 3 | 15.

if a | b, then a | bc for all integers c Example: 5 | 10, so 5 | 20, 5 | 30, 5 | 40, …

if a | b and b | c, then a | c Example: 4 | 8 and 8 | 24, so 4 | 24.

Primes

A positive integer p greater than 1 is called prime if the only positive factors of p are 1 and p.Note: 1 is not a prime

A positive integer that is greater than 1 and is not prime is called composite.

The fundamental theorem of arithmetic:Every positive integer can be written uniquely as the product of primes, where the prime factors are written in order of increasing size.

Primes

Examples:

3·53·5

48 =48 =

17 =17 =

100 =100 =

512 =512 =

515 =515 =

28 =28 =

15 =15 =

2·2·2·2·3 = 22·2·2·2·3 = 244·3·3

1717

2·2·5·5 = 22·2·5·5 = 222·5·522

2·2·2·2·2·2·2·2·2 = 22·2·2·2·2·2·2·2·2 = 299

5·1035·103

2·2·72·2·7

Primes

If n is a composite integer, then n has a prime divisor less than or equal .

This is easy to see: if n is a composite integer, it must have at least two prime divisors. Let the largest two be p1 and p2. Then p1p2 <= n.

p1 and p2 cannot both be greater than , because then p1p2 > n.

n

n

The Division Algorithm

Let a be an integer and d a positive integer.Then there are unique integers q and r, with 0 r < d, such that a = dq + r.

In the above equation, d is called the divisor, a is called the dividend, q is called the quotient, and r is called the remainder.


Example:

When we divide 17 by 5, we have

17 = 53 + 2.

17 is the dividend, 5 is the divisor, 3 is called the quotient, and 2 is called the remainder.


Another example:

What happens when we divide -11 by 3 ?

Note that the remainder cannot be negative.

-11 = 3(-4) + 1.

-11 is the dividend, 3 is the divisor, -4 is called the quotient, and 1 is called the remainder.

Greatest Common Divisors

Let a and b be integers, not both zero.The largest integer d such that d | a and d | b is called the greatest common divisor of a and b.The greatest common divisor of a and b is denoted by gcd(a, b).Example 1: What is gcd(48, 72) ?The positive common divisors of 48 and 72 are 1, 2, 3, 4, 6, 8, 12, 16, and 24, so gcd(48, 72) = 24. Example 2: What is gcd(19, 72) ?The only positive common divisor of 19 and 72 is1, so gcd(19, 72) = 1.

Greatest Common Divisors

Using prime factorizations:

a = p1a1 p2

a2 … pnan , b = p1

b1 p2b2 … pn

bn ,where p1 < p2 < … < pn and ai, bi N for 1 i n

gcd(a, b) = p1min(a1, b1 ) p2

min(a2, b2 ) … pnmin(an, bn )

Example:

a = 60 = a = 60 = 2222 3 311 5 511

b = 54 = b = 54 = 2211 3 333 5 500

gcd(a, b) = gcd(a, b) = 2211 3 311 5 50 0 = 6 = 6

Relatively Prime Integers

Definition:Two integers a and b are relatively prime if gcd(a, b) = 1.

Examples:Are 15 and 28 relatively prime?Yes, gcd(15, 28) = 1.Are 55 and 28 relatively prime?Yes, gcd(55, 28) = 1.Are 35 and 28 relatively prime?No, gcd(35, 28) = 7.

Relatively Prime Integers

Definition:

The integers a1, a2, …, an are pairwise relatively prime if gcd(ai, aj) = 1 whenever 1 i < j n.

Examples:Are 15, 17, and 27 pairwise relatively prime?No, because gcd(15, 27) = 3.Are 15, 17, and 28 pairwise relatively prime?Yes, because gcd(15, 17) = 1, gcd(15, 28) = 1 and gcd(17, 28) = 1.

Least Common Multiples

Definition:The least common multiple of the positive integers a and b is the smallest positive integer that is divisible by both a and b.We denote the least common multiple of a and b by lcm(a, b).Examples:

lcm(3, 7) =lcm(3, 7) = 2121

lcm(4, 6) =lcm(4, 6) = 1212

lcm(5, 10) =lcm(5, 10) = 1010

Least Common Multiples

Using prime factorizations:

a = p1a1 p2

a2 … pnan , b = p1

b1 p2b2 … pn

bn ,where p1 < p2 < … < pn and ai, bi N for 1 i n

lcm(a, b) = p1max(a1, b1 ) p2

max(a2, b2 ) … pnmax(an, bn )

Example:a = 60 = a = 60 = 2222 3 311 5 511

b = 54 = b = 54 = 2211 3 333 5 500

lcm(a, b) = lcm(a, b) = 2222 3 333 5 51 1 = 4 = 427275 = 5405 = 540

GCD and LCM

a = 60 = a = 60 = 2222 3 311 5 511

b = 54 = b = 54 = 2211 3 333 5 500

lcm(a, b) = lcm(a, b) = 2222 3 333 5 51 1 = 540 = 540

gcd(a, b) = gcd(a, b) = 2211 3 311 5 50 0 = 6 = 6

Theorem: aTheorem: ab =b = gcd(a,b)gcd(a,b)lcm(a,b)lcm(a,b)

Modular Arithmetic

Let a be an integer and m be a positive integer.We denote by a mod m the remainder when a is divided by m.

Examples:

9 mod 4 =9 mod 4 = 11

9 mod 3 =9 mod 3 = 00

9 mod 10 =9 mod 10 = 99

-13 mod 4 =-13 mod 4 = 33

Congruences

Let a and b be integers and m be a positive integer. We say that a is congruent to b modulo m if m divides a – b.

We use the notation a b (mod m) to indicate that a is congruent to b modulo m.

In other words:a b (mod m) if and only if a mod m = b mod m.

Congruences

Examples:Is it true that 46 68 (mod 11) ?Yes, because 11 | (46 – 68).Is it true that 46 68 (mod 22)?Yes, because 22 | (46 – 68).For which integers z is it true that z 12 (mod 10)?It is true for any z{…,-28, -18, -8, 2, 12, 22, 32, …}

Theorem: Let m be a positive integer. The integers a and b are congruent modulo m if and only if there is an integer k such that a = b + km.

Congruences

Theorem: Let m be a positive integer. If a b (mod m) and c d (mod m), then a + c b + d (mod m) and ac bd (mod m).Proof: We know that a b (mod m) and c d (mod m) implies that there are integers s and t with b = a + sm and d = c + tm. Therefore,b + d = (a + sm) + (c + tm) = (a + c) + m(s + t) andbd = (a + sm)(c + tm) = ac + m(at + cs + stm).Hence, a + c b + d (mod m) and ac bd (mod m).

Congruences

Theorem: Let m be a positive integer. a b (mod m) iff a mod m = b mod m.Proof: Let a = mq1 + r1, and b = mq2 + r2.Only if part: a mod m = b mod m r1 = r2, therefore a – b = m(q1 – q2), and a b (mod m).If part: a b (mod m) implies a – b = mq mq1 + r1 – (mq2 + r2) = mq r1 – r2 = m(q – q1 + q2).Since 0 r1, r2 m, 0 |r1 - r2| m. The only multiple in that range is 0. Therefore r1 = r2, and a mod m = b mod m.

The Euclidean Algorithm

The Euclidean Algorithm finds the greatest common divisor of two integers a and b. For example, if we want to find gcd(287, 91), we divide 287 by 91:287 = 913 + 14We know that for integers a, b and c,if a | b and a | c, then a | (b + c).Therefore, any divisor (including their gcd) of 287 and 91 must also be a divisor of 287 - 913 = 14.Consequently, gcd(287, 91) = gcd(14, 91).


In the next step, we divide 91 by 14:

91 = 146 + 7

This means that gcd(14, 91) = gcd(14, 7).

So we divide 14 by 7:

14 = 72 + 0

We find that 7 | 14, and thus gcd(14, 7) = 7.

Therefore, gcd(287, 91) = 7.


In pseudocode, the algorithm can be implemented as follows:

procedure gcd(a, b: positive integers)x := ay := bwhile y 0begin r := x mod y x := y y := rend {x is gcd(a, b)}

Representations of Integers

Let b be a positive integer greater than 1.Then if n is a positive integer, it can be expressed uniquely in the form:

n = akbk + ak-1bk-1 + … + a1b + a0,

where k is a nonnegative integer,a0, a1, …, ak are nonnegative integers less than b,and ak 0.

Example for b=10:859 = 8102 + 5101 + 9100


Example for b=2 (binary expansion):(10110)2 = 124 + 122 + 121 = (22)10

Example for b=16 (hexadecimal expansion):(we use letters A to F to indicate numbers 10 to 15)(3A0F)16 = 3163 + 10162 + 15160 = (14863)10


How can we construct the base b expansion of an integer n?First, divide n by b to obtain a quotient q0 and remainder a0, that is,n = bq0 + a0, where 0 a0 < b.The remainder a0 is the rightmost digit in the base b expansion of n.Next, divide q0 by b to obtain:q0 = bq1 + a1, where 0 a1 < b.a1 is the second digit from the right in the base b expansion of n. Continue this process until you obtain a quotient equal to zero.


Example: What is the base 8 expansion of (12345)10 ?

First, divide 12345 by 8:12345 = 81543 + 1

1543 = 8192 + 7192 = 824 + 024 = 83 + 03 = 80 + 3

The result is: (12345)10 = (30071)8.


procedure base_b_expansion(n, b: positive integers)q := nk := 0while q 0begin ak := q mod b q := q/b k := k + 1end {the base b expansion of n is (ak-1 … a1a0)b }

Addition of Integers

How do we (humans) add two integers?

Example: 7583 + 4932

5511552211

111111 carrycarry

Binary expansions:Binary expansions: (1011)(1011)22

+ + (1010)(1010)22

1100

carrycarry11

1100

11

11(( ))22


Let a = (an-1an-2…a1a0)2, b = (bn-1bn-2…b1b0)2.

How can we algorithmically add these two binary numbers?First, add their rightmost bits:a0 + b0 = c02 + s0,where s0 is the rightmost bit in the binary expansion of a + b, and c0 is the carry.Then, add the next pair of bits and the carry:a1 + b1 + c0 = c12 + s1,where s1 is the next bit in the binary expansion of a + b, and c1 is the carry.


Continue this process until you obtain cn-1.

The leading bit of the sum is sn = cn-1.

The result is:a + b = (snsn-1…s1s0)2


Example:Add a = (1110)2 and b = (1011)2.

a0 + b0 = 0 + 1 = 02 + 1, so that c0 = 0 and s0 = 1.

a1 + b1 + c0 = 1 + 1 + 0 = 12 + 0, so c1 = 1 and s1 = 0.

a2 + b2 + c1 = 1 + 0 + 1 = 12 + 0, so c2 = 1 and s2 = 0.

a3 + b3 + c2 = 1 + 1 + 1 = 12 + 1, so c3 = 1 and s3 = 1.

s4 = c3 = 1.

Therefore, s = a + b = (11001)2.


procedure add(a, b: positive integers)c := 0for j := 0 to n-1begin d := (aj + bj + c)/2 sj := aj + bj + c – 2d c := dendsn := c{the binary expansion of the sum is (snsn-1…s1s0)2}

Applications of Number Theory

GCD as linear combination

If a and b are positive integers, then there exist integers s and t such that gcd(a,b) = sa + tb

ExampleExpress gcd(252, 198) = 18 as a linear combination of 252 and 198

252 = 1*198 + 54198 = 3*54 + 3654 = 1*36 + 1836 = 2*18 Start from next-to-last division

18 = 54 – 1*3636 = 198 – 3*5418 = 54 – 198 + 3*54 = 4*54 – 1*19854 = 252 – 1*198

18 = 4*252 – 5*198

Hashing Function How to assign memory location that the

data can be retrieved quickly h(k) = k mod m, where k is key and m is the

number of available memory locations

Example m = 111, h(064212848) = ? h(064212848) = 14 So this data will be placed at memory

location = 14

Caesar’s Encryption Invented by Julius Caesar to make

a secret messageAlgorithm1. Convert each alphabet in a message to its corresponding

number p: A = 0, B = 1,…, Z = 25, and p 25 only2. Select key k as a positive integer, Caesar himself often use k = 33. To encrypt, applies this formula:

f(p) = (p+k) mod 264. Then convert each number back to alphabet and pack the

ciphertext with its key 5. To decrypt, do step 1 and 2 then applies this formula:

f-1(p) = (p-k) mod 266. If p+k or p-k is greater than or lesser than 25, shift it with 25

Example: Caesar’s Encryption Encrypt “DISCRETE MATH”

D I S C R E T E M A T H

3 8 18 2 17 4 19 4 12 0 19 7

f(p+k) mod 26

6 11 21 5 20 7 22 7 15 3 22 10

G L V F U H W H P D W K

GLVFUHWH PDWK

Linear Congruences

Congruence of a form ax b(mod m), where m is positive integer, a and b are integers, x is a variable is called Linear Congruence

To solve this congruence is to find all possible x that satisfies the formula

Inverse modulo More details in Section 3.7

Chinese Reminder Theorem

from San-Tsu’s puzzle, ancient Chinese mathematician

can be solved using Modular arithmetic More details in Section 3.7

RSA Public key crypto system Based on modular exponential To encrypt

C = Me mod n To decrypt

Cd M(mod pq) More details see RSA section and

Example 11 and 12 on page 241 - 244

Matrices

Matrices A matrix is a rectangular array of numbers. An mn (“m by n”) matrix has exactly m

horizontal rows, and n vertical columns. An nn matrix is called a square matrix,

whose order is n. The plural of matrix is matrices.

07

15

32

a 32 matrix

Example

Row: 3 rows Column: 2 columns Size: 3 x 2 matrix Square: NO

3

2

1

1

0

1

Example

Row: 2 rows Column: 2 columns Size: 2 x 2 matrix Square: YES

43

21

Example

Rows and columns? Size? Square? Equal?

02

11A

2

1

1

1

2

0

B

0

1

1

2

0

2C

02

10D

Matrix Equality Two matrices A and B are equal iff they

have the same number of rows, the same number of columns, and all corresponding elements are equal.

061

023

61

23

Row and Column Order The rows in a matrix are usually indexed 1

to m from top to bottom. The columns are usually indexed 1 to n from left to right. Elements are indexed by row, then column.

nmmm

n

n

ji

aaa

aaa

aaa

a

,2,1,

,22,21,2

,12,11,1

, ][

A

Let The ith row of A is the 1 x n matrix

The jth column of A is the n x 1 matrix

nn

2n

1n

n2

22

12

n1

21

11

a

.

.

.

a

a

...

...

...

a

.

.

.

a

a

a

.

.

.

a

a

A

ini2i1 a...aa

nj

2j

1j

a

.

.

.

a

aith row

jth column

The (i,j)th element or entry of A is the element aij, that is, the

number in the ith row and jth column of A.

A convenient shorthand notation for expressing the matrix A is

to write A = [aij], which indicates that A is the matrix with its

(i,j)th element equal to aij.

nn

2n

1n

n2

22

12

n1

21

11

a

.

.

.

a

a

...

...

...

a

.

.

.

a

a

a

.

.

.

a

a

A

Matrix Arithmetic

Addition/Sum Multiplication/Product

Matrix Sums The sum A+B of two matrices A, B (which

must have the same number of rows, and the same number of columns) is the matrix (also with the same shape) given by adding corresponding elements.

A+B = [ai,j+bi,j]

511

911

311

39

80

62

Example: Addition

Find A + B, where:

43

21A

12

34B

Example: Addition

Find A + B, where:

65

43

21

A

54

32

10

B

Example: Addition

Find A + B, where:

65

43

21

A

543

210B

Matrix Products For an mk matrix A and a kn matrix B, the product AB

is the mn matrix:

i.e., element (i,j) of AB is given by the vector dot product of the ith row of A and the jth column of B (considered as vectors).

In other words, if AB = [cij], then

cij = ai1b1j + ai2b2j + … + aikbkj. Note: Matrix multiplication is not commutative!

k

jiji bac1

,,, ][

CAB

Matrix Product Example An example matrix multiplication to

practice in class:

31123

1501

1301

0202

0110

302

110

Example: Multiplication

Find AB, where:

Find BA

43

21A

12

34B


Find AB, where:

65

43

21

A

543

210B


Find AB, where:

12A

543

210B


Suppose A is a 6 × 8 matrix, B is an 8 × 5 matrix, and C is a 5 × 9 matrix.

Find the number of rows, the number of columns, and the number of entries in A(BC).


Find A2 where,

43

21A

Identity Matrices The identity matrix of order n, In, is the

order-n matrix with 1’s along the upper-left to lower-right diagonal and 0’s everywhere else.

100

010

001

if 0

if 1

ji

jinI

Identity matrix

Multiplying a matrix by an

appropriate sized identity matrix

does not change this matrix. In

other words, when A is an m x n

matrix, we have

AIn = ImA = A

Example: Identity matrix

Find identity matrix of each of the following matrices:

43

21A

12

34B

332

212

321

C

543

210D

Matrix Inverses For some (but not all) square matrices A,

there exists a unique multiplicative inverse A-1 of A, a matrix such that A-1A = In.

If the inverse exists, it is unique, and A-1A = AA-1.

We won’t go into the algorithms for matrix inversion...

Matrix Multiplication Algorithm

procedure matmul(matrices A: mk, B: kn)for i := 1 to m

for j := 1 to n begincij := 0

for q := 1 to kcij := cij + aiqbqj

end {C=[cij] is the product of A and B}

What’s the of itstime complexity?

(m)·

(n)·(

(1)+

(k) ·

(1))

Answer:(mnk)

Powers of Matrices

If A is an nn square matrix and p0, then: Ap AAA···A (A0 In)

Example:

p times

23

34

12

23

01

12

01

12

01

12

01

12

01

123

If A=[aij] is an mn matrix, the transpose of A (often written At or AT) is the nm matrix given by At = B = [bij] = [aji] (1in,1jm)

Matrix Transposition

Flipacross

diagonal

23

11

02

210

312t

Example: Transpose

Find transpose of the following matrices:

43

21A

543

210D

65

43

21

B

Symmetric Matrices A square matrix A is symmetric iff A=At. I.e.,

i,jn: aij = aji . Which is symmetric?

211

120

103

213

101

312

11

11

11

Definition 7: Symmetric

A square matrix A is called symmetric if A

= At. Thus A = [aij] is symmetric if aij = aji

for all i and j with 1 ≤ i ≤ n and 1 ≤ j ≤ n.

Example: Symmetric

14

42

024

231

412

12

13

31

Zero-One Matrices All elements of a zero-one matrix are 0 or 1

Representing False & True respectively. Useful for representing other structures.

E.g., relations, directed graphs The meet of A, B (both mn zero-one matrices):

AB : [aijbij] = [aij bij] The join of A, B:

AB : [aijbij]

Boolean Products Let A=[aij] be an mk zero-one matrix,

& let B=[bij] be a kn zero-one matrix, The boolean product of A and B is like

normal matrix , but using instead + in the row-column “vector dot product.”

ji

k

ij bac 1

][CA⊙B

Arithmetic/Boolean Products

ji

k

ij bac 1

][CA⊙B

k

jiji bac1

,,, ][

CAB

Boolean Powers For a square zero-one matrix A, and any

k0, the kth Boolean power of A is simply the Boolean product of k copies of A.

A[k] A⊙A⊙… A⊙

k times

Homework 3 Chapter 3

Section 3.1: 1, 5, 22, 39, 42

Section 3.2: 1, 7, 9, 19, 21

Section 3.3: 1, 3, 9, 11, 25, 27

Section 3.4: 1, 9, 17, 31, 32

Section 3.5: 1, 21, 25

Section 3.6: 1, 5, 19, 23

Section 3.7: 7, 21

Section 3.8: 1, 3, 5, 31, 33

Supplementary: none

461191 Discrete Mathematics Lecture 3: Algorithms, The Integers, and Matrices

Documents

Transcript of 461191 Discrete Mathematics Lecture 3: Algorithms, The Integers, and Matrices