The Fundamentals: Algorithms, Integers, & Matrices

Chapter 2

2.2 Growth of Functions

Big-O notation

A tool to analyze a program's efficiency.

An approximation of work an algorithm performs as a function of size of input

Work = O(input size)

Work

A measure of effort expended by computer in performing a computation

Commonly used Big-O values (orders of magnitude)

Big-O Name O(1) Constant time LowerO(log n) Logarithmic time O(n) Linear time O(n log n) O(n2) Quadratic time

O(n3) Cubic time O(nk) Polynomial time

O(2n) Exponential time HigherO(n!) Factorial time

Table of Common Running Times

N Linear N Log N Quadratic Cubic

10 10 10.0 100 100020 20 26.0 400 800030 30 44.3 900 2700040 40 64.0 1600 6400050 50 84.9 2500 12500060 60 106.7 3600 21600070 70 129.2 4900 34300080 80 152.2 6100 48800090 90 175.9 8100 729000

100 100 200.0 10000 1000000

Growth Rate of Some Functions

0100200300400500600700800900

1000

0 10 20 30 40 50 60 70 80 90 100

LinearnLognQuadraticCubic

Big-O

f(x) is O(g(x))if there are constants C and k such that

| f(x) | C | g(x) |

whenever x > k

Show f(x) = x2+2x+1 is O(x2)

Prove: |x2+2x+1| C | x2 |

• If we let k > 0, f(x) & g(x) are always positive, we can drop | |

Let C be the sum of f(x) coefficients x2+2x+1 1x2+2x2+1x2 = 4x2

» Note: x2 1x2 and 2x 2x2 and 11x2

» This inequality holds beginning at x=1: 44

Choosing k=1, C=4, x2+2x+1 is O(x2)

Show f(x) = 7x3 is O(x2)

Prove: |7x3| C | x2 |

Assuming k > 0, divide both sides by x2

7x C

» You choose a constant > any y on y=7x line» No such C exists, since x is arbitrarily large» (Choose any C, and there is an x > C)

7x3 is not O(x2)

Big-O Example algorithm

O(1) Assigning value to ith array element;

always the same number of steps

not necessarily short

a program which takes 320 steps,

reguardless of input values

O(log n) Binary search,

What power of 2 is greater than a input number?

A loop whose terminal value is being successively halved or doubled

Big-O Example algorithm

O(n) Printing all elements of an array; searching an unordered array; A loop which executes 1 to N times (where N is the input size, the number of data values being

processed)

O(n log n) Faster sorts; A loop whose index is being halved/doubled inside a loop executing from 1 to N

Big-O Example algorithm

O(n2) Slower sorts; A loop which executes from 1 to N times inside a loop executing from 1 to N

times

O(n3) Incrementing all the elements in a NxNxN array; A 1..N loop inside a 1..N loop inside a 1..N loop

O(nk) k levels of loops inside loops

Big-O Example algorithm

O(2n) List all the subsets of a set with n elements; practical only with small values of n; n=64 takes 5 years on a supercomputer

O(n!) List all the possible arrangements of a set with n elements

Big-O General Rules

a.) Multiplicative constants do not matter. O(3n) = O(n)

b.) Addition is done by taking the max. O(n2) + O(n) = O(max(n2, n)) = O(n2) O(n3) + O(log n) = O(max(n3, log n)) = O(n3)

c.) Multiplication remains multiplication. O(n) * O(log n) = O(n log n) O(n) * O(n2) = O(n3)

d.) Lower is always O(higher) for nbut closest is desired

f(n2) = O(n3)

How can a program with many control structures be categorized by only one?

A certain country taxes pet stores based only on the combined weight of all the pets.

Only two pets thrive in the country: parakeets and elephants

When reporting their taxes, pet shop owners generally only report the weight of the _____.

Example Complexities

4n3 + 20n + 30 = O(n3) n + 10000 = O(n) 4n4 + 20n + 30 = O(n4) 2n + n3 = O(2n) 200 = O(1)

Running Time of Statements Simple statements are: O(1)

initialization of variables; input/ouput

Loops are: O(g(n)f(n)) where g(n) is upper bound on number of loop iterations & f(n) is upper bound on the body of the loop If g(n) and f(n) are constant,

then this is constant time

Running Time of Statements (continued)

Conditional statements are

O(max(f(n),g(n))) where f(n) is upper bound on then part and g(n) is upper bound on else part

Blocks of statements with complexities:

f1(n), f2(n), ..,fk(n),

have complexity O(f1(n) + f2(n)+ ...+ fk(n))

Simple Analysis

cin >> n; // 1if (n > 20) // 1

cout << “Number is > 20” << endl; // 2else

cout << “Number is <= 20” << endl; // 2

T(n) = 2 + max (2,2) = 4 = O(1)

Example Analysis

cin >> n; // 1factorial = 1; // 1for (i = 2; i<=n; i++) // 1 initialization +1 test +

//(1 test + 1 inc) per iterationfactorial *= i; // 1

cout << factorial; // 1

T(n) = 5 + 3*n = O(n)

Another example

cin >> n; // 1if (n > 0) { // 1 factorial = 1; // 1 for (i = 2; i <=n; i++) // 1 initialization +1 test +

//(1 test + 1 inc) per iterationfactorial *= i; // 1

cout << factorial; // 1}else cout << “Can’t do factorial of” << n; // 2

T(n) = 2 + max(4+3*n, 2) = O(n)

Analysis of simple function callsint factorial(int n){ int fact=1,i; for (i = 2; i <=n; i++)

fact *= i; return fact;}

void main(){ int n; // 1

cin >> n; // 1 cout << factorial(n) << endl; //O(n)}

Main is O(n)

Analysis of Nested Loops

EXAMPLE 1:

for (int i = 0; i < n; i++) // n*O(n) = O(n2) for (int j=0; j<n;j++) // n*O(1) = O(n) x++; // O(1)

EXAMPLE 2:

for (int i = 0; i < n; i++) for (int j=i; j<n;j++) x++; // O(1)

T(n) = n+(n-1) + (n-2)+…+ 1 = n(n+1)/2 = O(n2)

Summing blocks of statements

for (int i = 0; i < n; i++) // O(n2) for (int j=0; j<n;j++) x++;

for (int i = 0; i < n; i++) // O(n) x++;

T(n) = O(n2) + O(n) = O(n2 + n) = O(n2)

Loops with different limits

for (int i = 0; i < n; i++) // n*O(m) = O(mn) for (int j=0; j<m; j++) // m*O(1) = O(m) x++; // O(1)

O(mn)

More complex loops

for (i = 1;i < n; i *=2) x++;

for (i=n; i > 0; i/=2) x++;

Repetitive halving or doubling results in logarithmic complexity. Both loops are O(log n)

The Integers and Division

2.4

Divides (|), Prime Number theory - the study of integers and their properties a | b (a divides b) - integer c where ac = b

3 | 12 since 12 = 3 * 4

Prime - an integer > 1 whose only factors are the number and 1 Composite - an integer > 1 that is not prime

Fundamental theorem of arithmetic Every positive int. can be written as a product of primes 100 = 2 * 2 * 5 * 5

If n is composite, it has a prime divisor n

Divisibility Tests

A number is divisible by 2 if its unit’s digit is divisible by 2. A number is divisible by 3 if the sum of the digits is divisible by 3. A number is divisible by 4 if the # made of the last two digits is div by 4. A number is divisible by 5 if its unit’s digit is divisible by 5. A number is divisible by 6 if its is divisible by 2 and 3. A number is divisible by 8 if the # made of the last 3 digits is div by 8. A number is divisible by 9 if the sum of the digits is divisible by 9.

The Division Algorithm

Given integer a and positive integer d. integers q,r with 0 rd, such that a = dq

+ r d is divisor, r remainder, q quotient, a dividend

quotient remainder divisor dividend

101 divided by 11 == quotient 9, remainder 2101 = 11* 9 + 2

-11 divided by 3 == quotient -4, remainder 1-11 = 3(-4) + 1

Greatest Common Divisor gcd(a,b) = largest integer d, such that d | a and d | b

gcd(24,36) = 12

Algorithm for finding gcd(a,b) Express a, b as powers of products of primes The products of lowest powers of all distinct primes is gcd

(intersection)

gcd(252,420) 252 = 22 32 7 420 = 22 3 5 7 gcd(252,420) = 22 3 7

= 84

Relatively Prime Two integers are relatively prime if their gcd is 1

17 and 22 are r.p. 6 and 22 are not r.p.

The integers a1, a2, …, an are pairwise relatively prime if

gcd(ai,aj) = 1, whenever 1 ijn 10, 17 and 21 are p.r.p. 10, 17 and 22 are not p.r.p. because gcd(10,22) = 2

Least Common Multiple lcm(a,b) = smallest positive integer that is divisible by both a and b

Finding lcm(a,b) Express a, b as powers of products of primes The products of highest powers of all distinct primes is lcm (union)

lcm(252,420) 252 = 22 32 7 420 = 22 3 5 7 gcd(252,420) = 22 32 5 7

= 1260

NOTE: ab = gcd(a,b) lcm(a,b)

Modular Arithmetic a mod m

the remainder when a is divided by m

17 mod 5 = 2

2001 mod 101 = 82 19 r 82

What about negative operands?-17 mod 5 The answer in C++ is –2 (-3 –2/5)

Text answer would be 3 (-4 + 3/5)

Mod 0 is not defined

Congruence a b (mod m) a Is Congruent to b modulo m

a mod m = b mod mOR

m divides a-b

17 5 (mod 6) is true6 divides (17 - 5) 17 mod 6 = 5 5 mod 6 = 5

24 14 (mod 6) is false6 does not divide (24 - 14) 24 mod 6 = 0 14 mod 6 = 2

Hashing

Applying a function to a key value mapping range of possible key values into a smaller range of addresses

h(k) = k mod m h(064212848) = 064212848 mod 111 = 14 h(037149212) = 037149212 mod 111 = 65

Cryptology

The study of secret messages

Caesar cipher f (p) = (p + 3) mod 26

“MEET YOU IN THE PARK” becomes“PHHW BRX LQ WKH SDUN”

Security Now Podcast

Leo LaPorte, twit.tv, leoville.com

Steve Gibson, grc.com

Decoder Ring (SN-31)

Basic Terminology• plaintext - the original message

• ciphertext - the coded message

• cipher - algorithm for transforming plaintext to ciphertext

• key - info used in cipher known only to sender/receiver

• encipher (encrypt) - converting plaintext to ciphertext

• decipher (decrypt) - recovering ciphertext from plaintext

• cryptography - study of encryption principles/methods

• cryptanalysis (codebreaking) - the study of principles/ methods of deciphering ciphertext without knowing key

• cryptology - the field of both cryptography and cryptanalysis

Cryptology

The study of secret messages

Caesar cipher f (p) = (p + 3) mod 26

“MEET YOU IN THE PARK” becomes“PHHW BRX LQ WKH SDUN”

English Letter Frequencies

Letter Pair Frequencies

For instance, given a section of English language, E tends to be very common, while X is very rare. Likewise, ST, NG, TH, and QU are common pairs of letters (termed bigrams or digraphs), while NZ and QJ are rare. The mnemonics phrase "ETAOIN SHRDLU" encodes the 12 most frequent letters in typical English language text.

EXCLUSIVE OR

The EXCLUSIVE OR is true when

exactly one of its operands is true.

p q p q

____________________

F F F

F T T

T F T

T T F

Reversible XOR

1100 plaintext

1010 encrypt

-----

0110 ciphertext

1010 decrypt

-----

1100 plaintext

Ch 7 / Foil 47

Random Numbers

Generating a sequence of random numbers is often useful In a game, it ensures that a player does not see

the same behavior each time In a simulation of a complex system,

random numbers can be used tohelp generate random events

– Car crash in a simulationof a highway system

– Likelihood of a gene in cell mutation– Weather simulation

Ch 7 / Foil 48

Uniform Random Numbers

Uniform random number sequence A sequence of random numbers where

– Each value in the sequence is drawn from the same range of numbers

– In each position of the sequence, any value in the number range is equally likely to occur

Ch 7 / Foil 49

Random Numbers

Examples Generate a uniform random

number sequence in the range1 to 6

– Use a fair six-sided dice– Each roll represents a new random number

Do two die produce uniform random numbers in the range 1 ... 12?

Generate a uniform random numbersequence in the range 1 to 2

– Use a fair coin Heads: 1, Tails: 2

Ch 7 / Foil 50

Random Numbers

We can write an algorithmfor generating what lookslike random numbers

Because it’s an algorithm,we know the rules for generating the next number The generated numbers are not really random

– They are properly called pseudorandom numbers

30 21 9 28 29 ...

Ch 7 / Foil 51

Stdlib Library

Provides in part an interface to functions that generate pseudorandom numbers rand() -- returns a uniform pseudorandom number (unsigned int)

from the inclusive interval 0 to RAND_MAX Consider rand.cpp

#include <iostream>#include <string>using namespace std;#include <stdlib.h>int main() { for (int i = 1; i <= 5; ++i)

cout << rand() << endl; return 0;

}

Ch 7 / Foil 52

Different Sequences

To produce a different sequence, invoke

void srand(unsigned int); Consider seed.cpp

int main() { cout << "Enter a seed: "; unsigned int Seed; cin >> Seed; srand(Seed); for (int i = 1; i <= 5; ++i) cout << rand() << endl; return 0;

}

Ch 7 / Foil 53

Different Sequences

To get a different sequence each time Need a method of setting the seed to a random value

– The standard method is to use the computer's clock as the value of the seed

– The function invocation time() can be used Returns an integral value of type time_t Invocation time(0) returns a suitable value for

generating a random sequence

Ch 7 / Foil 54

Randseed.cpp

#include <iostream>#include <string>using namespace std;#include <stdlib.h>#include <time.h>

int main() { srand((unsigned int) time(0)); for (int i = 1; i <= 5; ++i)

cout << rand() << endl; return 0;

}

Symmetric Key

Symmetric-key algorithms are a class of algorithms for cryptography that use trivially related, often identical, cryptographic keys for both decryption and encryption.

Diffie-Hellman Key Exchange

We both agree on a public key a I raise ab for some large integer b You raise ac for integer c We exchange values You raise ab to the c power I raise ac to the b power We both have abc for our session key (RNG seed)

No one eavesdropping can perform log ab or log ac for large values of a, b, c

Security Now 34, 7:30mins-11:30

RSA Encryption C = Me mod n

C = encrypted message M = plaintext message (converted to #s via ASCII) n is product of two primes pq e is exponent relatively prime to (p-1)(q-1) (public key)

The formula is invertible if p, q are known So, huge prime values are chosen for p,q (200 digits) n cannot be factored in a reasonable amount of time (2 billion years of computer time) Receiver of message is given decryption key: p & q

RSA Demo

RSA Encryption The RSA algorithm, invented in 1977 by Ron Rivest, Adi Shamir, and Leonard

Adleman, is used frequently for public-key encryption and decryption. It works as follows: Take two large prime numbers, p and q, and find their

product n = pq; n is called the modulus. Choose a number, e, less than n and relatively prime to (p1)(q1), and find d, an inverse of e modulo (p1)(q1), which means that ed 1 mod (p1)(q1);

e and d are called the public and private exponents, respectively. The public key is the pair (n,e); the private key is d. The factors p and q must be kept secret, or destroyed. It is difficult (presumably) to obtain the private key d from the public key (n,e). If one could factor n into p and q (which can be difficult for large values of p and q) however, then one could obtain the private key d.

  RSA Demo

Matrices

2.6

Matrix

A rectangular array of numbers

m n matrix m rows n columns

If m=n, the matrix is square Two matrices are equal if

they have same number of rows the same number of columns the corresponding entries in every position are equal

A is an m n matrix

a a a

a a a

a a a

n

n

m m mn

11 12 1

21 22 2

1 2

..

..

.. .. .. ..

..

aij

The element of matrix A on the ith row the jth column

A = [a ij] shorthand for denoting matrix A

Sum of matrices

A + B = [aij + bij]

1 0 1

2 2 3

3 4 0

3 4 1

1 3 0

1 1 2

4 4 2

3 1 3

2 5 2

Matrix Multiplication A is a m k matrix, B is a k n matrix

The product of A and B the sum of products of corresponding elements from ith row of A and jth column of B

AB = [cij]

cij = ai1b1j + ai2b2j + ai3b3j + … + aikbkj a bi x x j

x

k

1

Multiplication Example

1 0 4

2 1 1

3 1 0

0 2 2

2 4

1 1

3 0

2 0 12 4 0 0

4 1 3 8 1 0

6 1 0 12 1 0

0 2 6 0 2 0

c11 c12

c41 c42

Is Matrix Multiplication Commutative?

Does AB = BA? A B

1 1

2 1

2 1

1 1

AB

1 1

2 1

2 1

1 1

2 1 1 1

4 1 2 1

3 2

5 3

BA

2 1

1 1

1 1

2 1

2 2 2 1

1 2 1 1

4 3

3 2

The Transpose of Matrix A

A is an m n matrix

AT is an n m matrix obtained by interchanging the rows and cols of A

AT = [bij], bij = aji

A AT

7 2 4

1 0 9

7 1

2 0

4 9

Symmetric Matrix

A square matrix A is symmetic if A=AT

aij = aji for all i,j

1 1 0

1 0 1

0 1 1

a1,2

a2,1

The Identity Matrix of Order n

An n n matrix main diagonal is all 1 every other element is 0

A In = ImA = A

I I2 3

1 0

0 1

1 0 0

0 1 0

0 0 1

Powers for square matrices

Given A is an n n matrix

A0 = In

Ar = A A A A

r times

Zero-One (Logical) Matrices

Elements are 0/1 representing False/True

The join of logical matrices A and B aij bij

The meet of logical matrices A and B aij bij

Join and Meet of A and B

A B

1 0 1

0 1 0

0 1 0

1 1 0

A B A B v ^

1 1 1

1 1 0

0 0 0

0 1 0

Boolean Product A is a m k logical matrix, B is a k n logical matrix

The boolean product of logical matrices A and B

A B = [cij]

cij = (ai1 b1j) (ai2 b2j) (ai3 b3j) … (aik bkj)

Boolean Product Example

A B

1 0

0 1

1 0

1 1 0

0 1 1

A B

1 1 0

0 1 1

1 1 0

Boolean Powers

Given A is an n n logical matrix

A[0] = In

A[r] = A A A A

r times

Given A, find A2, A3

A[2] = A A

A[3] = A[2] A

A

0 0 1

1 0 0

1 1 0 0 0 1

1 0 0

1 1 0

0 0 1

1 0 0

1 1 0

1 1 0

0 0 1

1 0 1

o

1 1 0

0 0 1

1 0 1

0 0 1

1 0 0

1 1 0

1 0 1

1 1 0

1 1 1

o