1.2.1_Complexity Oct 2013

8/11/2019 1.2.1_Complexity Oct 2013

1/43

1

1.2.1 Complexity2ndedition

When size matters

Knowledge Component 1: Theoretical Foundations

Ian F. C. SmithEPFL, Switzerland

8/11/2019 1.2.1_Complexity Oct 2013

2/43

8/11/2019 1.2.1_Complexity Oct 2013

3/43

3

What there is to learn

At the end of modules 1.2.1 and 1.2.2, there will be

answers to the following questions:Are there certain tasks that computers cannot do

and if so, can faster computers help in these cases?

What are the cases when computational requirements

are nearly independentof task size?

Can small changes have a big effect?

Can one classifytasks in order to know whether or

not a program will perform well for full-scale tasks?

Why is engineering experience so valuable?

8/11/2019 1.2.1_Complexity Oct 2013

4/43

4/42

Complexity

Execution Time

Big Oh Notation

Classification in Big Oh Notation

Outline

8/11/2019 1.2.1_Complexity Oct 2013

5/43

5/42

Complexity

Complexity is a central theme in computer science

and is an important topic in CAE.

One of the most practical aspects involves theclassificationof algorithms according to their

ability to cope with different levels of computational

complexity.

There are three types of complexity:

computational, descriptiveand cognitive.

8/11/2019 1.2.1_Complexity Oct 2013

6/43

6/42

Three Types of Complexity

Computational complexity: This type is used to

classify well-structured tasks where the goal is to

find an efficient solution.

Although efficiency can be measured in terms of

use of memory and hardware resources,

algorithms are traditionally classified according to

factors that influence execution time.

8/11/2019 1.2.1_Complexity Oct 2013

7/43

7/42

Three Types of Complexity (contd.)

Descriptive complexity: A classification of the

level of difficulty involved with providing

descriptive structures to systems that have

varying degrees of intricacy.

Cognitive complexity: A classification of thelevel of difficulty related to describing and

simulating human thought for various activities.

8/11/2019 1.2.1_Complexity Oct 2013

8/43

8/42

This course provides an introduction only to the

fundamental concepts of computationalcomplexity.

The two other types of complexities do not yet have

fixed classification schemas.

Nevertheless, they are both important areas in

computer science and their sub-domains aregradually attracting the interest of engineers.

8/11/2019 1.2.1_Complexity Oct 2013

9/43

9/42

Complexity

Execution Time

Big Oh Notation


Outline

8/11/2019 1.2.1_Complexity Oct 2013

10/43

10/42

Execution Time

Rather than absolute values, trendswith respect toparticular factors are of most interest, viz.growth ofexecution time in relation to the increase in the sizeof the task.

Important factors:

Formulation of the problem (task complexity) Algorithm complexity Size of input Presentation of the desired results Hardware capacity (memory, peripherals, etc.) Operating environment

8/11/2019 1.2.1_Complexity Oct 2013

11/43

11/42

Execution Time (contd.)

In this course, we will concentrate on the first two

aspects. It is assumed that the size of the input and

requirements related to the results are not critical

factors.

It will be shown that under certain conditions the first

two aspects can create extreme situations that are

independentof hardware capacity and operating

environments.

These aspects will remain important regardless of

advances in computing equipment.

8/11/2019 1.2.1_Complexity Oct 2013

12/43

12/42

Example 1: Part A

Estimation of trends in execution times of a programupon modifying key parameters.

Part A

Task: Write a program that finds the minimum valueof the following function by sampling the solutionspace at regular intervals.

8/11/2019 1.2.1_Complexity Oct 2013

13/43

13/42

Example 1: Part A (contd.)

0

1

2

3

-15 -5 5 15

F(x)= 1 +x2/4000 cos(x) forx [-10,10]

(Griewanks function)

8/11/2019 1.2.1_Complexity Oct 2013

14/43

14/42

What would be the best number of samples?

All other aspects being equal, if the number of

samples is doubled, the execution time is doubled. If

the number of samples is tripled, the execution time

is tripled.

Therefore, a sampling algorithm of this type is said tobe linearwith respect to the number of samples

required.


8/11/2019 1.2.1_Complexity Oct 2013

15/43

15/42

Note:This is notthe best algorithm for this task.

What is the best algorithm?


8/11/2019 1.2.1_Complexity Oct 2013

16/43

16/42

AnswerTake the differential of f(x), set it to zero and solve

forx. This algorithm requires no sampling and

execution time is the same regardless of the range of

values ofx.

Discussion

Most tasks can be solved in several ways usingalgorithms that have varying sensitivity to execution

time.


8/11/2019 1.2.1_Complexity Oct 2013

17/43

17/42

Example 1: Part B

Part B

Task: Write a program that finds the minimum valueof the following function by sampling along each axis

at regular intervals.

(General form of Griewanks function in Nvariables)

N

i

i

N

i

x

Nii ixxf i

114000,1

))/(cos(1)(2

8/11/2019 1.2.1_Complexity Oct 2013

18/43

18/42

For a given number of variables N, the number of

samples increases by 2N if the number of sampling

points on each axis is doubled and by 3Nif the number

of axis points is tripled.

For constant N, this algorithm is said to be

polynomialwith respect to the number of sampling

points on each axis.

Example 1: Part B (contd.)

8/11/2019 1.2.1_Complexity Oct 2013

19/43

19/42

If the number of sampling points along each axis is

constant while N varies, the algorithm is said to be

exponentialwith respect to the number of

variables.


8/11/2019 1.2.1_Complexity Oct 2013

20/43

20/42


For 10 sampling points along each axis:

N = 2 requires 100 samplesN = 3 requires 1000 samplesN = 4 requires 10000 samplesN = 10 requires 10 billion samples

N Number of samples

10 20 30 40

1 10 20 30 40

2 100 400 900 1600

3 1000 8000 27000 64000

4 10000 160000 810000 2560000

Polynomial

Exponential

8/11/2019 1.2.1_Complexity Oct 2013

21/43

21/42

In practice, exponentially complex algorithms arefeasible only for small tasks.

The algorithm may be clear and concise and the

program may only comprise a few lines, butexecution may take hours, days or even longer.


8/11/2019 1.2.1_Complexity Oct 2013

22/43

22/42

What are three types of complexity?

What is a good measure of computational

complexity?

What is more desirable from a computationalpoint of view; polynomial or exponential

complexity? Why?

Review Quiz I

8/11/2019 1.2.1_Complexity Oct 2013

23/43

23/42

What are three types of complexity?

Computational complexity

Descriptive complexity

Cognitive complexity

Only computational complexity has fixedclassification schemas. This course includes one

of them.

Answers to Review Quiz I

8/11/2019 1.2.1_Complexity Oct 2013

24/43

24/42

What is a good measure of computationalcomplexity?

Trends with respect to growth of execution time inrelation to the increase in a parameter.

What is more desirable from a computationalpoint of view; polynomial or exponentialcomplexity?

Polynomial complexity because execution timegrows less rapidly with increases in task size.

Answers to Review Quiz I

8/11/2019 1.2.1_Complexity Oct 2013

25/43

25/42

Complexity

Execution Time

Big Oh Notation


Outline

8/11/2019 1.2.1_Complexity Oct 2013

26/43

26/42

One of the popular notations expressing the

relationship between task size and amount of

computational resources required.

It provides a model of relative execution time andtask size. Therefore, Big Oh notation is veryuseful for classifying levels of computationalcomplexity.

"Big Oh" Notation

8/11/2019 1.2.1_Complexity Oct 2013

27/43

27/42

If n = task size (integer, n>0)f(n) = execution time (>0)g(n) = relative execution time

Then there exists a positive constant csuch thatf(n) c g(n)

Note the inequality!

"Big Oh" Notation (contd.)

8/11/2019 1.2.1_Complexity Oct 2013

28/43

28/42

We are interested in upper bound, or worst case,estimates of execution time.

For small values of n, say n

8/11/2019 1.2.1_Complexity Oct 2013

29/43

29/42

"Big Oh" Notation: O(g(n))

The constant, c, contains machine specific aspects ofexecution (hardware, compiler, etc.). These aspectsare assumed to be independent of task size n (nn0).

The function g(n)represents the trend in executiontimefor varying values of task size, n.

The notation O(g(n))signifies "Order of g(n)".

8/11/2019 1.2.1_Complexity Oct 2013

30/43

30/42

Examples Revisited

Example 1, Part A

The complexity is O(n)with respect to the number

of samples taken in the interval [10,10]

Example 1, Part B

With three variables (N=3), the complexity is O(n3)

with respect to the number of samples, n, taken in

the interval [10, 10]

8/11/2019 1.2.1_Complexity Oct 2013

31/43

31/42

Examples Revisited (contd.)

Example 1, Part BWith 10 samples taken in the interval [10, 10] on

each axis, the complexity is O(10N)with respect to

the number of variables (axes), N.

8/11/2019 1.2.1_Complexity Oct 2013

32/43

32/42

Complexity

Execution Time

Big Oh Notation


Outline

8/11/2019 1.2.1_Complexity Oct 2013

33/43

33/42

Classifications in Big Oh Notation

Logarithmic timeO(log(n)): f(n) c log(n)

Linear time

O(n): f(n) c n

Polynomial timeO(n2): f(n) c n2

Exponential timeO(2n): f(n) c 2n

8/11/2019 1.2.1_Complexity Oct 2013

34/43

34/42

Classifications in Big Oh Notation(contd.)

Factorial timeO(n!): f(n) c n!

Double exponential timeO(nn): f(n) c nn

Note: Big Oh notation is independent of thenumber of commands a machine executesin a second.

8/11/2019 1.2.1_Complexity Oct 2013

35/43

35/42

Simplification

What are the Big Oh notations for the following?

1. f(n) = 5n3+ 4n2+ 3n + 5

2. f(n) = 3n3

+ 2n

3. f(n) = 3n + log(n)

4. f(n) = 6n! + 2n4+ n

8/11/2019 1.2.1_Complexity Oct 2013

36/43

36/42

Simplification Answers

1. f(n) = 5n3+ 4n2+ 3n + 5 O(n3)

2. f(n) = 3n3+ 2n O(2n)

3. f(n) = 3n + log(n) O(n)

4. f(n) = 6n! + 2n4+ n O(n!)

Explanations follow.

8/11/2019 1.2.1_Complexity Oct 2013

37/43

37/42

Simplification Answers (contd.)

1. f(n) = 5n3+ 4n2+ 3n + 5

f(n) = 5n3+ 4n2+ 3n + 5

f(n) 5n3+ 4n3+ 3n3+ 5n3

f(n) 17n3

f(n) is O(n3)

8/11/2019 1.2.1_Complexity Oct 2013

38/43

38/42


2. f(n) = 3n3+ 2n

n 3n3 2n 3n3/2n 3n3+ 2n 13(2n)

1 3 2 1.5 5 26

2 24 4 6 28 52

3 81 8 ~10 89 1044 192 16 12 208 208

5 375 32 11.7 407 416

6 648 64 10 712 832

Comparing the two right-hand-side columns,

f(n) 13(2n). Therefore, f(n) is O(2n)

8/11/2019 1.2.1_Complexity Oct 2013

39/43

39/42


3. f(n) = 3n + log n

log n < n

f(n) 4n

Therefore, f(n) is O(n)

8/11/2019 1.2.1_Complexity Oct 2013

40/43

40/42


4. f(n) = 6n! + 2n4+ n

n 6n! 2n4+ n 2n4+n / 6n! 6n!+2n4+n 6(6n!)

1 6 3 0.5 9 36

2 12 34 2.8 46 723 36 165 4.6 201 216

4 144 516 3.6 660 864

5 720 1255 1.7 1975 4320

Comparing the two right-hand-side columns,

f(n) 36(n!). Therefore, f(n) is O(n!)

8/11/2019 1.2.1_Complexity Oct 2013

41/43

41/42

Summary

Complexity is an important topic in engineering

Three types of complexity: computational,

descriptive, cognitive

The Big Oh notation is useful for characterizing

computational complexity

High complexity (e.g. exponential or factorialcomplexity) => excessive execution times for large

problems

8/11/2019 1.2.1_Complexity Oct 2013

42/43

42/42

Remarks

When complexity is exponential, it is not possible tofind solutions in a reasonable amount of time forlarge values of n

Increases in computer power does not help

Presence of well-defined algorithms is not the onlycriterion for computability. For example, a well-defined algorithm that has factorial complexity

would not be computable for high values of tasksize.

8/11/2019 1.2.1_Complexity Oct 2013

43/43

Further reading

Computers and Intractability: A Guide to the Theoryof NP-Completeness, M. Garey and D. Johnson, W.H.Freeman and Company, New York, 1979

Algorithmics, The Spirit of Computing, 3rd ed., D.Harel, Haddison-Wesley, 2004

Engineering Informatics - Fundamentals of

Computer-Aided Engineering, B. Raphael and I.F.C.Smith, Wiley, 2013

1.2.1_Complexity Oct 2013

Documents

Transcript of 1.2.1_Complexity Oct 2013