MATH2070 - Introduction · I Final Exam 70%. ... Univariate = Single variable x 2[a;b] ˆR. ......

MATH2070/2970 Optimisation

Introduction

Semester 2, 2012Lecturer: I.W. Guo

Lecture slides courtesy of J.R. Wishart

Motivation Univariate Multivariate Algebraic Gauss-Jordan Elimination

Course Information

I Lecture InformationI Optimisation: Weeks 1 – 7

I Contact InformationI Email: [email protected] Office: 807, Extension: 41274

I ConsultationI Wednesday 11am in Room 707A.

I AssessmentI Assignment 10%I Quiz 10%I Project (Fin Maths) 10%I Final Exam 70%

mailto:[email protected]


Course Readings

I Lecture Slides.I Split into relevant topicsI Available at the course website. Website

I Lecture NotesI Notes for both Optimisation & Financial Mathematics.I Available at Kopystop Website

Shop 3 / 55 Mountain St

Broadway NSW 2007

I Google Maps Link Map

http://www.maths.usyd.edu.au/u/UG/IM/MATH2070/

http://www.kopystop.com.au/

http://maps.google.com.au/maps?f=q&source=s_q&hl=en&geocode=&q=kopystop&sll=-33.883267,151.196295&sspn=0.001886,0.004128&ie=UTF8&hq=kopystop&hnear=&ll=-33.884164,151.195022&spn=0.003754,0.008256&z=18


Optimisation Outline

I Introduction and Motivation

I Linear Programming

I Non-linear Optimisation without constraints

I Non-linear Optimisation with constraints

I Dynamic Programming (At the end of the course)


Review of Introductory Material

Motivation

Optimising Differentiable Functions of One Variable

Optimising Differentiable Functions of Several Variables

System of Equations

Pivot Operations and variable simplification


Motivation



System of Equations



How are optimisation techniques used?

Motivation, In most basic format either,

I Minimise costs/time.

I Maximise output.

Examples:

I Timetabling (Cityrail, Sydney Buses)

I HR: Staff allocations.

I Manufacturing: Blending of raw materials

I Retail: Determine optimal prices.


Terminology

Definition (Parameters)

A set of values (fixed or variable) that are used to describe therelationship between quantities. More

Definition (Objective Function)

A function f(x) of a set of parameters x that one wishes tomaximise/minimise will be termed the ‘objective’ function. More

Definition (Constraints)

A set of conditions that the parameters need to be satisfied duringthe optimisation. More

http://en.wikipedia.org/wiki/Parameter

http://en.wikipedia.org/wiki/Mathematical_optimization#Optimization_problems

http://en.wikipedia.org/wiki/Constraint_(mathematics)


Water tank design example

Wish to minimise heat loss through surface area from an openrectangular water storage tank that has a fixed volume, V .

Dimensions: x, y and z.

Formulate Problem

Minimise,S = 2xy + 2yz + xz

such that,

V = xyz ≡ constant and x, y, z > 0.

Terminology

Parameters : x, y and z.Constraints : V = xyz and x, y, z > 0.Objective function : S(x, y, z) = 2xy + 2xz + xz


Evaluation

Simplification obtained by eliminating z variable using theconstraint.

Note, V = xyz ⇒ z = Vxy which gives,

Simplified Problem

Minimise,

S(x, y) = 2xy +2V

x+

V

y

such that,x, y > 0 and V ≡ constant.

How do we optimise such a function?


Motivation



System of Equations



Review of Univariate optimisation

Single Variable Calculus

Univariate = Single variable x ∈ [a, b] ⊂ R.

Objective function f : [a, b]−→R.

Location of Extrema

At which values of x do the extrema (maxima or minima) occur forthe function f(x)?

Two types of Extrema

Local Extrema

Global Extrema


Formal definition of the two types of Extrema

Let x ∈ [a, b] and f : [a, b]−→R.

Definition (Local maximum/minimum)

A value x0 is said to be a local maximum (minimum) iff(x0) ≥ f(x) (f(x0) ≤ f(x)) for all x in a neighbourhood of x0

xx∗x0

f(x)

Figure 1: Plot of a function that has one maximum and two minima.


Formal definition of the two types of Extrema

Let x ∈ [a, b] and f : [a, b]−→R.

Definition (Global maximum/minimum)

A value x∗ is said to be a global maximum (minimum) iff(x∗) ≥ f(x) (f(x∗) ≤ f(x)) for all x ∈ [a, b]

xx∗x0

f(x)

Figure 2: Plot of a function that has global minimum at x∗.


x−1 1

f(x)

Figure 3: Plot of a function that has global extrema at the endpoints.


x−1 1x∗x∗

f(x)

Figure 4: Plot of a function that has global extrema inside the domain.


Linear vs Non-linear Extrema

Linear functions

Extremum always attained at constraint boundaries

A local extremum is also a global extremum

Non-linear functions

Extrema may be in the interior as well as at boundaries.

A local extremum is not necessarily a global extremum.

Linear Nonlinear


Finding the local extrema

If f is differentiable on (a, b). That is, f ′(x) exists and is welldefined. Previous calculus methods can be used.

Find stationary points by checking the first derivative

Definition

A point x0 ∈ (a, b) is said to be a stationary point of f if

f ′(x0) = 0

�� A stationary point is also referred to as a critical point.

The behaviour of the stationary point can be determined by the..


Finding the local extrema (cont.)

Basic second derivative test

If f ′(x0) = 0 for some x0 ∈ (a, b) and

I If f ′′(x0) < 0, then x0 is the location of a local maximum.

I If f ′′(x0) > 0, then x0 is the location of a local minimum.

I If f ′′(x0) = 0, then test fails, x0 is possibly a point ofinflection.

f(x)

t

f ′′(t) > 0

g(x)

s

g′′(s) < 0


Finding the local extrema (cont.)

Generalised higher derivative test

Let m be a positive integer and assume that there exists anx0 ∈ (a, b) such that

f (1)(x0) = f (2)(x0) = . . . = f (2m−1)(x0) = 0.

Then the following holds

I If f (2m)(x0) < 0, then x0 is the location of a local maximum.

I If f (2m)(x0) > 0, then x0 is the location of a local minimum.

I If f (2m)(x0) = 0, then test fails, x0 is possibly a point ofinflection?


Motivation



System of Equations



Review of Multivariate optimisation

Multivariate Calculus

Multivariate = Many variables

x = (x1, x2, . . . , xn) ∈ D ⊂ Rn.

f = f(x1, x2, . . . , xn) = Objective function of those n variables

Necessary condition for stationary points

∂f∂x1

= ∂f∂x2

= · · · = ∂f∂xn

= 0 .

Sufficient conditions for maxima/minima of multivariate functionsconsidered later in the course.


Non-linear functions

Extrema may be in the interior as well as at boundaries.

A local extremum is not necessarily a global extremum.

Definition (Local maximum/minimum)

A point x0 = (x1, x2, . . . , xd) ∈ D is said to be a local maximum(minimum) if f(x0) ≥ f(x) (f(x0) ≤ f(x)) for all x in aneighbourhood of x0

Definition (Global maximum/minimum)

A value x∗ ∈ D is said to be a global maximum (minimum) iff(x∗) ≥ f(x) (f(x∗) ≤ f(x)) for all x ∈ D


Multivariate example

02

46 0

2

4

6−1

0

1

Figure 5: 3-dimensional function with stationary points.


Other type of multivariate Optimisation

Linear problems

Linear programming problem

Maximise/Minimise

f(x1, x2, . . . , xn) = c1x1 + c2x2 + · · ·+ cnxn,

such that,a1x1 + a2x2 + . . .+ anxn ≤ C,

where ci, ai and C are constants,


Linear Programming Solution?

In a similar vein to the univariate case,

Optimal solution for Linear problems

Extremum are always attained at constraint corner points

To see this note that if partial derivatives are set to zero:

∂f

∂x1= 0⇒ c1 = 0

......

...

∂f

∂xn= 0⇒ cn = 0.

Thus need to consider the boundaries for optimal solution

Solutions can be difficult in high dimensional problems.


Example : Manufacturing problem

A company manufactures two types of drugs by using threedifferent resources. The resources have limited supply. Thecompany wishes to maximise its profit.

I Each unit of drug earns the following profit:I Drug 1 earns a profit of $3,000.I Drug 2 earns a profit of $5,000.

I Each unit of drug uses the following resources:I Drug 1 uses 1 gm of Resource 1 and 3 gm of Resource 2.I Drug 2 uses 2 gm of Resource 2 and 2 gm of Resource 3.

I Each resource has the following supply limit.I Only 4 gm of Resource 1 are available.I Only 18 gm of Resource 2 are available.I Only 12 gm of Resource 3 are available.


Model formulation.

Let x1, x2 represent the number of units of Drug 1 and Drug 2produced.

Write problem mathematically with:

Maximize: Z = 3x1 + 5x2

subject to: x1 ≤ 43x1 + 2x2 ≤ 18

2x2 ≤ 12

with: x1 ≥ 0 , x2 ≥ 0 .

Problem formulation

Very important to be able to formulate problem mathematically.


Motivation



System of Equations



Elementary Row Operations

Consider the following system of linear equations:

x1 −x2 x3 = −22x1 +x2 −x3 = −5−x1 +2x2 3x3 = 0

(1)

Written in augmented matrix notation, 1 −1 1 −22 1 −1 −5−1 2 3 0

Solution vector x = (x∗1, x

∗2, x∗3) exists that satisfies (1).

Method uses Elementary Row Operations.


Elementary Row Operations

The three elementary row operations used are

1. Interchange two rows (denoted Ri ↔ Rj);

2. Multiply (or divide) any row by a non-zero constant (denotedRi → aRi);

3. Add a (non-zero) multiple of one row to any other row(denoted Ri → Ri + aRj).

Interchange of rows

Interchange of rows is not used in linear programming methods.

Solution is unchanged

Elementary row operations do not change the solution (x∗1, x∗2, x

∗3)

Return to example.


Return to the previous matrix:

T1 =

x1 x2 x3 RHS1 −1 1 −2 ← R1

2 1 −1 −5 ← R2

−1 2 3 0 ← R3

then the row-operations: R3 → R3 +R1 and R2 → R2 − 2R1

transform T1 to the equivalent tableau:

T2 =

x1 x2 x3 RHS

1 −1 1 −20 3 −3 −10 1 4 −2


Motivation



System of Equations



Pivot operations

Let aij be any non-zero element (called the pivot element) of thecoefficient matrix A of a given system of linear equations: i.e. aijis the element in row i and column j of the corresponding tableau.Then to pivot on aij 6= 0, denoted Pij ,

1. Divide row i by aij ;

2. Transform to zero all elements akj , k 6= i (i.e. the elements inthe same column j as aij except row i) by adding suitablemultiples of row i.

Result, a11 a12 · · · a1na21 a22 · · · a2n

...... aij

...an1 an2 · · · ann

Pij−→

b11 b12 0 b1nb21 b22 0 b2n

...... 1

...bn1 bn2 0 bnn


Pivot Example

Recall example tableau

x1 x2 x3 RHS1 −1 1 −22 1 −1 −5−1 2 3 0

.

Pivot the tableau T1 on the element a21 = 2 (row 2, column 1).

T2 = P21T1 =

x1 x2 x3 RHS0 −3/2 3/2 −9/21 1/2 −1/2 −5/20 5/2 5/2 −5/2

T2 is also equivalent to T1 since the pivot operation does notchange the solution set (x1, x2, x3) (Only uses ERO’s).


Example: Pivot solution

Pivot on the example matrix to obtain identity.

x1 x2 x3 RHS1 −1 1 −22 1 −1 −5−1 2 3 0

1 −1 1 −2P11 : 0 3 −3 −1

0 1 4 −21 0 0 −7/3

P22 : 0 1 −1 −1/30 0 5 −5/31 0 0 −7/6

P33 : 0 1 0 −2/30 0 1 −1/3


More variables than equations

Consider now the following system of three equations in fivevariables,

x1 −x2 +x3 −x4 = −22x1 +x2 −x3 +x5 = 5−x1 +2x2 +3x3 +x4 +2x5 = 0

I Express (x1, x2, x3) in terms of (x4, x5).

I The variables we solve for (x1, x2, x3), are called the basicvariables.

I They must be linearly independent.

I The remaining variables (x4, x5) are called non-basic variables:


Method of solving new system

Basic–Variables Non–Basic

x1 x2 x3 x4 x5 RHS

1 −1 1 −1 0 −22 1 −1 0 1 5−1 2 3 1 2 0

1 −1 1 −1 0 −20 3 −3 2 1 90 1 4 0 2 −21 0 0 −1/3 1/3 10 1 −1 2/3 1/3 30 0 5 −2/3 5/3 −51 0 0 −1/3 1/3 10 1 0 8/15 2/3 20 0 1 −2/15 1/3 −1


Solution for new equations

Basic–Variables Non–Basic

x1 x2 x3 x4 x5 RHS

1 0 0 −1/3 1/3 10 1 0 8/15 2/3 20 0 1 −2/15 1/3 −1

Solution given by:

x1 = 1 + 13x4 − 1

3x5

x2 = 2− 815x4 − 2

3x5

x3 = −1 + 215x4 − 1

3x5

MATH2070 - Introduction · I Final Exam 70%. ... Univariate = Single variable x 2[a;b] ˆR. ......

Documents

Transcript of MATH2070 - Introduction · I Final Exam 70%. ... Univariate = Single variable x 2[a;b] ˆR. ......