Mathematical Programming

Computer Algorithms

Mathematical Programming

ECE 665Professor Maciej Ciesielski

By DFG

2Department of Electrical Engineering - UMASS

Overview Classical optimization deals with unconstrained

optimization problems.

For the minimization problem: Min f(x) subject to g(x) = 0

The solution is to move the constraints to the function and solve the unconstrained optimization problem.

• Penalty function method: replace f(x) byF(x) = f(x)+ M( g(x))2, M >> 0

• Lagrangian multipliers: replace f(x) byL(x,λ) = f(x) + Σ λi gi(x)At the optimum point, x*, L(x*, λ) = f(x*)


Outline

Mathematical programming paradigm Feasible solutions?

• Convexity• Concave• Concept example

Geometric solution Fundamental theorem of LP Simplex

• Example LP Modifications


Mathematical Programming Paradigm

A mathematical program is an optimization problem of the form:

• Maximize (or Minimize) f(x)

subject to:

• g(x) = 0• h(x) ≥ 0

where x = [x1,...xn] is a subset of Rn, the functions g and h are called constraints, and f is called the objective function.

MathematicalProgrammingparadigm


The Nature of Mathematical Programming

Of course there are some forms that deviate from this paradigm.

Important examples are:

Fuzzy mathematical program Goal program Multiple objectives Randomized program Stochastic program

Therefore it is typically a modeling issue to find an standard form to solve the mathematical programming.

http://carbon.cudenver.edu/~hgreenbe/glossary/second.php?page=F.html#Fuzzy_math_program

http://carbon.cudenver.edu/~hgreenbe/glossary/second.php?page=G.html#Goal_program


When is it feasible?

A point x is feasible if it is in X and satisfies the constraints:

• g(x) = 0 and h(x) ≥ 0.

A point x* is optimal if it is feasible and if the value of the objective function is:

• For Maximizationnot less than that of any other feasible solution: f(x*) ≥ f(x) for all feasible x.

• For Minimizationnot greater than that of any other feasible solution: f(x*) ≤ f(x) for all feasible x.

Min or Max f(x)

Subject to: g(x) = 0h(x) ≥ 0

Mathematical ProgrammingParadigm


Convexity

To obtain global optimum, it is important that the constraint set

{x | g(x) = 0, h(x) ≥ 0}

and the function f(x) be convex.

If the function is convex over a convex constraint set, then local minimun is also a globa1 minimum

Min or Max f(x)

Subject to: g(x) = 0h(x) ≥ 0

Mathematical ProgrammingParadigm


Convexity

Set convexity (If any two points are in the set, so is their line segment)

Convex set non-convex set

Function convexity

Convex function non-convex function

???


Convexity

A function f:X–>R is said to be convex if:

Its epigraph is convex.

X is a convex set, and for x,y X and a[0, 1]:

f(ax + (1-a)y) ≤ af(x) + (1-a)f(y)

Epigraph: region on or above the graph


Convexity

If the function is convex over a convex constraint set, then local minimun is also a globa1 minimum


Concave

It is define for maximization, and it is the negative of convexity.

A function f is concave if its hypograph is convex

Hypograph: region below the graph


Concave

If the function is concave over a convex constraint set, then local maximun is also a globa1 maximum


Concept Example

Can we minimize based on the local minimum?

f(x) = xTQx

Subject to:

Ax ≥ bx ≥ 0

Where Q is an n x n symmetric matrix, A is a constraint matrix, and b is a constraint vector.

QuadraticProgramming


Concept Example If xTQx > 0 for every x 0.

The matrix Q is positive definite. In which case its eigenvalues are non-negative, and the quadratic form is a convex function. Thus f is convex.

As Q is a symmetric matrix, these eingenvalues will be reals.

Ax ≥ b defines a convex constraint set, as its solution is a Rk space. (A convex polyhedron)

f and the constraint set are convex, therefore, finding the local minimum also finds the global minimum.


Linear Programming (LP) The goal of Linear programming is to:

(Max)Minimize: CTxSubject to: Ax ≥ b

x ≥ 0

Where CT is a coefficient vector for f, A is a constraint matrix, and b es a constraint vector.

The constraint set: {x | Ax ≥ b} is a convex polyhedron, and f is linear so it is convex.

Therefore, LP has convexity, and the local min/max is the global min/max


LP Example (Problem Enunciation)

Optimization problem:

2 types of products are made by the factory 3 machines are needed to make each product

Constraints: Machines are available for a limited time during one production session:

time on A ≤ 50 hours time on B ≤ 35 hours time on C ≤ 80 hours



To complete the product, each lot must be processed by all three machines for a certain number of hours Required manufacturing time

Machines A B C 1

Products 2Time Constraint: 50 35 80

10 5 55 5 15



Profit:

Product 1 can be sold for $100 per lot Product 2 can be sold for $80 per lot

Optimization Problem: Design the production schedule which

maximizes the profit of both products manufactured during one production session.


LP Example (Problem Formulation)

Formulation x1 = number of lots of product 1

x2 = number of lots of product 2

max f (x) = 100x1 + 80x2

10x1 + 5x2 ≤ 505x1 + 5x2 ≤ 355x1 + 15x2 ≤ 80


LP Example (Geometric Solution)

Geometric solution: Family of curves, constant for x1, x2

F(x) = 100 x1 + 80 x2 = 800 (const)

(a) 10x1 + 5x2 50 (b) 5x1 + 5x2 35(c) 5x1 + 15x2 80

a bx1

x2

10

5

15105 c

P2

P13

4

P0

F(x) = 800

Sol:X1 = 3X2 = 4

Extremepoint


LP Example (Standard Linear Program Approach)

Convert the problem to a standard Linear Program (LP):

Max f(x) = 100x1 + 8Ox2 + 0s1 + 0s2 + 0s3

10x1 + 5x2 + s1 = 50 Subject to: 5x1 + 5x2 + s2 = 35 5x1 + 15x2+ s3 = 80

Basis

New Matrix A

Why theBasis??


Basis and Basic solution

Why the Basis?• Because it finds a basic solution

Given a system of equalities: Ax = b

where: x = n-vector, b = m-vector, A = mn matrix

select a set of m linearly independent columns such that (mm) matrix B is nonsingular, i.e. |B| 0.

Then one may uniquely solve the equation: Bxb= b

where: xb is an m-element subvector of x.

Max f(x) = 100x1 + 8Ox2 + 0s1 + 0s2 + 0s3

Subject to: 10x1 + 5x2 + s1

= 505x1 + 5x2 + s2

= 35 5x1 + 15x2 + s3

= 80


Basis and Basic solution

By putting x=(xb,0) we obtain a solution to

Ax = b

Such a solution (with n-m components of x not associated with columns of Bmm) to the resulting set of equations is said to be a basic solution with regard to the basis B.

The Xb variables, associated with columns of B, are called basic variables.

Basic Feasible Solution Basic Feasible Solution

If a feasible solution (i.e. the one which satisfies all contraints) is also basic, it is called a basic feasible solution.

Max f(x) = 100x1 + 8Ox2 + 0s1 + 0s2 + 0s3

Subject to: 10x1 + 5x2 + s1

= 505x1 + 5x2 + s2

= 35 5x1 + 15x2 + s3

= 80

B.F.S x1 = 0 s1 = 50 s3 = 80 x2 = 0 s2 = 35

basic variables


Fundamental Theorem of Linear Programming

Given an LP in standard form

min CTxsubject to Ax = bx 0

where A is mn matrix of rank m (i.e. m < n and the m rows are linear independent),

• If there is a feasible solution, there is a basic feasible solution

• If there is an optimal feasible solution, there is an optimal basic feasible solution


Fundamental Theorem of Linear Programming

This theorem shows that it is necessary only to consider basic feasible solutions when seeking an optimal solution to a linear program, as the optimal value is always achieved as such a solution

Ultimate goalUltimate goal::

• find a basic feasible solution with a base B composed of original variables only, and which is optimum.


LP Example (Basic Feasible solution)

ƒ (x) = 100 x1 + 80 x2 + 0 s1 + 0 s2 + 0 s3

Solution: x1 = 0, x2 = 0, s1 = 50, s2 = 35, s3 = 80, is a basic feasible solution: point P0(0,0) Extreme point

Function value: F(0, 0, 50, 35, 80) = 0

1

210 5 1 0 0 505 5 0 1 0 3515 15 0 0 1 802

3

xxsss

basis


LP Example (Basic Feasible solution)

The basic feasible solution in the Geometric approach is in the Extreme Point.

a bx1

x2

10

5

15105 c

P2

P13

4

P0

Extremepoint


Simplex Method

Proceed from one basic feasible solution (extreme point) to another, in such a way as to continually decrease / increase the value of the f (x) until a minimum / maximum is reached.

General comments:General comments:

It is easy find initial basic feasible solutions with slack variables.

Finding initial basic solution is part of the Simplex method (Lue 84).


Simplex Algorithm

1. Select the column, such that the new resulting basic feasible solution will yield a lower / greater value to f(x) than the previous one.

2. Select the pivot element in that column.

By an elementary evaluation determine:

• Which vector aj should enter the basis (so that f(x) is reduced), and

• Which vector should leave the basis.


Simplex Algorithm

Construct a Simplex Table, [A, b ] appended by a row at the top with cost coefficients and current cost.

j =j = 1 2 3 4 5 0

rrjj 100 80 0 0 0 0yy1j1j 10 5 1 0 0 50

yy2j2j 5 5 0 1 0 35yy3j3j 5 15 0 0 1 80

yi1 yi2 yi3 yi4 yi5 yi0

Point P0 (0,0)F(0,0,50,35,80) = 0


Simplex Algorithm

Step 1. Select column q with rq > 0. (xq enters the basis) q : rq = max ri

Step 2. Select row p, which defines pivot element p in column q Which minimizes ratio:

p : = min for positive yiqi

0i

iq

yy

0i

iq

yy

0p

pq

yy


Simplex Algorithm

Step 3. Pivot on element ypq and update the table (update all rows, including the top one)

• In row p divide ypj by ypq , j = 0, 1,…, n

• Subtract from each row i p

ypj , j = 0, 1,…, n 0p

pq

yy


Simplex Algorithm Example

Pivot on y11

j =j = 1 2 3 4 5 0

rrjj 100 80 0 0 0 0yy1j1j 10 5 1 0 0 50yy2j2j 5 5 0 1 0 35yy3j3j 5 15 0 0 1 80

yi1 yi2 yi3 yi4 yi5 yi0 Point P0 (0,0)F(0,0,50,35,80) = 0

Step1: max rq

10

11

20

21

30

31

50 51035 75

80 165

yyyyyy

Step2: min of

Therfore: ypq = 10



Update table

j =j = 1 2 3 4 5 0

rrjj 100 80 0 0 0 0yy1j1j 10 5 1 0 0 50yy2j2j 5 5 0 1 0 35yy3j3j 5 15 0 0 1 80

yi1 yi2 yi3 yi4 yi5 yi0Point P0 (0,0)F(0,0,50,35,80) = 0

1iq iq

ij ij pj ij pjpq

y yy y y y y

y

Step 3.B: in row i ≠ p

1 1

11 10pj j j

pq

y y yy y

Step 3.A: in row p

j =j = 1 2 3 4 5 0

rrjj 100 80 0 0 0 0yy1j1j 1 0.5 0.1 0 0 5yy2j2j 5 5 0 1 0 35yy3j3j 5 15 0 0 1 80


j =j = 1 2 3 4 5 0

rrjj 0 30 -10 0 0 -500yy1j1j 1 0.5 0.1 0 0 5yy2j2j 0 2.5 -0.5 1 0 10yy3j3j 0 12.5 -0.5 0 1 55

yi1 yi2 yi3 yi4 yi5 yi0 Point P1 (5,0)F(5,0,0,10,55) = 500



Pivot on y22

j =j = 1 2 3 4 5 0

rrjj 0 30 -10 0 0 -500yy1j1j 1 0.5 0.1 0 0 5yy2j2j 0 2.5 -0.5 1 0 10yy3j3j 0 12.5 -0.5 0 1 55


Step1: max rq

10

12

20

22

30

32

5 100.510 42.555 4.4

12.5

yyyyyy

Step2: min of

Therfore: ypq = 4


j =j = 1 2 3 4 5 0

rrjj 100 80 0 0 0 0yy1j1j 1 0.5 0.1 0 0 5yy2j2j 5 5 0 1 0 35yy3j3j 5 15 0 0 1 80



Update table

Point P1 (5,0)F(5,0,0,10,55) = 500

1iq iq

ij ij pj ij pjpq

y yy y y y y

y

Step 3.B: in row i ≠ p

2 2

22 2.5pj j j

pq

y y yy y

Step 3.A: in row p

Point P2 (3,4)F(3,4,0,0,5) = 620

j =j = 1 2 3 4 5 0

rrjj 0 30 -10 0 0 -500yy1j1j 1 0.5 0.1 0 0 5yy2j2j 0 1 -0.2 0.4 0 4yy3j3j 0 12.5 -0.5 0 1 55


j =j = 1 2 3 4 5 0

rrjj 0 0 -4 -12 0 -620yy1j1j 1 0 1 0 0 3yy2j2j 0 1 -0.2 0.4 0 4yy3j3j 0 0 2 -5 1 5




Pivot on

j =j = 1 2 3 4 5 0

rrjj 0 0 -4 -12 0 -620yy1j1j 1 0 1 0 0 3yy2j2j 0 1 -0.2 0.4 0 4yy3j3j 0 0 2 -5 1 5


Step1: all rq < 0Terminate algorithm

Sol:X1 = 3X2 = 4


Linear Programming Modifications

1. Maximization

2. Unconstrained xmax c t x = min (-c t x)

let x = x+ - x - ; x+, x - 0)

min (c t x+ - c t x -)

[A, -A] = b

x+

x -

x+, x - 0

s. to



3. Constraint set: Ax b

Slack variables, xs : Ax + xs = b

[A, I ] = bxxxxss

x , xs 0min [cT , 0]

xxs

Columns corresponding to xs form a basic solution



4. Constraint set: Ax ≥ b

Surplus variables, xp : Ax - xp = b

[A, -I ] = bxxxxpp

x , xp 0min [cT , 0]

xxp


Linear Programming Modifications5. Artificial variables, xa : Ax - xp + xa = b

(cannot star with negative basis of xp )

[A, -I , I ] = b

x , xp , xa 0

min [cT , 0 , k I ]

xxp

xa

xxp

xaS. to

Columns corresponding to xa form a basic solution

Mathematical Programming

Documents

Transcript of Mathematical Programming