Computer Algorithms

Mathematical Programming

ECE 665Professor Maciej Ciesielski

By DFG

2Department of Electrical Engineering - UMASS

Overview Classical optimization deals with unconstrained

optimization problems.

For the minimization problem: Min f(x) subject to g(x) = 0

The solution is to move the constraints to the function and solve the unconstrained optimization problem.

• Penalty function method: replace f(x) byF(x) = f(x)+ M( g(x))2, M >> 0

• Lagrangian multipliers: replace f(x) byL(x,λ) = f(x) + Σ λi gi(x)At the optimum point, x*, L(x*, λ) = f(x*)

3Department of Electrical Engineering - UMASS

Outline

Mathematical programming paradigm Feasible solutions?

• Convexity• Concave• Concept example

Geometric solution Fundamental theorem of LP Simplex

• Example LP Modifications

4Department of Electrical Engineering - UMASS

Mathematical Programming Paradigm

A mathematical program is an optimization problem of the form:  

• Maximize (or Minimize) f(x)

  subject to:

• g(x) = 0• h(x) ≥ 0

 where x = [x1,...xn] is a subset of Rn, the functions g and h are called constraints, and f is called the objective function.

MathematicalProgrammingparadigm

5Department of Electrical Engineering - UMASS

The Nature of Mathematical Programming

Of course there are some forms that deviate from this paradigm.

Important examples are:

Fuzzy mathematical program Goal program Multiple objectives Randomized program Stochastic program

Therefore it is typically a modeling issue to find an standard form to solve the mathematical programming.

6Department of Electrical Engineering - UMASS

When is it feasible?

A point x is feasible if it is in X and satisfies the constraints:

• g(x) = 0 and h(x) ≥ 0.

A point x* is optimal if it is feasible and if the value of the objective function is:

• For Maximizationnot less than that of any other feasible solution: f(x*) ≥ f(x) for all feasible x.

• For Minimizationnot greater than that of any other feasible solution: f(x*) ≤ f(x) for all feasible x.

Min or Max f(x)

Subject to: g(x) = 0h(x) ≥ 0

Mathematical ProgrammingParadigm

7Department of Electrical Engineering - UMASS

Convexity

To obtain global optimum, it is important that the constraint set

 {x | g(x) = 0, h(x) ≥ 0}

and the function f(x) be convex.

If the function is convex over a convex constraint set, then local minimun is also a globa1 minimum

Min or Max f(x)

Subject to: g(x) = 0h(x) ≥ 0

Mathematical ProgrammingParadigm

8Department of Electrical Engineering - UMASS

Convexity

Set convexity (If any two points are in the set, so is their line segment)

Convex set non-convex set

Function convexity

Convex function non-convex function

???

9Department of Electrical Engineering - UMASS

Convexity

A function f:X–>R is said to be convex if:

Its epigraph is convex.

X is a convex set, and for x,y X and a[0, 1]:

f(ax + (1-a)y) ≤ af(x) + (1-a)f(y)

Epigraph: region on or above the graph

10Department of Electrical Engineering - UMASS

Convexity

If the function is convex over a convex constraint set, then local minimun is also a globa1 minimum

11Department of Electrical Engineering - UMASS

Concave

It is define for maximization, and it is the negative of convexity.

A function f is concave if its hypograph is convex

Hypograph: region below the graph

12Department of Electrical Engineering - UMASS

Concave

If the function is concave over a convex constraint set, then local maximun is also a globa1 maximum

13Department of Electrical Engineering - UMASS

Concept Example

Can we minimize based on the local minimum?

f(x) = xTQx

Subject to:

Ax ≥ bx ≥ 0

Where Q is an n x n symmetric matrix, A is a constraint matrix, and b is a constraint vector.

QuadraticProgramming

14Department of Electrical Engineering - UMASS

Concept Example If xTQx > 0 for every x 0.

The matrix Q is positive definite. In which case its eigenvalues are non-negative, and the quadratic form is a convex function. Thus f is convex.

As Q is a symmetric matrix, these eingenvalues will be reals.

Ax ≥ b defines a convex constraint set, as its solution is a Rk space. (A convex polyhedron)

f and the constraint set are convex, therefore, finding the local minimum also finds the global minimum.

15Department of Electrical Engineering - UMASS

Linear Programming (LP) The goal of Linear programming is to:

(Max)Minimize: CTxSubject to: Ax ≥ b

x ≥ 0

Where CT is a coefficient vector for f, A is a constraint matrix, and b es a constraint vector.

The constraint set: {x | Ax ≥ b} is a convex polyhedron, and f is linear so it is convex.

Therefore, LP has convexity, and the local min/max is the global min/max

16Department of Electrical Engineering - UMASS

LP Example (Problem Enunciation)

Optimization problem:

2 types of products are made by the factory 3 machines are needed to make each product

 Constraints: Machines are available for a limited time during one production session:

time on A ≤ 50 hours time on B ≤ 35 hours time on C ≤ 80 hours

17Department of Electrical Engineering - UMASS

LP Example (Problem Enunciation)

To complete the product, each lot must be processed by all three machines for a certain number of hours  Required manufacturing time

Machines A B C 1

Products 2Time Constraint: 50 35 80

10 5 55 5 15

18Department of Electrical Engineering - UMASS

LP Example (Problem Enunciation)

Profit:

Product 1 can be sold for $100 per lot Product 2 can be sold for $80 per lot

Optimization Problem:  Design the production schedule which

maximizes the profit of both products manufactured during one production session.

19Department of Electrical Engineering - UMASS

LP Example (Problem Formulation)

Formulation   x1 = number of lots of product 1

x2 = number of lots of product 2 

max f (x) = 100x1 + 80x2

10x1 + 5x2 ≤ 505x1 +   5x2 ≤ 355x1 +   15x2 ≤ 80

20Department of Electrical Engineering - UMASS

LP Example (Geometric Solution)

Geometric solution: Family of curves, constant for x1, x2

F(x) = 100 x1 + 80 x2 = 800 (const)

(a) 10x1 + 5x2 50 (b) 5x1 + 5x2 35(c) 5x1 + 15x2 80

a bx1

x2

10

5

15105 c

P2

P13

4

P0

F(x) = 800

Sol:X1 = 3X2 = 4

Extremepoint

21Department of Electrical Engineering - UMASS

LP Example (Standard Linear Program Approach)

Convert the problem to a standard Linear Program (LP):

Max f(x) = 100x1 + 8Ox2 + 0s1 + 0s2 + 0s3

10x1 + 5x2 + s1 = 50 Subject to: 5x1 + 5x2 + s2 = 35 5x1 + 15x2+ s3 = 80

Basis

New Matrix A

Why theBasis??

22Department of Electrical Engineering - UMASS

Basis and Basic solution

Why the Basis?• Because it finds a basic solution

 Given a system of equalities: Ax = b

 where: x = n-vector, b = m-vector, A = mn matrix

select a set of m linearly independent columns such that (mm) matrix B is nonsingular, i.e. |B| 0.

Then one may uniquely solve the equation: Bxb= b

where: xb is an m-element subvector of x.

Max f(x) = 100x1 + 8Ox2 + 0s1 + 0s2 + 0s3

Subject to: 10x1 + 5x2 + s1

= 505x1 + 5x2 + s2

= 35 5x1 + 15x2 + s3

= 80

23Department of Electrical Engineering - UMASS

Basis and Basic solution

By putting x=(xb,0) we obtain a solution to

Ax = b

Such a solution (with n-m components of x not associated with columns of Bmm) to the resulting set of equations is said to be a basic solution with regard to the basis B.

The Xb variables, associated with columns of B, are called basic variables.

Basic Feasible Solution Basic Feasible Solution

If a feasible solution (i.e. the one which satisfies all contraints) is also basic, it is called a basic feasible solution.

Max f(x) = 100x1 + 8Ox2 + 0s1 + 0s2 + 0s3

Subject to: 10x1 + 5x2 + s1

= 505x1 + 5x2 + s2

= 35 5x1 + 15x2 + s3

= 80

B.F.S x1 = 0 s1 = 50 s3 = 80 x2 = 0 s2 = 35

basic variables

24Department of Electrical Engineering - UMASS

Fundamental Theorem of Linear Programming

Given an LP in standard form

min CTxsubject to Ax = bx 0

where A is mn matrix of rank m (i.e. m < n and the m rows are linear independent), 

• If there is a feasible solution, there is a basic feasible solution

• If there is an optimal feasible solution, there is an optimal basic feasible solution

25Department of Electrical Engineering - UMASS

Fundamental Theorem of Linear Programming

This theorem shows that it is necessary only to consider basic feasible solutions when seeking an optimal solution to a linear program, as the optimal value is always achieved as such a solution

  Ultimate goalUltimate goal::

• find a basic feasible solution with a base B composed of original variables only, and which is optimum.

26Department of Electrical Engineering - UMASS

LP Example (Basic Feasible solution)

ƒ (x) = 100 x1 + 80 x2 + 0 s1 + 0 s2 + 0 s3   

Solution: x1 = 0, x2 = 0, s1 = 50, s2 = 35, s3 = 80, is a basic feasible solution: point P0(0,0) Extreme point

Function value: F(0, 0, 50, 35, 80) = 0

1

210 5 1 0 0 505 5 0 1 0 3515 15 0 0 1 802

3

xxsss

basis

27Department of Electrical Engineering - UMASS

LP Example (Basic Feasible solution)

The basic feasible solution in the Geometric approach is in the Extreme Point.

a bx1

x2

10

5

15105 c

P2

P13

4

P0

Extremepoint

28Department of Electrical Engineering - UMASS

Simplex Method

Proceed from one basic feasible solution (extreme point) to another, in such a way as to continually decrease / increase the value of the f (x) until a minimum / maximum is reached. 

General comments:General comments:

It is easy find initial basic feasible solutions with slack variables.

Finding initial basic solution is part of the Simplex method (Lue 84).

29Department of Electrical Engineering - UMASS

Simplex Algorithm

1. Select the column, such that the new resulting basic feasible solution will yield a lower / greater value to f(x) than the previous one.

2. Select the pivot element in that column.

By an elementary evaluation determine:

• Which vector aj should enter the basis (so that f(x) is reduced), and

• Which vector should leave the basis.

30Department of Electrical Engineering - UMASS

Simplex Algorithm

Construct a Simplex Table, [A, b ] appended by a row at the top with cost coefficients and current cost.

j =j = 1 2 3 4 5 0

rrjj 100 80 0 0 0 0yy1j1j 10 5 1 0 0 50

yy2j2j 5 5 0 1 0 35yy3j3j 5 15 0 0 1 80

yi1 yi2 yi3 yi4 yi5 yi0

Point P0 (0,0)F(0,0,50,35,80) = 0

31Department of Electrical Engineering - UMASS

Simplex Algorithm

Step 1. Select column q with rq > 0. (xq enters the basis) q : rq = max ri

Step 2. Select row p, which defines pivot element p in column q Which minimizes ratio:

p : = min for positive yiqi

0i

iq

yy

0i

iq

yy

0p

pq

yy

32Department of Electrical Engineering - UMASS

Simplex Algorithm

Step 3. Pivot on element ypq and update the table (update all rows, including the top one)

• In row p divide ypj by ypq , j = 0, 1,…, n

• Subtract from each row i p

ypj , j = 0, 1,…, n 0p

pq

yy

33Department of Electrical Engineering - UMASS

Simplex Algorithm Example

Pivot on y11

j =j = 1 2 3 4 5 0

rrjj 100 80 0 0 0 0yy1j1j 10 5 1 0 0 50yy2j2j 5 5 0 1 0 35yy3j3j 5 15 0 0 1 80

yi1 yi2 yi3 yi4 yi5 yi0 Point P0 (0,0)F(0,0,50,35,80) = 0

Step1: max rq

10

11

20

21

30

31

50 51035 75

80 165

yyyyyy

Step2: min of

Therfore: ypq = 10

34Department of Electrical Engineering - UMASS

Simplex Algorithm Example

Update table

j =j = 1 2 3 4 5 0

rrjj 100 80 0 0 0 0yy1j1j 10 5 1 0 0 50yy2j2j 5 5 0 1 0 35yy3j3j 5 15 0 0 1 80

yi1 yi2 yi3 yi4 yi5 yi0Point P0 (0,0)F(0,0,50,35,80) = 0

1iq iq

ij ij pj ij pjpq

y yy y y y y

y

Step 3.B: in row i ≠ p

1 1

11 10pj j j

pq

y y yy y

Step 3.A: in row p

j =j = 1 2 3 4 5 0

rrjj 100 80 0 0 0 0yy1j1j 1 0.5 0.1 0 0 5yy2j2j 5 5 0 1 0 35yy3j3j 5 15 0 0 1 80

yi1 yi2 yi3 yi4 yi5 yi0

j =j = 1 2 3 4 5 0

rrjj 0 30 -10 0 0 -500yy1j1j 1 0.5 0.1 0 0 5yy2j2j 0 2.5 -0.5 1 0 10yy3j3j 0 12.5 -0.5 0 1 55

yi1 yi2 yi3 yi4 yi5 yi0 Point P1 (5,0)F(5,0,0,10,55) = 500

35Department of Electrical Engineering - UMASS

Simplex Algorithm Example

Pivot on y22

j =j = 1 2 3 4 5 0

rrjj 0 30 -10 0 0 -500yy1j1j 1 0.5 0.1 0 0 5yy2j2j 0 2.5 -0.5 1 0 10yy3j3j 0 12.5 -0.5 0 1 55

yi1 yi2 yi3 yi4 yi5 yi0Point P1 (5,0)F(5,0,0,10,55) = 500

Step1: max rq

10

12

20

22

30

32

5 100.510 42.555 4.4

12.5

yyyyyy

Step2: min of

Therfore: ypq = 4

36Department of Electrical Engineering - UMASS

j =j = 1 2 3 4 5 0

rrjj 100 80 0 0 0 0yy1j1j 1 0.5 0.1 0 0 5yy2j2j 5 5 0 1 0 35yy3j3j 5 15 0 0 1 80

yi1 yi2 yi3 yi4 yi5 yi0

Simplex Algorithm Example

Update table

Point P1 (5,0)F(5,0,0,10,55) = 500

1iq iq

ij ij pj ij pjpq

y yy y y y y

y

Step 3.B: in row i ≠ p

2 2

22 2.5pj j j

pq

y y yy y

Step 3.A: in row p

Point P2 (3,4)F(3,4,0,0,5) = 620

j =j = 1 2 3 4 5 0

rrjj 0 30 -10 0 0 -500yy1j1j 1 0.5 0.1 0 0 5yy2j2j 0 1 -0.2 0.4 0 4yy3j3j 0 12.5 -0.5 0 1 55

yi1 yi2 yi3 yi4 yi5 yi0

j =j = 1 2 3 4 5 0

rrjj 0 0 -4 -12 0 -620yy1j1j 1 0 1 0 0 3yy2j2j 0 1 -0.2 0.4 0 4yy3j3j 0 0 2 -5 1 5

yi1 yi2 yi3 yi4 yi5 yi0

37Department of Electrical Engineering - UMASS

Simplex Algorithm Example

Pivot on

j =j = 1 2 3 4 5 0

rrjj 0 0 -4 -12 0 -620yy1j1j 1 0 1 0 0 3yy2j2j 0 1 -0.2 0.4 0 4yy3j3j 0 0 2 -5 1 5

yi1 yi2 yi3 yi4 yi5 yi0Point P2 (3,4)F(3,4,0,0,5) = 620

Step1: all rq < 0Terminate algorithm

Sol:X1 = 3X2 = 4

38Department of Electrical Engineering - UMASS

Linear Programming Modifications

1. Maximization

2. Unconstrained xmax c t x = min (-c t x)

let x = x+ - x - ; x+, x - 0)

min (c t x+ - c t x -)

[A, -A] = b

x+

x -

x+, x - 0

s. to

39Department of Electrical Engineering - UMASS

Linear Programming Modifications

3. Constraint set: Ax b

Slack variables, xs : Ax + xs = b

[A, I ] = bxxxxss

x , xs 0min [cT , 0]

xxs

Columns corresponding to xs form a basic solution

40Department of Electrical Engineering - UMASS

Linear Programming Modifications

4. Constraint set: Ax ≥ b

Surplus variables, xp : Ax - xp = b

[A, -I ] = bxxxxpp

x , xp 0min [cT , 0]

xxp

41Department of Electrical Engineering - UMASS

Linear Programming Modifications5. Artificial variables, xa : Ax - xp + xa = b

(cannot star with negative basis of xp )

[A, -I , I ] = b

x , xp , xa 0

min [cT , 0 , k I ]

xxp

xa

xxp

xaS. to

Columns corresponding to xa form a basic solution