University of Massachusetts at Amherst 665 Computer Algorithm Mathematical Programming Professor...
-
date post
22-Dec-2015 -
Category
Documents
-
view
220 -
download
7
Transcript of University of Massachusetts at Amherst 665 Computer Algorithm Mathematical Programming Professor...
University of Massachusetts at Amherst
665 Computer Algorithm Mathematical Programming
Professor Maciej CiesielskiSpring 2004
(DFG)
OUTLINE
1. Non Linear Programming2. Quadratic Programming3. Linear Programming
A. Geometric SolutionsB. Basic SolutionsC. Simplex MethodD. LP modifications
4. Integer ProgrammingA. UnimodularityB. Network Flow Problems
5. Conclusions
Mathematical Programming Nonlinear Programming
Minimize f(x) subject to:
g(x) = 0h(x) ≥ 0
where x = [x1,...xn]T, and all functions f, g, h, are differentiable.
{
Convexity
To obtain global optimum, it is important that the constraint set
{x | g(x) = 0,h(x) ≥ 0}
and the function f(x) be convex.
If the function is convex over a convex constraint set, then local minimun is also a globa1 minimum
Set convexity
Convex set non-convex set
Function convexity
Convex function non-convex function
Classical optimization deals with unconstrained optimization problems.
Solution: move the constraints to the function and solve the unconstrained optimization problem
For the minimization problem
Min f(x) subject to g(x) = 0
1. Penalty function method: replace f(x) by
F(x) = f(x)+ M( g(x))2, M >> 0
2. Lagrangian multipliers: replace f(x) by
L(x,λ) = f(x) + Σ λi gi(x)
At the optimum point, x*, L(x*, λ) = f(x*)
i
Quadratic Programming (QP)
Minimize xTQx
Subject to:
{Ax ≥ b
Where Q is an n x n symmetric matrix, A is a constraint matrix, and b is a constraint vector
Matrix Q is positive definite, i.e.,
xTQx > 0
For every x 0. Thus f is convex.
Linear Programming (LP)
Minimize cT
Subject to
Ax ≥ b
x ≥ 0
Where cT is a coefficient vector for f, A is a constraint matrix, and b es a constraint vector.
The constraint set
{x | Ax ≥ b}
Is a convex polyhedron, and f is linear, convex.
{
Linear Programming (LP)Linear Programming (LP)
Optimization problem:Optimization problem: • 2 types of products are made by the factory• 2 types of products are made by the factory• 3 machines are needed to make each • 3 machines are needed to make each
productproduct ConstraintsConstraints:: Machines are available for a Machines are available for a limited time during one production session:limited time during one production session: •• time on time on A A ≤≤ 50 50 hourshours • • time on time on B B ≤≤ 35 35 hourshours • • time on time on C C ≤≤ 80 hours 80 hours
To complete the product, each lot must be processed by all three machines for a certain number of hours Required manufacturing time
Machines A B C 1
2
Time Constraint: 50 35 80
10 5 5
5 5 15
Profit• Product 1 can be sold for $100 per lot• Product 2 can be sold for $80 per lot Optimization Problem: Design the production schedule which rnaximizes the profit of both products manufactured during one production session.
Formulation x1 = number of lots of product 1
x2 = number of lots of product 2
max f (x) = 100x1 + 80x2
10x1 + 5x2 ≤≤ 505x1 + 5x2 ≤≤ 355x1 + 15x2 ≤≤ 80
{
Convert the problem to a standard LinearProgram (LP):
Max f(x) = 100x1 + 8Ox2
10x1 + 5x2+ s1 = 50
s. to 5x1 + 5x2+ s2 = 35
5x1 + 15x2+ s3 = 80
Basis
New Matrix A
{
Geometric solutionGeometric solution
Family of curves, constant for Family of curves, constant for xx11,, xx22,,
F(x) = 100 xF(x) = 100 x1,1, ++ 80 x80 x22 = 800 = 800
(const)(const)
(a)(a)10x10x1,1, ++ 5x5x22 50 50
(b)(b) 5x 5x1, 1, + 5x+ 5x22 35 35
(c)(c) 5x 5x1, 1, + 15x+ 15x22 80 80
a bxx11
xx22
15
10
5
15105 c
P2
P13
4
P0
F(x) = 800F(x) = 800
Basic Solution Given a system of equalities
Ax=b where x = n - vector, b = m - vector, A = m x n matrix, select a set of m linearly independent columns such that (m x m) matrix B is nonsingular, i.e. /B / 0. Then one may uniquely solve the equation
Bxb= bwhere Xb is an m - element subvector of x.
By putting x=(xb,0) we obtain a solution to
Ax = b Such a solution (with n - m components of x not associated with columns of B m x m ) to the resulting set of equations is said to be a basic solution with regard to the basis B. The Xb variables, associated with columns of B, are called basic variables.
Basic Feasible SolutionBasic Feasible Solution
If a feasible solution (i.e. the one which satisfies all contraints) is also basic, it is called a basic feasible solution.
Fundamental Theorem of Linear Fundamental Theorem of Linear ProgrammingProgramming
Given an L.P. in standard formGiven an L.P. in standard form
min cmin cTTxx
Ax = bAx = b
x x 0 0
where where A A is is m m xx n n matrix of rank matrix of rank m m (i.e. (i.e. m < n m < n and and the the m m rows are linear independent),rows are linear independent),
s. to
If there is a feasible solution, there is a If there is a feasible solution, there is a basic feasible solutionbasic feasible solution
If there is an optimal feasible solution, there If there is an optimal feasible solution, there is an optimal basic feasible solutionis an optimal basic feasible solution
This theorem shows that it is necessary only to This theorem shows that it is necessary only to consider consider basic feasible solutionsbasic feasible solutions when seeking an when seeking an optimal solution to a linear program, as the optimal solution to a linear program, as the optimal value is always achieved as such a optimal value is always achieved as such a solutionsolution
Ultimate goalUltimate goal::
find a basic feasible solution with a base find a basic feasible solution with a base B B composed of composed of originaloriginal variables only, and which is variables only, and which is optimum.optimum.
Solution: Solution: xx1 1 = 0, x= 0, x2 2 = 0, s= 0, s1 1 = 50, s= 50, s2 2 = 35, s= 35, s3 3 = 80,= 80,
is is a basic feasible solution (point a basic feasible solution (point PP00 (0,0) (0,0)))
Function value:Function value: F(0, 0. 50, 35, 80) = 0F(0, 0. 50, 35, 80) = 0
ƒ (x) = 100 xƒ (x) = 100 x1,1, ++ 80 x80 x22 + + 0 s0 s1 1 ++ 0 s0 s2 2 ++ 0 s0 s3 3
1010 5 5 100100
55 5 5 010010
55 1515 001001
xx11
xx22
ss11
ss2,2,
ss33
5 05 0
3 53 5
8 08 0
=
basis
Simplex MethodSimplex Method
Proceed from one basic feasible solution (extreme Proceed from one basic feasible solution (extreme point) to another, in such a way as to continually point) to another, in such a way as to continually decrease the value of the decrease the value of the f (x) f (x) until a minimum is until a minimum is reached. reached.
General commentsGeneral comments
It is easy to find initial basic feasible solutions It is easy to find initial basic feasible solutions with with slack variablesslack variables..
Finding initial basic solution is part of the Simplex Finding initial basic solution is part of the Simplex method (Lue 84).method (Lue 84).
Simplex AlgorithmSimplex Algorithm
1.1. Select the column, such that the new resulting Select the column, such that the new resulting basic feasible solution will yield a basic feasible solution will yield a lower valuelower value to to f(x) f(x) than the previous one.than the previous one.
2.2. Select the pivot element in that column.Select the pivot element in that column.
By an elementary evaluation determineBy an elementary evaluation determine
• Which vector Which vector aajj should enter the basis (so that should enter the basis (so that f(x) f(x) is reduced), andis reduced), and
• Which vector should leave the basis. Which vector should leave the basis.
Construct a Simplex Tableau, [Construct a Simplex Tableau, [A, b A, b ] appended by ] appended by a row at the top with cost coefficients and current a row at the top with cost coefficients and current cost.cost.
j =j = 1 2 3 4 5 0
rrjj 100 80 0 0 0 0
yy1j1j 10 5 1 0 0 50
yy2j2j 5 5 0 1 0 35
yy3j3j 5 15 0 0 1 80
yi1 yi2 yi3 yi4 yi5 yi0
Point P0 (0,0)
F(0,0,50,35,80) = 0
Step 1Step 1.. Select Select columncolumn q q with with rrqq > 0. > 0.
((xxqq enters the basis)enters the basis)
q : rq : rqq = max r = max rii
Step 2.Step 2. Select Select pivot elementpivot element p p in column in column q q
Which minimizes ratio: Which minimizes ratio:
p :p : = min for positive = min for positive yyiqiq
yiq
yi0
ypq
yp0
yiq
yi0
i
Step 3Step 3.. PivotPivot on element on element yypqpq and update the and update the
tableau (update all rows, including the tableau (update all rows, including the top one)top one)
• In row In row pp divide divide yypjpj by by yypqpq , , j = 0, 1,…, nj = 0, 1,…, n
• Subtract from each row Subtract from each row i i p p
•• yypjpj , , j = 0, 1,…, nj = 0, 1,…, n yp q
yi q
Pivot on Pivot on yy1 11 1
j =j = 1 2 3 4 5 0
rrjj 100 80 0 0 0 0
yy1j1j 10 5 1 0 0 50
yy2j2j 5 5 0 1 0 35
yy3j3j 5 15 0 0 1 80
yi1 yi2 yi3 yi4 yi5 yi0
Update the tableUpdate the tablej =j = 1 2 3 4 5 0
rrjj 0 30 -10 0 0 -500
yy1j1j 1 0.5 0.1 0 0 5
yy2j2j 0 2.5 -0.5 1 0 10
yy3j3j 0 12.5 -0.5 0 1 55
yi1 yi2 yi3 yi4 yi5 yi0
Point P0 (0,0)F(0,0,50,35,80) = 0
Point P1 (5,0)F(5,0,0,10,55) = 500
Pivot on Pivot on yy2 22 2
Update the tableUpdate the tablej =j = 1 2 3 4 5 0
rrjj 0 0 -4 -12 0 -620
yy1j1j 1 0 0.2 -0.2 0 3
yy2j2j 0 1 -0.2 0.4 0 4
yy3j3j 0 0 2 -5 1 5
yi1 yi2 yi3 yi4 yi5 yi0
Point P1 (3,4)F(3,4,0,0,5) = 620
j =j = 1 2 3 4 5 0
rrjj 0 30 -10 0 0 -500
yy1j1j 1 0.5 1 0 0 5
yy2j2j 0 2.5 -0.5 1 0 10
yy3j3j 0 12.5 -0.5 0 1 55
yi1 yi2 yi3 yi4 yi5 yi0
Point P1 (5,0)F(5,0,0,10,55) = 500
LP ModificationsLP Modifications
1. Maximization1. Maximization
2. Unconstrained x2. Unconstrained x
max c max c t t x = min (-c x = min (-c t t x)x)
let x = xlet x = x++ - x - x -- ; x; x++, x , x -- 0) 0)
min (c min (c t t xx++ - c - c t t x x --))
[A, -A] = bxx++
x x --
xx++, x , x -- 0 0
s. tos. to
3. Constraint set:3. Constraint set: Ax Ax b b
Slack variablesSlack variables,, xxss : A : Axx + x + xss = b = b
[A, I ] = bxx
xxss
x , xx , xs s 0 0
min [cT , 0]
xx
xxss
Columns corresponding to Columns corresponding to xxss form a basic form a basic solutionsolution
4. Constraint set:4. Constraint set: Ax Ax b b
Surplus variablesSurplus variables,, xxpp : A : Ax x - x - xpp = b = b
[A, -I ] = bxx
xxpp
x , xx , xp p 0 0
min [cT , 0]
xx
xxpp
Artificial variablesArtificial variables,, xxaa : A: Axx - x - xpp + x + xaa = b = b
(cannot star with negative basis of(cannot star with negative basis of xxp p ))
[A, -I , I ] = b
x , xx , xpp , , xxaa 0 0
min [cT , 0 , k I ]
xxxxpp
xxaa
xxxxpp
xxaaS. to
Columns corresponding to Columns corresponding to xxaa form a basic form a basic solutionsolution
Integer ProgrammingInteger Programming
Integer ProgramInteger Program
max cmax cTTxx
Ax Ax b b
xxii integer integer
Typically, Typically, xxii = 0,1 = 0,1
(0,1 Integer Programming)(0,1 Integer Programming)
s. to
IP ExampleIP Example: : matching problemmatching problem
Given a graph Given a graph G G , find maximal set of edges in , find maximal set of edges in G, G, such that no two edges are adjacent to the same such that no two edges are adjacent to the same vertex.vertex.• Maximum matching: matching of maximum Maximum matching: matching of maximum
cardinality.cardinality.
• Weighted matching: matching with Weighted matching: matching with
)(max iew
1 3
2e1 e2
e3
otherwise
matchinginincludedisif
0
e1x i
i
1 3
2e1 e2
e3
321 xxx max
10x
1xx
1xx
1xx
tos
i
32
31
21
,
.
In matrix form:In matrix form:
10x
bAxtos
i ,.
xcTmax
110
101
011
e
e
e
e.g.
3
2
1
A
Where Where b = b = [ 1,…1][ 1,…1]T T , , A = A = incidence matrix of incidence matrix of GG
1 2 3
max max 1x1x11 + + 1x1x22 + + 1x1x33
1
1
1
110
101
011
s.to
3
2
1
x
x
x
One solution to this IP problem:One solution to this IP problem:
xx11 , , xx22 , , xx33 = 0, 1 = 0, 1
xx11 = 1 , x= 1 , x22 = 0 , x= 0 , x33 = 0= 0
Other possible solutions:Other possible solutions:
xx11 = 0 , x= 0 , x22 = 1 , x= 1 , x33 = 0, or x= 0, or x11 = 0 , x= 0 , x22 = 0 , x= 0 , x33 = = 1 1
QuestionQuestion
Can the solution to this IP problem be obtained Can the solution to this IP problem be obtained by dropping the integrality constraint:by dropping the integrality constraint:
xxii = 0, 1 = 0, 1
And solving the LP problem instead?And solving the LP problem instead?
In our example, solution to the IP is In our example, solution to the IP is notnot obtainable from LP.obtainable from LP.
ReasonReason:: matrix matrix A A does not have certain property does not have certain property (total unimodularity) needed to guarantee integer (total unimodularity) needed to guarantee integer solutions.solutions.
If If xxii = 0, 1 = 0, 1 is relaxed, such that is relaxed, such that xxii 0 , 0 , then then the solution to the associated LP problem is non – the solution to the associated LP problem is non – integer:integer:
2
1xxx 321
Reason: the Reason: the A A matrix matrix
110
101
011
Is not Is not totally unimodulartotally unimodular: |: |A| = - 2A| = - 2
Unimodularity: LP -> IPUnimodularity: LP -> IP
Given a constraint set in standard formGiven a constraint set in standard form
where where A, b A, b are integerare integer
Partition Partition A = A = [[B/N B/N ]; ]; x = x = [[xxBB, x, xN N ]]
B B is nonsingular is nonsingular m m xx m m basis, basis, NN is non-basic is non-basic
0x
bAx
i
bNxBxx
xNBAx NB
N
B
/
Basic solution is Basic solution is
bBx
0x1
B
N
In particular, when In particular, when B = I B = I and and B B -1-1 = = I thenI then
XXBB = = b b
a solution can be obtained by inspection (as in a solution can be obtained by inspection (as in the initial step of Simplex method).the initial step of Simplex method).
Since Since xxBB = B = B –1 –1 bb with with xxNN = 0, b = 0, b integerinteger
A sufficient condition for a basic solution A sufficient condition for a basic solution xxBB to be to be integer is thatinteger is that
B B –1 –1 be an integer matrixbe an integer matrix
UnimodularityUnimodularity
A square matrix A square matrix B B is called is called unimodularunimodular if if
D = |det B| = 1D = |det B| = 1
An integer matrix An integer matrix A A is is totally unimodulartotally unimodular if every if every square, nonsingular submatrix of square, nonsingular submatrix of A A is unimodular.is unimodular.
Equivalently: Equivalently: A A is totally unimodular if every is totally unimodular if every subdeterminant of subdeterminant of A A is 0, +1, or –1.is 0, +1, or –1.
Recall that for Recall that for B B nonsingular: nonsingular: Bted
BB 1
Where Where B B ++ , , adjoint matrix adjoint matrix
= [ = [ i j i j cofactor of element cofactor of element a a i ji j in in det det A A ]]TT
B B ++ and and det B det B are integer if are integer if BB –1 –1 is integer is integer
333231
232221
131211
bbb
bbb
bbb
B
T
211222111321231122132312
311232113113331113323312
312232213123332132233322
bbbbbbbbbbbb
bbbbbbbbbbbb
bbbbbbbbbbbb
B
)()()(
)()()(
)()()(
Cofactor of Cofactor of aai j i j :: Determinant obtained by Determinant obtained by omitting the iomitting the ithth row and the j row and the jthth column of column of AA and and then multiplying by (-1) then multiplying by (-1) i + ji + j . .
For For B B unimodularunimodular, , B B –1–1 integerinteger::
If If A A is is totally unimodulartotally unimodular, every basis matrix , every basis matrix BB is is unimodular and every basic solutionunimodular and every basic solution
( x( xBB , , yyNN) = ( B ) = ( B –1–1 b, 0 ) b, 0 )
Is Is integerinteger..
In particular, the optimal solution is integerIn particular, the optimal solution is integer
Theorem 1Theorem 1::
If If A A is totally unimodular then every basic is totally unimodular then every basic solution of solution of Ax = b Ax = b is is integerinteger..
For LP’s with equality constraints total For LP’s with equality constraints total unimodularity is sufficient but not necessary.unimodularity is sufficient but not necessary.
For LP with inequality constraints, For LP with inequality constraints, Ax Ax b b, , total unimodularity of total unimodularity of AA is both is both necessary necessary and sufficient and sufficient for all extreme points of for all extreme points of s = {x : Ax s = {x : Ax b , x b , x 0 } 0 } to be integer for to be integer for every integer vector every integer vector b.b.
ConclusionConclusion::
Any IP with totally unimodular constraint Any IP with totally unimodular constraint matrix can be solved as an LP.matrix can be solved as an LP.
Totally unimodular matrix : Totally unimodular matrix : a a i j i j = 0 , +1, -1 = 0 , +1, -1
Also, every determinant of Also, every determinant of A A must be must be 0, +1 or 0, +1 or -1-1
Special Linear Programs with Integer Special Linear Programs with Integer SolutionsSolutions
are LP with the property that they possess are LP with the property that they possess optimal solutions in integers.optimal solutions in integers.
• max flowmax flow
• min – cost flowmin – cost flow
• assignment problemassignment problem
• shortest pathshortest path
• transportation problemtransportation problem
Network flow problems:Network flow problems:
Some Totally Unimodular Linear ProgramsSome Totally Unimodular Linear Programs
Assignment problemAssignment problem(special case of min – cost capacitated flow (special case of min – cost capacitated flow problems)problems)
[ [ m m jobs x jobs x m m men ]men ]
otherwise
jobtoassignedismanif
0
ji1x ji
c i j = cost of assigning man i to job j
ijij
ij xcMax
s. to.s. to.
10,
)1(1,1
)1(1,1
1
1
ij
ij
m
j
ij
m
i
x
manperjobmix
jobpermanmjx
1
1
1
1
1
1
X
111
1
1
1
111
1
1
1
111
1
1
1
X11 X12 X13 X21 X22 X23 X31 X32 X33
xcTmaxIn matrix notation:In matrix notation:
Ax = 1 Ax = 1 wherewhere
11
0
1
1
0
11
0
1
1
0
11
1
1
A
m 2m m 2...
Exactly Exactly m m 1’s in each row1’s in each row
Exactly 2 1’s in each columnExactly 2 1’s in each column
Network Flow ProblemsNetwork Flow Problems::
• a a set of originsset of origins VV11 ; each origin i ; each origin i V V11 ; ; supplies asupplies a11 of of
commodity.commodity.
• a a set of destinationsset of destinations VV22 ; ; each destinationeach destination j j V V22 ; ; has a has a
demand demand bbii of commodity.of commodity.
• costcost per unit commodity; per unit commodity; ccijij associated with sending associated with sending
commodity through (commodity through (i, j i, j ).).
Given:Given:
Constraint SetConstraint Set: : (Totally unimodular)(Totally unimodular)
jiji
2i
1
1i
iΓij
ji
iΓji
ij
cx0
iΓVib
Vi0
iΓVia
xx
))((for
for
))((for
)(),()(),(
nodendestinatiofor
nodesernalintfor
nodesourcefor
)(),()(),( f
0
f
xxiΓij
ji
iΓji
ij
Special case: for single source, single Special case: for single source, single destination, max flow problemdestination, max flow problem
in any case:in any case: Ax Ax b b
No matter which problem it No matter which problem it isis
cutmin
flowmax
flowtcosmin
the general format is the general format is A’x A’x b b
It can be shown thatIt can be shown that• If If A A is totally unimodular then [is totally unimodular then [A/I A/I ] is also totally ] is also totally
unimodularunimodular• The transpose of totally unimodular matrix is also The transpose of totally unimodular matrix is also
totally unimodulartotally unimodular
• ..
I
AA'
where where A = A = incidence matrix of the corresponding incidence matrix of the corresponding digraph, totally unimodular. digraph, totally unimodular. I = I = identity matrix, identity matrix, A’ A’ is is totally unimodulartotally unimodular..
• Exploit a Exploit a special structurespecial structure of the problem of the problem (total unimodularity, etc.) to obtain integer (total unimodularity, etc.) to obtain integer solutions by solving simpler problems.solutions by solving simpler problems.
• TransformTransform the problem to another problem the problem to another problem for which an approximate solution is easier for which an approximate solution is easier to find.to find.
• Once the structure of the problem is well Once the structure of the problem is well understood, understood, use heuristicuse heuristic, but… stay away , but… stay away from brute force approach.from brute force approach.
No efficient algorithms exist for general Integer No efficient algorithms exist for general Integer Programming problemsProgramming problems
• Graph TheoryGraph Theory
• Mathematical OptimizationMathematical Optimization
A large number of CAD problems can be cast in A large number of CAD problems can be cast in analytical form:analytical form:
For some problems – efficient algorithms exist.For some problems – efficient algorithms exist.
For others – need to resort to For others – need to resort to heuristicheuristic, , suboptimal solutions.suboptimal solutions.
Some known successful heuristic approachesSome known successful heuristic approaches
• Simulated annealingSimulated annealing• LP roundingLP rounding
ConclusionsConclusions