Post on 16-Jan-2016
Paper review of ENGG*6140 Optimization Techniques
Paper ReviewPaper Review
-- Interior-Point Methods-- Interior-Point Methods
for LPfor LP
Yanrong HuYanrong Hu
Ce YuCe Yu
Mar. 11, 2003Mar. 11, 2003
Outline
General introduction The original algorithm description A variant of Karmarkar’s algorithm
- a detailed simple example Another example More issues
Interior-point methods for linear programming:
The most dramatic new development in operations research (linear programming) during the 1980s.
Opened in 1984 by a young mathematician at AT&T Bell Labs, N. Karmarkar.
Polynomial-time algorithm and has great potential for solving huge linear programming problems beyond the reach of the simplex method.
Karmarkar’s method stimulated development in both interior-point and simplex methods.
General introduction
Difference between Interior point methods and the simplex method
The big difference between Interior point methods and the simplex method lies in the nature of trial solutions.
General introduction
Simplex methodInterior-point method
Trial solutions
CPF (Corner Point Feasible) solutions
Interior points (points inside the boundary of the feasible region)
complexity
worst case:# iterations can increase exponentially in the number of variables n:
Polynomial time n2
Karmarkar’s original Algorithm
Karmarkar assumes that the LP is given in Canonical form of the problem:
0 X
1 1X
0 AX s.t.
CX zMin
Assumptions:
0 zmin 2.
0 AX satisfies )1
,...,1
,1
(X 1.
nnn
To apply the algorithm to LP problem in standard form, a transformation is needed.
Min Z = CX
s.t. AX b
X 0
Karmarkar’s original Algorithm
An example of transformation
Min z = y1+y2 (z=cy)
s.t. y1+2y2 2 (AYb)
y1, y2 0
Min z=5x1 + 5x2
s.t. 3x1+8x2+3x3-2x4=0 (AX=0)
x1+ x2+ x3+ x4=1 (1X =1)
xj 0, j=1,2,3,4
from standard form to canonical form
Karmarkar’s original Algorithm
The principal idea:
The algorithm creates a sequence of points having
decreasing values of the objective function. In the step, the point is brought into the center of the simplex by a projective transformation.
)()2()1()0( ,...,,, kxxxx
thk )(kx
The steps of Karmarkar’s algorithm are:
Step 0. start with the solution point
Step k. Define:
)1
,...,1
,1
(0 nnnX
1
...1
k
knkk
ADP
xxdiagD and compute:
newk
newkk
p
pTnew
Tk
TTp
YD
YDX
c
cr
nnnY
CDPPPPIc
1
)1
,...,1
,1
(
)()(
1
1
and compute the step length parameters n
n
nnr
3
)1( and
)1(
1
Karmarkar’s original Algorithm
X is brought to the center by:
XD
XDY
k
k1
1
1
A Variant
The Affine Variant of Karmarkar’s Algorithm:
Concept 1: Shoot through the interiorinterior of the feasible region toward an
optimal solution.
Concept 2: Move in a direction that improves the objective function value
at the fastestfastest feasible rate.
Concept 3: TransformTransform the feasible region to place the current trail solution
near its centercenter, thereby enabling a large improvement when
concept 2 is implemented.
An Example
Max Z= x1+ 2x2
s.t. x1+ x28
x1,x2 0Optimal solution
is (x1,x2)=(0,8)
with Z=16
The Problem:
An Example
The direction is perpendiculars to (and toward) the objective function line. (3,4)=(2,2)+(1,2), where the vector (1,2) is the gradientgradient of the Objective Function. The gradient is the coefficientcoefficient of the objective function.
Concept 1: Shoot through the interior interior of the feasible region toward an optimal solution.
The algorithm begins with an initial solution that lies in the interiorinterior of the feasible region. Arbitrarily choose (x1,x2)=(2,2) to be the initial solution.
Concept 2: Move in a direction that improves the objective function value at the fastest fastest feasible rate.
Max Z= x1+ 2x2
s.t. x1+ x28
x1,x2 0
An Example
The augmented form can be
written in general as:
0
0
0
0 [8],b 1], 1, [1,A ,
x
x ,
0
2
1
3
2
1
x
xc
so
Max Z = cT X
s.t. AX = b
X 0
The algorithm actually operates on the augmented form.
Max Z = x1+ 2x2
s.t. x1+ x2+ x3 = 8
x1,x2,x3 0
letting x3 be the slack Max Z = x1+ 2x2
s.t. x1+ x28
x1,x2 0
Initial solution: (2,2)
Gradient of obj. fn.: (1,2)
(2,2,4)
(1,2,0)
An Example
- Feasible region : the triangle
- Optimum (0,8,0), Initial solution: (2,2,4)
- Gradient of Obj. fn. : cT =[ 1, 2, 0 ]
Show The augmented form graphically Using Projected GradientProjected Gradient to implement Concept 1 & 2:
Adding the gradient to the initial leads to (3,4,4)= (2, 2, 4)+ (1,2,0) (infeasible)
To remain feasible, the algorithm project the point (3,4,4) down onto the feasible triangle.
To do so, the projected gradient – gradient projected onto the feasible region – is used.
Then the next trial solution move in the direction of projected gradient
An Interior-point Algorithm
a formula is available for computing the projected gradient:
projection matrix : P = I-AT(AAT)-1A
projected gradient: cp = Pc
Using Projected GradientProjected Gradient to implement Concept 1 & 2
3
2
3
1
3
13
1
3
2
3
13
1
3
1
3
2
P
1
1
0
pc
now we are ready to move from (2,2,4)
in the direction of cp to a new point:
1
1
0
4
4
2
2
4
4
2
2
pcx
determines how far we move. large too close to the boundary small more iterations
we have chosen =0.5, so the new trial solution move to:
x = ( 2, 3, 3)
An Interior-point Algorithm
Concept 3:
TransformTransform the feasible region to place the current trail solution near its centercenter, thereby enabling a large improvement when concept 2 is implemented.
Centering scheme for implementing Concept 3
Why -- The centering scheme keeps turning the direction of the projected gradient to point more nearly toward an optimal solution as the algorithm converges toward this solution.
How -- Simply changing the scalechanging the scale (units) for each of the variable so that the trail solution becomes equidistant from the constraint boundaries in the new coordinate system.
Define D=diag{x},
xDx 1~ bring x to the center in the new coordinate
)1,1,1(~ x
An Interior-point Algorithm
Initial trial solution
(x1,x2,x3) =(2,2,4)
0
8422 ..
42Max Z
3
~
2
~
1
~
3
~
2
~~
1
2
~~
1
xxx
xxxts
xx
)1,1,1(),,(~
3
~
2
~
1xxx
In this new coordinate system, the problem becomes:
4
2
2
4
100
02
10
002
1
:are variablerescaled The
3
2
1
3
2
1
1~
x
x
x
x
x
x
xDx
Summary and Illustration of the Algorithm
400
020
002
matrix diagonal
ingcorrespond thebe DLet Step1.
D
1
1
1
4
2
2
4
100
02
10
002
1
3
2
1
3
2
1
1~
x
x
x
x
x
x
xDx
Iteration 1. Given the initial solution (x1,x2,x3) = (2,2,4)
nx
x
x
x
D
...000
...............
0...00
0...00
0...00
3
2
1
Step1. given the current trial solution (x1,x2,…,xn),
set
Summary of the General Procedure
X is brought to the center (1,1,1) by:
Summary and Illustration of the Algorithm
0
4
2
]0,2,1[
400
020
002
]4,2,2[
400
020
002
]1,1,1[
become have c andA
s,coordinate new In these step2.
~
~
Dcc
ADA
DccandADA ~~
calculate Step2.
Summary of the General Procedure (cont.)
Iteration 1. (cont.)
To compute projected gradient:
Summary and Illustration of the Algorithm
2
3
1
3
1
3
1
3
13
1
6
5
6
13
1
6
1
6
5
)(-I P
ismatrix projected thestep3.
~
~1-
T~~T~
cPc
isgradientprojectedthe
AAAA
p
~
~1-
T~~T~
)(-I P
matrix projected
theCalculate step3.
cPc
gradientprojectedthe
AAAA
p
Iteration 1. (cont.)
Summary of the General Procedure (cont.)
Compute projected gradient:
Summary and Illustration of the Algorithm
2
14
74
5
2
3
1
2
5.0
1
1
1
1
1
1~
pcv
x
step4. define v as the absolute value of the negative component of cp having
the largest value, so that v=|-2|=2.
In this coordinate, the algorithm moves from the current trial solution to the next one.
step4. Determine how far to move by identify v. Then make
move by calculating
pcvx
1
1
1~
Iteration 1. (cont.)
Summary and Illustration of the Algorithm
22
72
5
2
14
74
5
400
020
002~
3
2
1
xD
x
x
x
this completes the iteration 1. The new solution will be used to start the next iteration.
Step 5. Back to the original coordinate by calculating
~
xDx
Step 5. In the original coordinate, the solution is
Iteration 1. (cont.)Summary of the General Procedure (cont.)
0 1 2 3 4 5 6 7 8 90
1
2
3
4
5
6
7
8
9
x1
x2
(0,8) optimal
Z=16=x1+2x2
(2.5,3.5)
(2,2)
x1+2x2=8
Summary and Illustration of the Algorithm
Iteration 2. Given the current trial solution ,2)2
7,
2
5(x3)x2,(x1,
200
02
70
002
5
Dxx that such
matrix diagonal ingcorrespond thebe Let DStep1.~
D
15
4160
13312
11
45
37
45
14
9
245
14
90
41
18
79
2
18
7
18
13
pcandP
X is brought to the center (1,1,1,) in the new coordinate, and move to (0.83, 1.4, 0.5), corresponds (2.08, 4.92,1.0) in the original coordinate
0 1 2 3 4 5 6 7 8 90
1
2
3
4
5
6
7
8
9
x1
x2
(0,8) optimal
Z=16=x1+2x2
(2.5,3.5)
(2,2)
x1+2x2=8
(2.08, 4.92)
Summary and Illustration of the Algorithm
Iteration 3. Given the current trial solution x=(2.08, 4.92, 1.0)
X is brought to the center (1,1,1,) in the new coordinate, and move to ( 0.54,1.30,0.50), corresponds (1. 13, 6.37, 0.5) in the original coordinate
0.100
092.40
0008.2
D
0 1 2 3 4 5 6 7 8 90
1
2
3
4
5
6
7
8
9
x1
x2
(0,8) optimal
Z=16=x1+2x2
(2.5,3.5)
(2,2)
x1+2x2=8
(2.08, 4.92)
(1.13,6.37)
79.1
05.1
63.1
pc
An Example
Effect of rescaling of each iteration: Sliding the optimal solution toward (1,1,1) while the other BF solutions tend to slide away.
A B
C D
Summary and Illustration of the Algorithm
More iterations. Starting from the current trial solution x, following the steps, x is moving toward the optimum (0,8). When the trial solution is virtually unchanged from the proceeding one, then the algorithm has virtually converged to an optimal solution. So stop.
0 1 2 3 4 5 6 7 8 90
1
2
3
4
5
6
7
8
9
x1
x2
(0,8) optimal
Z=16=x1+2x2
(2.5,3.5)
(2,2)
x1+2x2=8
(2.08, 4.92)
(1.13,6.37)
Another Example
The problem
MAX Z = 5x1 + 4x2
ST 6x1 + 4x2<=24 x1 + 2x2<=6 -x1 + x2<=1 x2<=2 x1, x2>=0
Augmented form
MAX Z = 5x1 + 4x2
ST 6x1 + 4x2 + x3=24 x1 + 2x2 + x4=6
-x1 + x2 + x5=1 x2 + x6=2
xj>=0, j=1,2,3,4,5,6
0 0.5 1 1.5 2 2.5 3 3.5 4 4.50
0.5
1
1.5
2
2.5
x1
(3,1.5) optimal
Z=21=5x1+4x2
0 0.5 1 1.5 2 2.5 3 3.5 4 4.50
0.5
1
1.5
2
2.5
x1
x2
(3,1.5) optimal
Z=21=5x1+4x2
100010
000011
001021
000146
A
2
1
6
24
b
000045c
Another Example
Starting from an initial solution x=(1,1,14,3,1,1)
The trial solution at each iteration
~
~
~
~1-
T~~T~
~
~
1
1
1
)(-I P
diag{x}D
xDx
cv
x
cPc
gradientprojectedthe
AAAA
Dcc
ADA
p
p
0 0.5 1 1.5 2 2.5 3 3.5 4 4.50
0.5
1
1.5
2
2.5
x1
x2
(3,1.5) optimal
Z=21=5x1+4x2
(1,1)
(1.67,1.41)(2.23,1.51)
(3.05,1.39)
More issues
Interior-point methods is designed for dealing with big problems. Although the claim that it’s much faster than the simplex method is controversy, many tests on huge LP problems show its outperformance.
After Karmarkar’s paper, many related methods have been developed to extend the applicability of Karmarkar’s algorithm, e.g.
• Infeasible interior points method -- remove the assumption that there always exits a nonempty interior.
• Methods applying to LP problems in standard form.
More issues
• Methods dealing with finding initial solution, and estimating the optimal solution.
•Methods working with primal-dual problems.
• Studies about moving step-long/short steps.
•Studies about efficient implementation and complexity of various methods.
Karmarkar’s paper not only started the development of interior point methods, but also encouraged rapid improvement of simplex methods.
Reference
[1] N. Karmarkar, 1984, A New Polynomial - Time Algorithm for Linear Programming, Combinatorica 4 (4), 1984, p. 373-395.
[2] M.J. Todd, (1994), Theory and Practice for Interior-point method, ORSA Jounal on Computing 6 (1), 1994, p. 28-31.
[3] I. Lustig, R. Marsten, D. Shanno, (1993), Interior-point Methods for Linear Programming: Computational State of the Art, ORSA Journal on Computing, 6 (1), 1994, p. 1-14.
[4] Hillier,Lieberman,Introduction to Operations Research (7th edition) 320-334
[5] Taha, Operations Research: An Introduction (6th edition) 336-345
[6] E.R. Barnes, 1986, A Variation on Karmarkar’s Algorithm for Sloving Linear Programming problems, Mathematical Programming 36, 1986, p. 174-182.
[7] R.J. Vanderbei, M.S. Meketon and B.A. Freeman, A Modification of karmarkar’s Linear Programming Algorithm, Algorithmica: An International Journal in Computer Science 1 (4), 1986 p. 395 – 407.
[8] D. Gay (1985) A Variant of Karmarkar’s Linear Programming Algorithm for Problems in Standard form, Mathematical Programming 37 (1987) 81-90
more…
Yanrong HuYanrong Hu
Ce YuCe Yu
Mar. 11 2003Mar. 11 2003
Paper Review of ENGG*6140 Optimization Techniques