The Ellipsoid Algorithmeaton.math.rpi.edu/faculty/Mitchell/courses/matp... · Introduction The...
Transcript of The Ellipsoid Algorithmeaton.math.rpi.edu/faculty/Mitchell/courses/matp... · Introduction The...
The Ellipsoid Algorithm
John E. Mitchell
Department of Mathematical SciencesRPI, Troy, NY 12180 USA
February 2018
Mitchell The Ellipsoid Algorithm 1 / 28
9
Introduction
Outline
1 Introduction
2 Assumptions
3 Each iteration
4 Complexity analysis
5 Separation and optimization
6 Relevance to integer optimization
7 NP-Hard problems
Mitchell The Ellipsoid Algorithm 2 / 28
Introduction
Introduction
The simplex algorithm requires exponentially many iterations in theworst case for known pivot rules.
In contrast, the ellipsoid algorithm requires a polynomial number ofiterations.
The ellipsoid algorithm was developed in the 1970s in the SovietUnion, building on earlier Soviet work on nonlinear programming.
Mitchell The Ellipsoid Algorithm 3 / 28
Introduction
New York Times, November 7, 1979
Mitchell The Ellipsoid Algorithm 4 / 28
Introduction
New York Times, November 7, 1979
Published: November 7, 1979Copyright © The New York Times
Mitchell The Ellipsoid Algorithm 5 / 28
Introduction
New York Times, November 7, 1979
Published: November 7, 1979Copyright © The New York TimesMitchell The Ellipsoid Algorithm 6 / 28
Introduction
Find a feasible solution
The algorithm is a high-dimensional version of binary search.
It can be stated as an algorithm to determine whether a feasiblesolution exists to a system of inequalities:
Does there exist x 2 IRn satisfying aT
ix ci for i = 1, . . . ,m?
We letP := {x 2 IR
n : aT
i x ci for i = 1, . . . ,m}.
Mitchell The Ellipsoid Algorithm 7 / 28
Assumptions
Outline
1 Introduction
2 Assumptions
3 Each iteration
4 Complexity analysis
5 Separation and optimization
6 Relevance to integer optimization
7 NP-Hard problems
Mitchell The Ellipsoid Algorithm 8 / 28
Assumptions
Assumptions
We need to make two assumptions:Assumption 1: If P is nonempty then it includes points
contained in a ball of radius R centered at the origin.
Assumption 1 can be justified through the use of Cramer’s Rule, whichlets us bound the norm of any extreme point of P.
Mitchell The Ellipsoid Algorithm 9 / 28
1 % 4
Assumptions
Assumption 2
Assumption 2: If P is nonempty then it contains a ball of
radius r .
Assumption 2 is like a nondegeneracy assumption.
Cramer’s Rule also lets us place a lower bound on the distancebetween two extreme points.
Technically, we can always make Assumption 2, because the size ofthe numbers in the parameters can be used to define anotherparameter � and then each ci can be increased by � in such a way that
the new system has a solution if and only if the old system has asolution, andthe new system contains a ball of radius r if it is nonempty.
Mitchell The Ellipsoid Algorithm 10 / 28
radius r X p
Assumptions
Assumption 2
Assumption 2: If P is nonempty then it contains a ball of
radius r .
Assumption 2 is like a nondegeneracy assumption.
Cramer’s Rule also lets us place a lower bound on the distancebetween two extreme points.
Technically, we can always make Assumption 2, because the size ofthe numbers in the parameters can be used to define anotherparameter � and then each ci can be increased by � in such a way that
the new system has a solution if and only if the old system has asolution, andthe new system contains a ball of radius r if it is nonempty.
Mitchell The Ellipsoid Algorithm 10 / 28
Assumptions
Assumption 2
Assumption 2: If P is nonempty then it contains a ball of
radius r .
Assumption 2 is like a nondegeneracy assumption.
Cramer’s Rule also lets us place a lower bound on the distancebetween two extreme points.
Technically, we can always make Assumption 2, because the size ofthe numbers in the parameters can be used to define anotherparameter � and then each ci can be increased by � in such a way that
the new system has a solution if and only if the old system has asolution, andthe new system contains a ball of radius r if it is nonempty.
Mitchell The Ellipsoid Algorithm 10 / 28
Each iteration
Outline
1 Introduction
2 Assumptions
3 Each iteration
4 Complexity analysis
5 Separation and optimization
6 Relevance to integer optimization
7 NP-Hard problems
Mitchell The Ellipsoid Algorithm 11 / 28
Each iteration
Each iterationAt each iteration we have an ellipsoid which contains (part of) P, if P isnonempty.
We check the center x̄ of the ellipsoid: does it satisfy
aT
i x̄ ci for i = 1, . . . ,m?
If YES: we are done.
Otherwise, we have a violated constraint aT
ix cj .
We define a new ellipsoid that contains the intersection of the oldellipsoid with the halfspace {x 2 IRn : aT
ix aT
ix̄}.
If n = 1 then this would divide our search space in half.
In IRn, we don’t get such a good improvement. Nonetheless, the size ofthe ellipsoid shrinks enough that we get convergence in a number ofiterations polynomial in n.
Mitchell The Ellipsoid Algorithm 12 / 28
Each iteration
Illustration: trying to find a point in P
Ek
xk
P
Mitchell The Ellipsoid Algorithm 13 / 28
Each iteration
Illustration: trying to find a point in P
Ek
xk
aT
ix = aT
ixk
P
Mitchell The Ellipsoid Algorithm 13 / 28
Each iteration
Illustration: trying to find a point in P
Ek
xk
aT
ix = aT
ixk
P
Ek+1
xk+1
Mitchell The Ellipsoid Algorithm 13 / 28
E "={ xER" :( x-x "TY!(x-xMeI }A
positivedefinite,
symmetricVolume:proportional
t odeft(Mu)
Each iteration
Illustration: trying to find a point in P
P
Ek+1
xk+1
Mitchell The Ellipsoid Algorithm 13 / 28
Each iteration
Complexity analysis
Initial ellipsoid: Ball of radius R centered at the origin.
How many “Noes” are enough?
If the ellipsoid is too small to contain a ball of radius r then P = ;.The algorithm updates from an ellipsoid E to an ellipsoid E 0.It can be shown that the volumes of these ellipsoids are related as:
Vol(E 0) e�1/(2(n+1))Vol(E).
Mitchell The Ellipsoid Algorithm 14 / 28
Complexity analysis
Outline
1 Introduction
2 Assumptions
3 Each iteration
4 Complexity analysis
5 Separation and optimization
6 Relevance to integer optimization
7 NP-Hard problems
Mitchell The Ellipsoid Algorithm 15 / 28
Complexity analysis
How many iterations?
How many “Noes” to go from a ball of radius R to a ball of radius r?Let v and V denote the volumes of balls of radius r and R,respectively. Let E0 be the initial ellipsoid, let Ek be the ellipsoid afterk iterations. We want the smallest value of k so that
Vol(Ek ) v .
We also have
Vol(Ek ) he�1/(2(n+1))
ik
V = e�k/(2(n+1))
V .
Thus it suffices to choose k sufficiently large that
e�k/(2(n+1))
V v .
Mitchell The Ellipsoid Algorithm 16 / 28
Complexity analysis
How many iterations, part 2
It suffices to choose k sufficiently large that
e�k/(2(n+1))
V v .
Equivalently, after dividing by V and taking logs of both sides, werequire
�k/(2(n + 1)) ln⇣
v
V
⌘,
which can be restated as requiring
k � 2(n + 1) ln✓
V
v
◆.
The ratio V/v can be chosen so that ln(V/v) is O(n3), so we have analgorithm that requires O(n4) iterations.
Mitchell The Ellipsoid Algorithm 17 / 28
Complexity analysis
Work per iteration
In exact arithmetic, would need to work with square roots.
Can show that it is OK to approximate the square roots and still havethe volume of the ellipsoid decrease at almost the same rate.
Mitchell The Ellipsoid Algorithm 18 / 28
Complexity analysis
Finding optimal solutions to linear programs
If feasible, cut on the objective function.
So restrict attention to the halfspace of points at least as good as thecurrent center.
Mitchell The Ellipsoid Algorithm 19 / 28
Separation and optimization
Outline
1 Introduction
2 Assumptions
3 Each iteration
4 Complexity analysis
5 Separation and optimization
6 Relevance to integer optimization
7 NP-Hard problems
Mitchell The Ellipsoid Algorithm 20 / 28
Separation and optimization
Convex feasibility problems
DefinitionGiven a convex set C ✓ IRn and a point x̄ 2 IRn, the separation problemis to determine whether x̄ 2 C. If the answer is NO, a separatinghyperplane aT x = c is determined, where a 2 IRn, c 2 IR, aT x c for
all x 2 C, and aT x̄ > c.
Mitchell The Ellipsoid Algorithm 21 / 28
§• I
a t >a
↳ = Ca Tx< c
Separation and optimization
Equivalence of separation and feasibilty
TheoremAssume the convex set C contains a ball of radius r . If the separation
problem for C can be solved in polynomial time then the feasibility
problem for C can also be solved in polynomial time.
Proof.We use the ellipsoid algorithm. At each iteration, we test whether thecenter xk of the current ellipsoid is in C. If it is not then the separationproblem solution returns a valid constraint for C which we can use toupdate the ellipsoid.
The overall complexity is the product of two polynomial functions,which is polynomial.
Mitchell The Ellipsoid Algorithm 22 / 28
Separation and optimization
Equivalence of separation and feasibilty
TheoremAssume the convex set C contains a ball of radius r . If the separation
problem for C can be solved in polynomial time then the feasibility
problem for C can also be solved in polynomial time.
Proof.We use the ellipsoid algorithm. At each iteration, we test whether thecenter xk of the current ellipsoid is in C. If it is not then the separationproblem solution returns a valid constraint for C which we can use toupdate the ellipsoid.
The overall complexity is the product of two polynomial functions,which is polynomial.
Mitchell The Ellipsoid Algorithm 22 / 28
⇐"⇐÷€€÷i
Separation and optimization
Equivalence of separation and optimizationConsider the optimization problem
minx2IRn cT x
subject to x 2 C ✓ IRn (CP)
where C is a convex set and c 2 IRn.
TheoremThe problem (CP) can be solved in polynomial time if and only if the
separation problem for C can be solved in polynomial time.
This result follows from using the ellipsoid algorithm.We can optimize over C using the ellipsoid algorithm, generatingseparating hyperplanes either:
by solving the separation problem to separate the current centerof the ellipsoid from C, orby cutting on the objective function if x 2 C.
Mitchell The Ellipsoid Algorithm 23 / 28
Separation and optimization
Equivalence of separation and optimizationConsider the optimization problem
minx2IRn cT x
subject to x 2 C ✓ IRn (CP)
where C is a convex set and c 2 IRn.
TheoremThe problem (CP) can be solved in polynomial time if and only if the
separation problem for C can be solved in polynomial time.
This result follows from using the ellipsoid algorithm.We can optimize over C using the ellipsoid algorithm, generatingseparating hyperplanes either:
by solving the separation problem to separate the current centerof the ellipsoid from C, orby cutting on the objective function if x 2 C.
Mitchell The Ellipsoid Algorithm 23 / 28
Separation and optimization
Equivalence of separation and optimizationConsider the optimization problem
minx2IRn cT x
subject to x 2 C ✓ IRn (CP)
where C is a convex set and c 2 IRn.
TheoremThe problem (CP) can be solved in polynomial time if and only if the
separation problem for C can be solved in polynomial time.
This result follows from using the ellipsoid algorithm.We can optimize over C using the ellipsoid algorithm, generatingseparating hyperplanes either:
by solving the separation problem to separate the current centerof the ellipsoid from C, orby cutting on the objective function if x 2 C.
Mitchell The Ellipsoid Algorithm 23 / 28
÷±÷€€÷÷.
Ex<<7×4
Relevance to integer optimization
Outline
1 Introduction
2 Assumptions
3 Each iteration
4 Complexity analysis
5 Separation and optimization
6 Relevance to integer optimization
7 NP-Hard problems
Mitchell The Ellipsoid Algorithm 24 / 28
Relevance to integer optimization
Integer optimization as a linear programConsider the integer optimization problem
minx cT x
subject to x 2 S (IP)x 2 Zn ✓ IRn
for some subset S, and where Z denotes the set of integers. Let C
denote the convex hull of the set of integer points in S, so
C = conv(S \ Zn)
(The convex hull of a set T is the smallest convex set containing T . Aformal definition will be given next week.)
All the extreme points of C are feasible integer points, so we could inprinciple solve (IP) by solving the linear program
minx cT x
subject to x 2 C ✓ IRn (LP)
Mitchell The Ellipsoid Algorithm 25 / 28
.
÷÷%t¥I÷÷÷.
Relevance to integer optimization
“Solving” the linear programThe difficulty with this approach is that it is very hard to get an explicitformulation for C.
If we can solve the separation problem for C in polynomial time thenwe can solve (LP) in polynomial time, which means we can solve (IP)in polynomial time.
Thus, for NP-complete problems, we cannot expect to solve theseparation problem in polynomial time.
Conversely, if we can solve the optimization problem in polynomial timethen we can solve the separation problem in polynomial time.
For example, we can solve the problem of finding a minimum weightperfect matching on a graph G = (V ,E) in polynomial time, and indeedthe separation problem of determining whether a point x is in theconvex hull of the set of perfect matchings can be solved in O(|V |4)time.
Mitchell The Ellipsoid Algorithm 26 / 28
NP-Hard problems
Outline
1 Introduction
2 Assumptions
3 Each iteration
4 Complexity analysis
5 Separation and optimization
6 Relevance to integer optimization
7 NP-Hard problems
Mitchell The Ellipsoid Algorithm 27 / 28
NP-Hard problems
NP-hard problems
A problem is called NP-hard if it at least as hard as an NP-completeproblem, in the sense that the NP-complete problem can bepolynomially reduced to the NP-hard problem.
An NP-hard problem need not be a feasibility problem.
In particular, optimization versions of NP-complete integerprogramming feasibility problems are NP-hard.
Mitchell The Ellipsoid Algorithm 28 / 28