Linear Programming Piyush Kumar Welcome to COT 5405.
-
Upload
gillian-mccoy -
Category
Documents
-
view
220 -
download
0
Transcript of Linear Programming Piyush Kumar Welcome to COT 5405.
![Page 1: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/1.jpg)
Linear Programming
Piyush Kumar
Welcome to COT 5405
![Page 2: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/2.jpg)
Optimization
![Page 3: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/3.jpg)
For example
Min
subject to
0
Tc x
Ax b
x
This is what is known as a standard linear program.
![Page 4: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/4.jpg)
Linear Programming
Significance A lot of problems can be converted to LP
formulationo Perceptrons (learning), Shortest path, max flow, MST,
matching, …
Accounts for major proportion of all scientific computations Helps in finding quick and dirty solutions to NP-hard optimization problems Both optimal (B&B) and approximate (rounding)
![Page 5: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/5.jpg)
Graphing 2-Dimensional LPs
Example 1:
x30 1 2
y
0
1
2
4
3
Feasible Region
x 0 y 0
x + 2 y 2
y 4
x 3
Subject to:
Maximize x + y
Optimal Solution
These LP animations were created by Keely Crowston.
![Page 6: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/6.jpg)
Graphing 2-Dimensional LPs
Example 2:
Feasible Region
x 0 y 0
-2 x + 2 y 4
x 3
Subject to:
Minimize ** x - y
Multiple Optimal
Solutions!4
1
x31 2
y
0
2
0
3
1/3 x + y 4
![Page 7: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/7.jpg)
Graphing 2-Dimensional LPs
Example 3:
Feasible Region
x 0y 0
x + y 20
x 5
-2 x + 5 y 150
Subject to:
Minimize x + 1/3 y
Optimal Solution
x
3010 20
y
0
10
20
40
0
30
40
![Page 8: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/8.jpg)
y
x0
1
2
3
4
0 1 2
3
x3010 20
y
0
10
20
40
0
30
40
Do We Notice Anything From These 3 Examples?
x
y
0
1
2
3
4
0 1 2
3
Extreme point
![Page 9: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/9.jpg)
A Fundamental Point
If an optimal solution exists, there is always a corner point optimal solution!
y
x0
1
2
3
4
0 1 2
3
x3010 20
y
0
10
20
40
0
30
40x
y
0
1
2
3
4
0 1 2
3
![Page 10: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/10.jpg)
Graphing 2-Dimensional LPs
Example 1:
x30 1 2
y
0
1
2
4
3
Feasible Region
x 0y 0
x + 2 y 2
y 4
x 3
Subject to:
Maximize x + y
Optimal Solution
Initial Corner pt.
Second Corner pt.
![Page 11: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/11.jpg)
And We Can Extend this to Higher Dimensions
![Page 12: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/12.jpg)
Then How Might We Solve an LP?
The constraints of an LP give rise to a geometrical shape - we call it a polyhedron.
If we can determine all the corner points of the polyhedron, then we can calculate the objective value at these points and take the best one as our optimal solution.
The Simplex Method intelligently moves from corner to corner until it can prove that it has found the optimal solution.
![Page 13: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/13.jpg)
But an Integer Program is Different
x
y
0
1
2
3
4
0 1 2
3
Feasible region is a set of discrete points.
Can’t be assured a corner point solution.
There are no “efficient” ways to solve an IP.
Solving it as an LP provides a relaxation and a bound on the solution.
![Page 14: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/14.jpg)
Linear Programs in higher dimensions
minimize z = 7x1 + x2 + 5x3
subject to x1 - x2 + 3x3 >= 10
5x1 + 2x2 - x3 >= 6
x1, x2, x3 0
What happens at (2,1,3)?What does it tell us about z* = optimal value of z?
![Page 15: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/15.jpg)
LP Upper bounds
Any feasible solution to LP gives an upper bound on z*
So now we know z* <= 30. How do we construct a lower bound? z* >= 16? [Y/N]?
![Page 16: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/16.jpg)
Lower bounding an LP
7x1+x2+5x3 >= (x1-x2+3x3) + (5x1+2x2-x3)
>= 16
Find suitable multipliers ( >0 ?) to construct lower bounds. How do we choose the multipliers?
![Page 17: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/17.jpg)
The Dual
maximize z’ = 10y1 + 6y2
subject to y1 + 5y2 <= 7
-y1 + 2y2 <= 1
3y1 – y2 <= 5
y1, y2 0
What is the dual of a dual? Every feasible solution of the dual gives a lower bound on z*
![Page 18: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/18.jpg)
The Primal
minimize z = 7x1 + x2 + 5x3
subject to x1 - x2 + 3x3 >= 10
5x1 + 2x2 - x3 >= 6
x1, x2, x3 0
Every feasible solution of the primal is an upper bound on the solution to the dual.
![Page 19: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/19.jpg)
Primal – Dual picture
0 Z*
DualSolutions
Primal Solutions
Strong OptimalityPrimal = Dual at opt
![Page 20: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/20.jpg)
Duality
A variable in the dual is paired with a constraint in the primal Objective function of the dual is determined by the right hand side of the primal constraints The constraint matrix of the dual is the transpose of the constraint matrix in the primal.
![Page 21: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/21.jpg)
Duality PropertiesSome relationships between the primal and dual problems:
1. If one problem has feasible solutions and a bounded objective function (and so has an optimal solution), then so does the other problem, so both the weak and the strong duality properties are applicable
2. If the optimal value of the primal is unbounded then the dual is infeasible.
3. If the optimal value of the dual is unbounded then the primal is infeasible.
![Page 22: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/22.jpg)
In Matrix terms
Min
subject to
0
Tc x
Ax b
x
1 1, ,A c xmxn nx nx
![Page 23: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/23.jpg)
LP Geometry
Forms a n dimensional polyhedron
Is convex : If z1 and z2 are two feasible solutions then λz1+ (1- λ)z2 is also feasible.Extreme points can not be written as a convex combination of two feasible points.
![Page 24: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/24.jpg)
LP Geometry
The normals to the halfspaces defining the polyhedron are formed by the coefficents of the constraints. Rows of A form the normals to the hyperplanes defining the primal LP pointing inside the polyhedron.
![Page 25: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/25.jpg)
LP Geometry
Extreme point theorem: If there exists an optimal solution to an LP Problem, then there exists one extreme point where the optimum is achieved. Local optimum = Global Optimum
![Page 26: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/26.jpg)
LP: AlgorithmsSimplex. (Dantzig 1947) Developed shortly after WWII in response to logistical
problems:used for 1948 Berlin airlift.
Practical solution method that moves from one extreme point to a neighboring extreme point.
Finite (exponential) complexity, but no polynomial implementation known.
Courtesy Kevin Wayne
![Page 27: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/27.jpg)
LP: Polynomial Algorithms
Ellipsoid. (Khachian 1979, 1980) Solvable in polynomial time: O(n4 L) bit operations.
o n = # variables o L = # bits in input
Theoretical tour de force. Not remotely practical.
Karmarkar's algorithm. (Karmarkar 1984) O(n3.5 L). Polynomial and reasonably efficient
implementations possible.
Interior point algorithms. O(n3 L). Competitive with simplex!
o Dominates on simplex for large problems. Extends to even more general problems.
![Page 28: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/28.jpg)
Ellipsoid Method
Courtesy S. Boyd
![Page 29: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/29.jpg)
Barrier Algorithms
Simplex solution path
Barrier central path
o Predictor
o Corrector
Optimum
Interior Point Methods
![Page 30: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/30.jpg)
Back to LP Basics
![Page 31: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/31.jpg)
Standard form of LP
Min
subject to
0
Tc x
Ax b
x
1 1, ,A c xmxn nx nx
![Page 32: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/32.jpg)
Standard form of the Dual
Max b
subject to
0
T
T
y
A y c
y
1 1, ,A c xmxn nx nx
![Page 33: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/33.jpg)
Weak Duality
( )T T T T T Tb y Ax y x A y x c c x
We will not prove strong duality in this classbut assume it.
![Page 34: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/34.jpg)
Complementary solutions
For any primal feasible (but suboptimal) x, its complementary solution y is dual infeasible, with cx=ybFor any primal optimal x*, its complementary solution y* is dual optimal, with cx*=y*b=z*
Duality Gap = cx-yb
![Page 35: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/35.jpg)
Complementary slackness
x*, y* are feasible, then they are optimal for (P) and (D) iff
For I = 1..m if yi* > 0
oThen aix* = bi
For J = 1..n if xj* > 0
oThen y*Aj = ci
ai are rows of A and Aj are the columns of A
![Page 36: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/36.jpg)
Complementary slackness
x*, y* are simultaneously optimal for (P) and (D) iff y*(Ax* - b) = 0 (y*A – c)x* = 0
Summary: If a variable is positive, its dual constraint is tightOr if a constraint is loose its dual variable is zero.
![Page 37: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/37.jpg)
Complementary Slackness
Proof? y*(Ax* - b) - (y*A – c)x*
= y*Ax* - y*b - y*Ax* + cx*
= cx* - y*b = 0( But all terms are non-negative )Hence all must be zero!
![Page 38: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/38.jpg)
Primal-Dual Algorithms
Find a feasible solution for both P and D. Try to satisfy the complementary slackness conditions.
![Page 39: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/39.jpg)
Algorithm Design Techniques
LP Relaxation Rounding
oRound the fractional solution obtained by solving LP-relaxation.
oRuns fast Primal Dual Schema
o(iteratively constructs primal n dual solutions)
![Page 40: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/40.jpg)
objective
Linear Program
LP optimum
feasible solutions
y
x
![Page 41: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/41.jpg)
objective
IP optimum
Integer Program
rounding down optimum of LP relaxation
feasible solutions =
y
x
optimum ofLP relaxation
![Page 42: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/42.jpg)
Linear Relaxations
What happens if the optimal of a LP-Relaxation is Integral? There are a class of IPs for which this is guaranteed to happen Transportation problems MaxFlow problems In general (Unimodularity) … Exact
Relaxation
![Page 43: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/43.jpg)
Lower Bounds
Assume minimization problemAny relaxation of the original IP has a _____________ optimal objective function value than the optimal objective function value of the original IP
z*relaxation z*
z*relaxation is called a __________________ on z*
Difference between these two values is called the relaxation gap
![Page 44: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/44.jpg)
Upper Bounds
Any feasible solution to the original IP has a _____________ objective function value than the optimal objective function value of the original IP
zfeasible z*
zfeasible is called an __________________ on z*
Heuristic techniques can be used to find “good” feasible solutions Efficient, may be beneficial if optimality can be
sacrificed Usually application- or problem-specific
![Page 45: Linear Programming Piyush Kumar Welcome to COT 5405.](https://reader038.fdocuments.in/reader038/viewer/2022110209/56649e315503460f94b2215e/html5/thumbnails/45.jpg)
Vertex Cover
Introduction to LP Rounding A simple 2-approximation using LP Better than 2-factor approx?