CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599:...
Transcript of CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599:...
![Page 1: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/1.jpg)
CS599: Convex and Combinatorial OptimizationFall 2013
Lecture 9: Convex Optimization Problems
Instructor: Shaddin Dughmi
![Page 2: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/2.jpg)
Announcements
Homework: Due beginning of next classMust submit a hard copy, unless you have a good excuseIf using late days, due by Monday in Shaddin’s mailbox
Today: Convex Optimization ProblemsRead all of B&V Chapter 4.
![Page 3: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/3.jpg)
Outline
1 Convex Optimization Basics
2 Common Classes
3 Interlude: Positive Semi-Definite Matrices
4 More Convex Optimization Problems
![Page 4: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/4.jpg)
Recall: Convex Optimization ProblemA problem of minimizing a convex function (or maximizing a concavefunction) over a convex set.
minimize f(x)subject to x ∈ X
X ⊆ Rn is convex, and f : Rn → R is convexTerminology: decision variable(s), objective function, feasible set,optimal solution/value, ε-optimal solution/value
Convex Optimization Basics 1/24
![Page 5: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/5.jpg)
Standard Form
Instances typically formulated in the following standard form
minimize f(x)subject to gi(x) ≤ 0, for i ∈ C1.
aᵀi x = bi, for i ∈ C2.
gi is convexTerminology: equality constraints, inequality constraints,active/inactive at x, feasible/infeasible, unbounded
In principle, every convex optimization problem can be formulatedin this form (possibly implicitly)
Recall: every convex set is the intersection of halfspaces
When f(x) is immaterial (say f(x) = 0), we say this is convexfeasibility problem
Convex Optimization Basics 2/24
![Page 6: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/6.jpg)
Standard Form
Instances typically formulated in the following standard form
minimize f(x)subject to gi(x) ≤ 0, for i ∈ C1.
aᵀi x = bi, for i ∈ C2.
gi is convexTerminology: equality constraints, inequality constraints,active/inactive at x, feasible/infeasible, unboundedIn principle, every convex optimization problem can be formulatedin this form (possibly implicitly)
Recall: every convex set is the intersection of halfspaces
When f(x) is immaterial (say f(x) = 0), we say this is convexfeasibility problem
Convex Optimization Basics 2/24
![Page 7: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/7.jpg)
Standard Form
Instances typically formulated in the following standard form
minimize f(x)subject to gi(x) ≤ 0, for i ∈ C1.
aᵀi x = bi, for i ∈ C2.
gi is convexTerminology: equality constraints, inequality constraints,active/inactive at x, feasible/infeasible, unboundedIn principle, every convex optimization problem can be formulatedin this form (possibly implicitly)
Recall: every convex set is the intersection of halfspaces
When f(x) is immaterial (say f(x) = 0), we say this is convexfeasibility problem
Convex Optimization Basics 2/24
![Page 8: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/8.jpg)
Local and Global Optimality
FactFor a convex optimization problem, every locally optimal feasiblesolution is globally optimal.
ProofLet x be locally optimal, and y be any other feasible point.The line segment from x to y is contained in the feasible set.By local optimality f(x) ≤ f(θx+ (1− θ)y) for θ sufficiently closeto 1.Jensen’s inequality then implies that y is suboptimal.
f(x) ≤ f(θx+ (1− θ)y) ≤ θf(x) + (1− θ)f(y)
f(x) ≤ f(y)
Convex Optimization Basics 3/24
![Page 9: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/9.jpg)
Local and Global Optimality
FactFor a convex optimization problem, every locally optimal feasiblesolution is globally optimal.
ProofLet x be locally optimal, and y be any other feasible point.
The line segment from x to y is contained in the feasible set.By local optimality f(x) ≤ f(θx+ (1− θ)y) for θ sufficiently closeto 1.Jensen’s inequality then implies that y is suboptimal.
f(x) ≤ f(θx+ (1− θ)y) ≤ θf(x) + (1− θ)f(y)
f(x) ≤ f(y)
Convex Optimization Basics 3/24
![Page 10: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/10.jpg)
Local and Global Optimality
FactFor a convex optimization problem, every locally optimal feasiblesolution is globally optimal.
ProofLet x be locally optimal, and y be any other feasible point.The line segment from x to y is contained in the feasible set.
By local optimality f(x) ≤ f(θx+ (1− θ)y) for θ sufficiently closeto 1.Jensen’s inequality then implies that y is suboptimal.
f(x) ≤ f(θx+ (1− θ)y) ≤ θf(x) + (1− θ)f(y)
f(x) ≤ f(y)
Convex Optimization Basics 3/24
![Page 11: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/11.jpg)
Local and Global Optimality
FactFor a convex optimization problem, every locally optimal feasiblesolution is globally optimal.
ProofLet x be locally optimal, and y be any other feasible point.The line segment from x to y is contained in the feasible set.By local optimality f(x) ≤ f(θx+ (1− θ)y) for θ sufficiently closeto 1.
Jensen’s inequality then implies that y is suboptimal.
f(x) ≤ f(θx+ (1− θ)y) ≤ θf(x) + (1− θ)f(y)
f(x) ≤ f(y)
Convex Optimization Basics 3/24
![Page 12: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/12.jpg)
Local and Global Optimality
FactFor a convex optimization problem, every locally optimal feasiblesolution is globally optimal.
ProofLet x be locally optimal, and y be any other feasible point.The line segment from x to y is contained in the feasible set.By local optimality f(x) ≤ f(θx+ (1− θ)y) for θ sufficiently closeto 1.Jensen’s inequality then implies that y is suboptimal.
f(x) ≤ f(θx+ (1− θ)y) ≤ θf(x) + (1− θ)f(y)
f(x) ≤ f(y)
Convex Optimization Basics 3/24
![Page 13: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/13.jpg)
RepresentationTypically, by problem we mean a family of instances, each of which isdescribed either explicitly via problem parameters, or given implicitlyvia an oracle, or something in between.
Convex Optimization Basics 4/24
![Page 14: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/14.jpg)
RepresentationTypically, by problem we mean a family of instances, each of which isdescribed either explicitly via problem parameters, or given implicitlyvia an oracle, or something in between.
Explicit RepresentationA family of linear programs of the following form
maximize cTxsubject to Ax � b
x � 0
may be described by c ∈ Rn, A ∈ Rm×n, and b ∈ Rm.
Convex Optimization Basics 4/24
![Page 15: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/15.jpg)
RepresentationTypically, by problem we mean a family of instances, each of which isdescribed either explicitly via problem parameters, or given implicitlyvia an oracle, or something in between.
Oracle RepresentationAt their most abstract, convex optimization problems of the followingform
minimize f(x)subject to x ∈ X
are described via a separation oracle for X and epi f .
Given additional data about instances of the problem, namely a range[L,H] for its optimal value and a ball of volume V containing X , theellipsoid method returns an ε-optimal solution using onlypoly(n, log(H−Lε ), log V ) oracle calls.
Convex Optimization Basics 4/24
![Page 16: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/16.jpg)
RepresentationTypically, by problem we mean a family of instances, each of which isdescribed either explicitly via problem parameters, or given implicitlyvia an oracle, or something in between.
Oracle RepresentationAt their most abstract, convex optimization problems of the followingform
minimize f(x)subject to x ∈ X
are described via a separation oracle for X and epi f .
Given additional data about instances of the problem, namely a range[L,H] for its optimal value and a ball of volume V containing X , theellipsoid method returns an ε-optimal solution using onlypoly(n, log(H−Lε ), log V ) oracle calls.
Convex Optimization Basics 4/24
![Page 17: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/17.jpg)
RepresentationTypically, by problem we mean a family of instances, each of which isdescribed either explicitly via problem parameters, or given implicitlyvia an oracle, or something in between.
In BetweenConsider the following fractional relaxation of the Traveling SalesmanProblem, described by a network (V,E) and distances de on e ∈ E.
min∑
e dexes.t.∑
e∈δ(S) xe ≥ 2, ∀S ⊂ V, S 6= ∅.x � 0
Representation of LP is implicit, in the form of a network. Using thisrepresentation, separation oracles can be implemented efficiently, andused as subroutines in the ellipsoid method.
Convex Optimization Basics 4/24
![Page 18: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/18.jpg)
RepresentationTypically, by problem we mean a family of instances, each of which isdescribed either explicitly via problem parameters, or given implicitlyvia an oracle, or something in between.
In BetweenConsider the following fractional relaxation of the Traveling SalesmanProblem, described by a network (V,E) and distances de on e ∈ E.
min∑
e dexes.t.∑
e∈δ(S) xe ≥ 2, ∀S ⊂ V, S 6= ∅.x � 0
Representation of LP is implicit, in the form of a network. Using thisrepresentation, separation oracles can be implemented efficiently, andused as subroutines in the ellipsoid method.
Convex Optimization Basics 4/24
![Page 19: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/19.jpg)
Equivalence
Next up: we look at some common classes of convex optimizationproblemsTechnically, not all of them will be convex in their naturalrepresentationHowever, we will show that they are “equivalent” to a convexoptimization problem
EquivalenceLoosly speaking, two optimization problems are equivalent if anoptimal solution to one can easily be “translated” into an optimalsolution for the other.
NoteDeciding whether an optimization problem is equivalent to a tractableconvex optimization problem is, in general, a black art honed byexperience. There is no silver bullet.
Convex Optimization Basics 5/24
![Page 20: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/20.jpg)
Equivalence
Next up: we look at some common classes of convex optimizationproblemsTechnically, not all of them will be convex in their naturalrepresentationHowever, we will show that they are “equivalent” to a convexoptimization problem
EquivalenceLoosly speaking, two optimization problems are equivalent if anoptimal solution to one can easily be “translated” into an optimalsolution for the other.
NoteDeciding whether an optimization problem is equivalent to a tractableconvex optimization problem is, in general, a black art honed byexperience. There is no silver bullet.
Convex Optimization Basics 5/24
![Page 21: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/21.jpg)
Equivalence
Next up: we look at some common classes of convex optimizationproblemsTechnically, not all of them will be convex in their naturalrepresentationHowever, we will show that they are “equivalent” to a convexoptimization problem
EquivalenceLoosly speaking, two optimization problems are equivalent if anoptimal solution to one can easily be “translated” into an optimalsolution for the other.
NoteDeciding whether an optimization problem is equivalent to a tractableconvex optimization problem is, in general, a black art honed byexperience. There is no silver bullet.
Convex Optimization Basics 5/24
![Page 22: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/22.jpg)
Outline
1 Convex Optimization Basics
2 Common Classes
3 Interlude: Positive Semi-Definite Matrices
4 More Convex Optimization Problems
![Page 23: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/23.jpg)
Linear Programming
We have already seen linear programming
minimize cᵀxsubject to Ax ≤ b
Common Classes 6/24
![Page 24: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/24.jpg)
Linear Fractional Programming
Generalizes linear programming
minimize cᵀx+deᵀx+f
subject to Ax ≤ beᵀx+ f ≥ 0
The objective is quasiconvex (in fact, quasilinear) over thehalfspace where the denominator is nonnegative.
Can be reformulated as an equivalent linear program
1 Change variables to y = xeᵀx+f and z = 1
eᵀx+f
2 (y, z) is a solution to the above iff eᵀy + fz = 1
minimize cᵀy + dzsubject to Ay ≤ bz
z ≥ 0
eᵀy + fz = 1
Common Classes 7/24
![Page 25: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/25.jpg)
Linear Fractional Programming
Generalizes linear programming
minimize cᵀx+deᵀx+f
subject to Ax ≤ beᵀx+ f ≥ 0
The objective is quasiconvex (in fact, quasilinear) over thehalfspace where the denominator is nonnegative.Can be reformulated as an equivalent linear program
1 Change variables to y = xeᵀx+f and z = 1
eᵀx+f
2 (y, z) is a solution to the above iff eᵀy + fz = 1
minimize cᵀy + dzsubject to Ay ≤ bz
z ≥ 0y = x
eᵀx+f
z = 1eᵀx+f
eᵀy + fz = 1
Common Classes 7/24
![Page 26: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/26.jpg)
Linear Fractional Programming
Generalizes linear programming
minimize cᵀx+deᵀx+f
subject to Ax ≤ beᵀx+ f ≥ 0
The objective is quasiconvex (in fact, quasilinear) over thehalfspace where the denominator is nonnegative.Can be reformulated as an equivalent linear program
1 Change variables to y = xeᵀx+f and z = 1
eᵀx+f
2 (y, z) is a solution to the above iff eᵀy + fz = 1
minimize cᵀy + dzsubject to Ay ≤ bz
z ≥ 0
�����y = xeᵀx+f
�����z = 1
eᵀx+f
eᵀy + fz = 1
Common Classes 7/24
![Page 27: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/27.jpg)
Example: Optimal Production Variant
n products, m raw materialsEvery unit of product j uses aij units of raw material iThere are bi units of material i availableProduct j yields profit cj dollars per unit, and requires aninvestment of ej dollars per unit to produce, with f as a fixed costFacility wants to maximize “Return rate on investment”
maximize cᵀxeᵀx+f
subject to aᵀi x ≤ bi, for i = 1, . . . ,m.xj ≥ 0, for j = 1, . . . , n.
Common Classes 8/24
![Page 28: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/28.jpg)
Geometric Programming
DefinitionA monomial is a function f : Rn+ → R+ of the form
f(x) = cxa11 xa22 . . . xann ,
where c ≥ 0, ai ∈ R.A posynomial is a sum of monomials.
A Geometric Program is an optimization problem of the following form
minimize f0(x)subject to fi(x) ≤ bi, for i ∈ C1.
hi(x) = bi, for i ∈ C2.x � 0
where fi’s are posynomials, hi’s are monomials, and bi > 0 (wlog 1).
InterpretationGP model volume/area minimization problems, subject to constraints.
Common Classes 9/24
![Page 29: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/29.jpg)
Geometric Programming
DefinitionA monomial is a function f : Rn+ → R+ of the form
f(x) = cxa11 xa22 . . . xann ,
where c ≥ 0, ai ∈ R.A posynomial is a sum of monomials.
A Geometric Program is an optimization problem of the following form
minimize f0(x)subject to fi(x) ≤ bi, for i ∈ C1.
hi(x) = bi, for i ∈ C2.x � 0
where fi’s are posynomials, hi’s are monomials, and bi > 0 (wlog 1).
InterpretationGP model volume/area minimization problems, subject to constraints.
Common Classes 9/24
![Page 30: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/30.jpg)
Geometric Programming
DefinitionA monomial is a function f : Rn+ → R+ of the form
f(x) = cxa11 xa22 . . . xann ,
where c ≥ 0, ai ∈ R.A posynomial is a sum of monomials.
A Geometric Program is an optimization problem of the following form
minimize f0(x)subject to fi(x) ≤ bi, for i ∈ C1.
hi(x) = bi, for i ∈ C2.x � 0
where fi’s are posynomials, hi’s are monomials, and bi > 0 (wlog 1).
InterpretationGP model volume/area minimization problems, subject to constraints.
Common Classes 9/24
![Page 31: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/31.jpg)
Example: Designing a Suitcase
A suitcase manufacturer is designing a suitcaseVariables: h, w,dWant to minimize surface area 2(hw + hd+ wd) (i.e. amount ofmaterial used)Have a target volume hwd ≥ 5Practical/aesthetic constraints limit aspect ratio: h/w ≤ 2, h/d ≤ 3Constrained by airline to h+ w + d ≤ 7
minimize 2hw + 2hd+ 2wdsubject to h−1w−1d−1 ≤ 1
5hw−1 ≤ 2hd−1 ≤ 3h+ w + d ≤ 7h,w, d ≥ 0
More interesting applications involve optimal component layout in chipdesign.
Common Classes 10/24
![Page 32: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/32.jpg)
Example: Designing a Suitcase
A suitcase manufacturer is designing a suitcaseVariables: h, w,dWant to minimize surface area 2(hw + hd+ wd) (i.e. amount ofmaterial used)Have a target volume hwd ≥ 5Practical/aesthetic constraints limit aspect ratio: h/w ≤ 2, h/d ≤ 3Constrained by airline to h+ w + d ≤ 7
minimize 2hw + 2hd+ 2wdsubject to h−1w−1d−1 ≤ 1
5hw−1 ≤ 2hd−1 ≤ 3h+ w + d ≤ 7h,w, d ≥ 0
More interesting applications involve optimal component layout in chipdesign.
Common Classes 10/24
![Page 33: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/33.jpg)
Designing a Suitcase in Convex Form
minimize 2hw + 2hd+ 2wdsubject to h−1w−1d−1 ≤ 1
5hw−1 ≤ 2hd−1 ≤ 3h+ w + d ≤ 7h,w, d ≥ 0
Change of variables to h̃ = log h, w̃ = logw, d̃ = log d
minimize 2eh̃+w̃ + 2eh̃+d̃ + 2ew̃+d̃
subject to e−h̃−w̃−d̃ ≤ 15
eh̃−w̃ ≤ 2
eh̃−d̃ ≤ 3
eh̃ + ew̃ + ed̃ ≤ 7
Common Classes 11/24
![Page 34: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/34.jpg)
Designing a Suitcase in Convex Form
minimize 2hw + 2hd+ 2wdsubject to h−1w−1d−1 ≤ 1
5hw−1 ≤ 2hd−1 ≤ 3h+ w + d ≤ 7h,w, d ≥ 0
Change of variables to h̃ = log h, w̃ = logw, d̃ = log d
minimize 2eh̃+w̃ + 2eh̃+d̃ + 2ew̃+d̃
subject to e−h̃−w̃−d̃ ≤ 15
eh̃−w̃ ≤ 2
eh̃−d̃ ≤ 3
eh̃ + ew̃ + ed̃ ≤ 7
Common Classes 11/24
![Page 35: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/35.jpg)
Geometric Programs in Convex Form
minimize f0(x)subject to fi(x) ≤ bi, for i ∈ C1.
hi(x) = bi, for i ∈ C2.x � 0
where fi’s are posynomials, hi’s are monomials, and bi > 0 (wlog 1).
In their natural parametrization by x1, . . . , xn ∈ R+, geometricprograms are not convex optimization problems
However, the feasible set and objective function are convex in thevariables y1, . . . , yn ∈ R where yi = log xi
Common Classes 12/24
![Page 36: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/36.jpg)
Geometric Programs in Convex Form
minimize f0(x)subject to fi(x) ≤ bi, for i ∈ C1.
hi(x) = bi, for i ∈ C2.x � 0
where fi’s are posynomials, hi’s are monomials, and bi > 0 (wlog 1).
In their natural parametrization by x1, . . . , xn ∈ R+, geometricprograms are not convex optimization problemsHowever, the feasible set and objective function are convex in thevariables y1, . . . , yn ∈ R where yi = log xi
Common Classes 12/24
![Page 37: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/37.jpg)
Geometric Programs in Convex Form
minimize f0(x)subject to fi(x) ≤ bi, for i ∈ C1.
hi(x) = bi, for i ∈ C2.x � 0
where fi’s are posynomials, hi’s are monomials, and bi > 0 (wlog 1).
Each monomial cxa11 xa22 . . . xakk can be rewritten as a convex
function cea1y1+a2y2+...+akyk
Therefore, each posynomial becomes the sum of these convexexponential functionsInequality constraints and objective become convexEquality constraint cxa11 x
a22 . . . xakk = b reduces to an affine
constraint a1y1 + a2y2 . . . akyk = log bc
Common Classes 12/24
![Page 38: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/38.jpg)
Outline
1 Convex Optimization Basics
2 Common Classes
3 Interlude: Positive Semi-Definite Matrices
4 More Convex Optimization Problems
![Page 39: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/39.jpg)
Symmetric MatricesA matrix A ∈ Rn×n is symmetric if and only if it is square and Aij = Ajifor all i, j.
We denote the cone of n× n symmetric matrices by Sn.
FactA matrix A ∈ Rn×n is symmetric if and only if it is orthogonallydiagonalizable.
i.e. A = QDQᵀ where Q is an orthogonal matrix andD = diag(λ1, . . . , λn).The columns of Q are the (normalized) eigenvectors of A, withcorresponding eigenvalues λ1, . . . , λnEquivalently: As a linear operator, A scales the space along anorthonormal basis QThe scaling factor λi along direction qi may be negative, positive,or 0.
Interlude: Positive Semi-Definite Matrices 13/24
![Page 40: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/40.jpg)
Symmetric MatricesA matrix A ∈ Rn×n is symmetric if and only if it is square and Aij = Ajifor all i, j.
We denote the cone of n× n symmetric matrices by Sn.
FactA matrix A ∈ Rn×n is symmetric if and only if it is orthogonallydiagonalizable.
i.e. A = QDQᵀ where Q is an orthogonal matrix andD = diag(λ1, . . . , λn).The columns of Q are the (normalized) eigenvectors of A, withcorresponding eigenvalues λ1, . . . , λnEquivalently: As a linear operator, A scales the space along anorthonormal basis QThe scaling factor λi along direction qi may be negative, positive,or 0.
Interlude: Positive Semi-Definite Matrices 13/24
![Page 41: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/41.jpg)
Symmetric MatricesA matrix A ∈ Rn×n is symmetric if and only if it is square and Aij = Ajifor all i, j.
We denote the cone of n× n symmetric matrices by Sn.
FactA matrix A ∈ Rn×n is symmetric if and only if it is orthogonallydiagonalizable.
i.e. A = QDQᵀ where Q is an orthogonal matrix andD = diag(λ1, . . . , λn).The columns of Q are the (normalized) eigenvectors of A, withcorresponding eigenvalues λ1, . . . , λnEquivalently: As a linear operator, A scales the space along anorthonormal basis QThe scaling factor λi along direction qi may be negative, positive,or 0.
Interlude: Positive Semi-Definite Matrices 13/24
![Page 42: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/42.jpg)
Positive Semi-Definite MatricesA matrix A ∈ Rn×n is positive semi-definite if it is symmetric andmoreover all its eigenvalues are nonnegative.
We denote the cone of n× n positive semi-definite matrices by Sn+We use A � 0 as shorthand for A ∈ Sn+
A = QDQᵀ where Q is an orthogonal matrix andD = diag(λ1, . . . , λn), where λi ≥ 0.As a linear operator, A performs nonnegative scaling along anorthonormal basis Q
NotePositive definite, negative semi-definite, and negative definite definedsimilarly.
Interlude: Positive Semi-Definite Matrices 14/24
![Page 43: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/43.jpg)
Positive Semi-Definite MatricesA matrix A ∈ Rn×n is positive semi-definite if it is symmetric andmoreover all its eigenvalues are nonnegative.
We denote the cone of n× n positive semi-definite matrices by Sn+We use A � 0 as shorthand for A ∈ Sn+
A = QDQᵀ where Q is an orthogonal matrix andD = diag(λ1, . . . , λn), where λi ≥ 0.As a linear operator, A performs nonnegative scaling along anorthonormal basis Q
NotePositive definite, negative semi-definite, and negative definite definedsimilarly.
Interlude: Positive Semi-Definite Matrices 14/24
![Page 44: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/44.jpg)
Positive Semi-Definite MatricesA matrix A ∈ Rn×n is positive semi-definite if it is symmetric andmoreover all its eigenvalues are nonnegative.
We denote the cone of n× n positive semi-definite matrices by Sn+We use A � 0 as shorthand for A ∈ Sn+
A = QDQᵀ where Q is an orthogonal matrix andD = diag(λ1, . . . , λn), where λi ≥ 0.As a linear operator, A performs nonnegative scaling along anorthonormal basis Q
NotePositive definite, negative semi-definite, and negative definite definedsimilarly.
Interlude: Positive Semi-Definite Matrices 14/24
![Page 45: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/45.jpg)
Geometric Intuition for PSD Matrices
For A � 0, let q1, . . . , qn be the orthonormal eigenbasis for A, andlet λ1, . . . , λn ≥ 0 be the corresponding eigenvalues.The linear operator x→ Ax scales the qi component of x by λiWhen applied to every x in the unit ball, the image of A is anellipsoid with principal directions q1, . . . , qn and correspondingdiameters 2λ1, . . . , 2λn
When A is positive definite (i.e.λi > 0), and therefore invertible, theellipsoid is the set
{x : xTA−1x ≤ 1
}Interlude: Positive Semi-Definite Matrices 15/24
![Page 46: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/46.jpg)
Useful Properties of PSD Matrices
If A � 0, thenxTAx ≥ 0 for all xThe quadratic function xTAx is convexA = BTB for some matrix B.
Interpretation: PSD matrices encode the “pairwise similarity”relationships of a family of vectorsInterpretation: The quadratic form xTAx is the length of an affinetransformation of x, namely ||Bx||22
A has a positive semi-definite square root A12
A12 = Qdiag(
√λ1, . . . ,
√λn)Qᵀ
A can be expressed as a sum of vector outer-products (xxᵀ)
As it turns out, each of the above is also sufficient for A � 0 (assumingA is symmetric).
Interlude: Positive Semi-Definite Matrices 16/24
![Page 47: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/47.jpg)
Useful Properties of PSD Matrices
If A � 0, thenxTAx ≥ 0 for all xThe quadratic function xTAx is convexA = BTB for some matrix B.
Interpretation: PSD matrices encode the “pairwise similarity”relationships of a family of vectorsInterpretation: The quadratic form xTAx is the length of an affinetransformation of x, namely ||Bx||22
A has a positive semi-definite square root A12
A12 = Qdiag(
√λ1, . . . ,
√λn)Qᵀ
A can be expressed as a sum of vector outer-products (xxᵀ)
As it turns out, each of the above is also sufficient for A � 0 (assumingA is symmetric).
Interlude: Positive Semi-Definite Matrices 16/24
![Page 48: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/48.jpg)
Outline
1 Convex Optimization Basics
2 Common Classes
3 Interlude: Positive Semi-Definite Matrices
4 More Convex Optimization Problems
![Page 49: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/49.jpg)
Quadratic Programming
Minimizing a convex quadratic function over a polyhedron.
minimize xᵀPx+ cᵀx+ dsubject to Ax ≤ b
P � 0
Objective can be rewritten as (x− x0)ᵀP (x− x0) for some centerx0
Sublevel sets are scaled copies of an ellipsoid centered at x0More Convex Optimization Problems 17/24
![Page 50: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/50.jpg)
Examples
Constrained Least SquaresGiven a set of measurements (a1, b1), . . . , (am, bm), where ai ∈ Rn isthe i’th input and bi ∈ R is the i’th output, fit a linear function minimizingmean square error, subject to known bounds on the linear coefficients.
minimize ||Ax− b||22 = xᵀAᵀAx− 2bᵀAx+ bᵀbsubject to li ≤ xi ≤ ui, for i = 1, . . . , n.
More Convex Optimization Problems 18/24
![Page 51: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/51.jpg)
Examples
Distance Between PolyhedraGiven two polyhedra Ax � b and Cx � d, find the distance betweenthem.
minimize ||z||22 = zᵀIzsubject to z = y − x
Ax � bBy � d
More Convex Optimization Problems 18/24
![Page 52: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/52.jpg)
Conic Optimization Problems
This is an umbrella term for problems of the following form
minimize cᵀxsubject to Ax+ b ∈ K
Where K is a convex cone (e.g. Rn+, positive semi-definite matrices,etc). Evidently, such optimization problems are convex.
As shorthand, the cone containment constraint is often written usinggeneralized inequalities
Ax+ b �K 0
−Ax �K b
. . .
More Convex Optimization Problems 19/24
![Page 53: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/53.jpg)
Conic Optimization Problems
This is an umbrella term for problems of the following form
minimize cᵀxsubject to Ax+ b ∈ K
Where K is a convex cone (e.g. Rn+, positive semi-definite matrices,etc). Evidently, such optimization problems are convex.
As shorthand, the cone containment constraint is often written usinggeneralized inequalities
Ax+ b �K 0
−Ax �K b
. . .
More Convex Optimization Problems 19/24
![Page 54: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/54.jpg)
Example: Second Order Cone Programming
We will exhibit an example of a conic optimization problem with K asthe second order cone
K = {(x, t) : ||x||2 ≤ t}
Linear Program with Random ConstraintsConsider the following optimization problem, where each ai is agaussian random variable with mean ai and covariance matrix Σi.
minimize cᵀxsubject to aᵀi x ≤ bi w.p. at least 0.9, for i = 1, . . . ,m.
ui := aᵀi x is a univariate normal r.v. with mean ui := aᵀi x and
stddev σi :=√xᵀΣix = ||Σ
12i x||2
ui ≤ bi with probability φ( bi−uiσi), where φ is the CDF of the
standard normal random variable.Since we want this probability to exceed 0.9, we require that
bi − uiσi
≥ φ−1(0.9) ≈ 1.3 ≈ 1/0.77
||Σ12i x||2 ≤ 0.77(bi − aᵀi x)
More Convex Optimization Problems 20/24
![Page 55: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/55.jpg)
Example: Second Order Cone Programming
Linear Program with Random ConstraintsConsider the following optimization problem, where each ai is agaussian random variable with mean ai and covariance matrix Σi.
minimize cᵀxsubject to aᵀi x ≤ bi w.p. at least 0.9, for i = 1, . . . ,m.
ui := aᵀi x is a univariate normal r.v. with mean ui := aᵀi x and
stddev σi :=√xᵀΣix = ||Σ
12i x||2
ui ≤ bi with probability φ( bi−uiσi), where φ is the CDF of the
standard normal random variable.Since we want this probability to exceed 0.9, we require that
bi − uiσi
≥ φ−1(0.9) ≈ 1.3 ≈ 1/0.77
||Σ12i x||2 ≤ 0.77(bi − aᵀi x)
More Convex Optimization Problems 20/24
![Page 56: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/56.jpg)
Example: Second Order Cone Programming
Linear Program with Random ConstraintsConsider the following optimization problem, where each ai is agaussian random variable with mean ai and covariance matrix Σi.
minimize cᵀxsubject to aᵀi x ≤ bi w.p. at least 0.9, for i = 1, . . . ,m.
ui := aᵀi x is a univariate normal r.v. with mean ui := aᵀi x and
stddev σi :=√xᵀΣix = ||Σ
12i x||2
ui ≤ bi with probability φ( bi−uiσi), where φ is the CDF of the
standard normal random variable.
Since we want this probability to exceed 0.9, we require thatbi − uiσi
≥ φ−1(0.9) ≈ 1.3 ≈ 1/0.77
||Σ12i x||2 ≤ 0.77(bi − aᵀi x)
More Convex Optimization Problems 20/24
![Page 57: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/57.jpg)
Example: Second Order Cone Programming
Linear Program with Random ConstraintsConsider the following optimization problem, where each ai is agaussian random variable with mean ai and covariance matrix Σi.
minimize cᵀxsubject to aᵀi x ≤ bi w.p. at least 0.9, for i = 1, . . . ,m.
ui := aᵀi x is a univariate normal r.v. with mean ui := aᵀi x and
stddev σi :=√xᵀΣix = ||Σ
12i x||2
ui ≤ bi with probability φ( bi−uiσi), where φ is the CDF of the
standard normal random variable.Since we want this probability to exceed 0.9, we require that
bi − uiσi
≥ φ−1(0.9) ≈ 1.3 ≈ 1/0.77
||Σ12i x||2 ≤ 0.77(bi − aᵀi x)
More Convex Optimization Problems 20/24
![Page 58: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/58.jpg)
Semi-Definite Programming
These are conic optimization problems where the cone in question isthe set of positive semi-definite matrices.
minimize cᵀxsubject to x1F1 + x2F2 . . . xnFn +G � 0
Where F1, . . . , Fn are matrices, and � refers to the positivesemi-definite cone Sn+.
ExamplesFitting a distribution, say a Gaussian, to observed data. Variable isa positive semi-definite covariance matrix.As a relaxation to combinatorial problems that encode pairwiserelationships: e.g. finding the maximum cut of a graph.
More Convex Optimization Problems 21/24
![Page 59: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/59.jpg)
Semi-Definite Programming
These are conic optimization problems where the cone in question isthe set of positive semi-definite matrices.
minimize cᵀxsubject to x1F1 + x2F2 . . . xnFn +G � 0
Where F1, . . . , Fn are matrices, and � refers to the positivesemi-definite cone Sn+.
ExamplesFitting a distribution, say a Gaussian, to observed data. Variable isa positive semi-definite covariance matrix.As a relaxation to combinatorial problems that encode pairwiserelationships: e.g. finding the maximum cut of a graph.
More Convex Optimization Problems 21/24
![Page 60: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/60.jpg)
Quasiconvex Optimization Problems
More Convex Optimization Problems 22/24
![Page 61: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/61.jpg)
Example
More Convex Optimization Problems 23/24
![Page 62: CS599: Convex and Combinatorial Optimization Fall …shaddin/cs599fa13/slides/lec9-10.pdfCS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems](https://reader031.fdocuments.in/reader031/viewer/2022022010/5b05dc2a7f8b9a5c308c1f47/html5/thumbnails/62.jpg)
A Note on Tractability
More Convex Optimization Problems 24/24