Lecture 14: Linear Programming Relaxation and Rounding

Lecture 14: Linear Programming Relaxation andRounding

Rafael Oliveira

University of WaterlooCheriton School of Computer Science

rafael.oliveira.teaching@gmail.com

June 24, 2021

1 / 100

Overview

Part IWhy Relax & Round?

Vertex Cover

Set Cover

Conclusion

Acknowledgements

2 / 100

Motivation - NP-hard problems

Many important optimization problems are NP-hard to solve.

What do we do when we see one?

1 Find approximate solutions in polynomial time!2 Sometimes we even do that for problems in P (but we want much

much faster solutions)

Integer Linear Program (ILP):

minimize cT x

subject to Ax ≤ b

x ∈ Nn

Advantage of ILPs: very expressive language to formulateoptimization problems (capture many combinatorial optimizationproblems)

Disadvantage of ILPs: capture even NP-hard problems (thus NP-hard)

But we know how to solve LPs. Can we get partial credit in life?

3 / 100

What do we do when we see one?

1 Find approximate solutions in polynomial time!2 Sometimes we even do that for problems in P (but we want much

minimize cT x

subject to Ax ≤ b

x ∈ Nn

4 / 100

What do we do when we see one?1 Find approximate solutions in polynomial time!

2 Sometimes we even do that for problems in P (but we want muchmuch faster solutions)

minimize cT x

subject to Ax ≤ b

x ∈ Nn

5 / 100

What do we do when we see one?1 Find approximate solutions in polynomial time!2 Sometimes we even do that for problems in P (but we want much

minimize cT x

subject to Ax ≤ b

x ∈ Nn

6 / 100

minimize cT x

subject to Ax ≤ b

x ∈ Nn

7 / 100

minimize cT x

subject to Ax ≤ b

x ∈ Nn

8 / 100

minimize cT x

subject to Ax ≤ b

x ∈ Nn

9 / 100

minimize cT x

subject to Ax ≤ b

x ∈ Nn

10 / 100

Example

Maximum Independent Set:

G (V ,E ) graph.

Independent set S ⊆ V such that u, v ∈ S ⇒ {u, v} 6∈ E .

Integer Linear Program:

maximize∑v∈V

subject to xu + xv ≤ 1 for {u, v} ∈ E

xv ∈ {0, 1} for v ∈ V

11 / 100

Relax... & Round!In our quest to get efficient (exact or approximate) algorithms forproblems of interest, the following strategy is very useful:

1 Formulate combinatorial optimization problem as ILP

2 Derive LP from the ILP by removing the integral constraints

This is called an LP relaxation.

3 We are still minimizing the same objective function, but over a(potentially) larger set of solutions.

opt(LP) ≤ opt(ILP)

4 Solve LP optimally using efficient algorithm.

1 If solution to LP has integral values, then it is a solution to ILP and weare done

2 If solution has fractional values, then we have to devise roundingprocedure that transforms

fractional solutions → integral solutions

opt(LP) ≤ c · opt(ILP)

12 / 100

13 / 100

14 / 100

15 / 100

16 / 100

4 Solve LP optimally using efficient algorithm.1 If solution to LP has integral values, then it is a solution to ILP and we

are done

17 / 100

are done2 If solution has fractional values, then we have to devise rounding

procedure that transforms

18 / 100

Not all LPs created equalWhen solving LP

minimize cT x

subject to Ax = b

x ≥ 0

it is important to understand geometry of feasible set & how nice thecorner points are, as they are the candidates to optimum solution.

Let P := {x ∈ Rn≥0 | Ax = b}

Vertex Solutions: a solution x ∈ P is a vertex solution if 6 ∃y 6= 0such that x + y ∈ P and x − y ∈ P

Extreme Point Solutions: x ∈ P is an extreme point solution if∃u ∈ Rn such that x is the unique optimum solution to the LP withconstraint P and objective uT x .

Basic Solutions: let supp(x) := {i ∈ [n] | xi > 0} be the set ofnonzero coordinates of x . Then x ∈ P is a basic solution ⇔ thecolumns of A indexed by supp(x) are linearly independent.

19 / 100

minimize cT x

subject to Ax = b

x ≥ 0

Let P := {x ∈ Rn≥0 | Ax = b}

20 / 100

minimize cT x

subject to Ax = b

x ≥ 0

Let P := {x ∈ Rn≥0 | Ax = b}

21 / 100

minimize cT x

subject to Ax = b

x ≥ 0

Let P := {x ∈ Rn≥0 | Ax = b}

22 / 100

minimize cT x

subject to Ax = b

x ≥ 0

Let P := {x ∈ Rn≥0 | Ax = b}

23 / 100

Vertex Cover

Set Cover

Conclusion

Acknowledgements

24 / 100

Vertex Cover

Setup:

Input: a graph G (V ,E ).

Output: Minimum number of vertices that “touches” all edges ofgraph. That is, minimum set S such that for each edge {u, v} ∈ Ewe have

|S ∩ {u, v}| ≥ 1.

Weighted version: associate to each vertex v ∈ V a cost cv ∈ R≥0.

1 Setup ILP:

minimize∑u∈V

cu · xu

subject to xu + xv ≥ 1 for {u, v} ∈ E

xu ∈ {0, 1} for u ∈ V

25 / 100

Vertex Cover

Setup:

|S ∩ {u, v}| ≥ 1.

1 Setup ILP:

minimize∑u∈V

cu · xu

xu ∈ {0, 1} for u ∈ V

26 / 100

Vertex Cover

Setup:

|S ∩ {u, v}| ≥ 1.

1 Setup ILP:

minimize∑u∈V

cu · xu

xu ∈ {0, 1} for u ∈ V

27 / 100

Simple 2-approximation (unweighted)

1 List edges of E in any order. Set S = ∅

2 For each {u, v} ∈ E :1 If S ∩ {u, v} = ∅, then S ← S ∪ {u, v}

3 return S

Proof of correctness:

By construction, S is a vertex cover.

If added elements to S k times, then |S | = 2k and G has a matchingof size k, which means that optimum vertex cover is at least k .

Thus, we get a 2-approximation.

28 / 100

1 List edges of E in any order. Set S = ∅2 For each {u, v} ∈ E :

1 If S ∩ {u, v} = ∅, then S ← S ∪ {u, v}

3 return S

29 / 100

1 If S ∩ {u, v} = ∅, then S ← S ∪ {u, v}3 return S

30 / 100

31 / 100

32 / 100

33 / 100

34 / 100

What can go wrong in the weighted case?

Original Algo Heuristic: pick lowest weight only

35 / 100

Vertex Cover - LP relaxation

1 Setup ILP:

minimize∑u∈V

cu · xu

xu ∈ {0, 1} for u ∈ V

2 Drop integrality constraints

minimize∑u∈V

cu · xu

0 ≤ xu ≤ 1 for u ∈ V

3 Solve LP. Get optimal solution z for LP, where z = (zu)u∈V .

4 Round LP as follows: round zv to nearest integer.

36 / 100

1 Setup ILP:

minimize∑u∈V

cu · xu

xu ∈ {0, 1} for u ∈ V

minimize∑u∈V

cu · xu

0 ≤ xu ≤ 1 for u ∈ V

37 / 100

1 Setup ILP:

minimize∑u∈V

cu · xu

xu ∈ {0, 1} for u ∈ V

minimize∑u∈V

cu · xu

0 ≤ xu ≤ 1 for u ∈ V

38 / 100

1 Setup ILP:

minimize∑u∈V

cu · xu

xu ∈ {0, 1} for u ∈ V

minimize∑u∈V

cu · xu

0 ≤ xu ≤ 1 for u ∈ V

39 / 100

Vertex Cover - Analysis1 Drop integrality constraints

minimize∑u∈V

cu · xu

0 ≤ xu ≤ 1 for u ∈ V

2 Solve LP. Get optimal solution z for LP.

3 Round zv to nearest integer. That is yv =

{1, if zv ≥ 1/2

0, if 0 ≤ zv < 1/24 y is an integral cover by construction5 each edge is covered, since given {u, v} ∈ E , at least one of zu, zv is≥ 1/2 (by feasibility of LP)

6 Cost of y is:∑u∈V

cu · yu ≤∑u∈V

cu · (2 · zu) ≤ 2 · OPT (ILP)

40 / 100

minimize∑u∈V

cu · xu

0 ≤ xu ≤ 1 for u ∈ V

{1, if zv ≥ 1/2

0, if 0 ≤ zv < 1/2

4 y is an integral cover by construction5 each edge is covered, since given {u, v} ∈ E , at least one of zu, zv is≥ 1/2 (by feasibility of LP)

cu · (2 · zu) ≤ 2 · OPT (ILP)

41 / 100

minimize∑u∈V

cu · xu

0 ≤ xu ≤ 1 for u ∈ V

{1, if zv ≥ 1/2

0, if 0 ≤ zv < 1/24 y is an integral cover by construction

5 each edge is covered, since given {u, v} ∈ E , at least one of zu, zv is≥ 1/2 (by feasibility of LP)

cu · (2 · zu) ≤ 2 · OPT (ILP)

42 / 100

minimize∑u∈V

cu · xu

0 ≤ xu ≤ 1 for u ∈ V

{1, if zv ≥ 1/2

cu · (2 · zu) ≤ 2 · OPT (ILP)

43 / 100

Vertex Cover - Analysis

minimize∑u∈V

cu · xu

0 ≤ xu ≤ 1 for u ∈ V

{1, if zv ≥ 1/2

cu · (2 · zu) ≤ 2 · OPT (ILP)

44 / 100

Vertex Cover

Set Cover

Conclusion

Acknowledgements

45 / 100

Set CoverSetup:

Input: a finite set U and a collection S1,S2, . . . ,Sn of subsets of U.

Output: The fewest collection of sets I ⊆ [n] such that⋃i∈I

Sj = U.

Weighted version: associate to each set Si a weight wi ∈ R≥0.

1 Setup ILP:

minimize∑i∈[n]

wi · xi

subject to∑i :v∈Si

xi ≥ 1 for v ∈ U

xi ∈ {0, 1} for i ∈ [n]

46 / 100

Set CoverSetup:

Sj = U.

1 Setup ILP:

minimize∑i∈[n]

wi · xi

xi ∈ {0, 1} for i ∈ [n]

47 / 100

Set CoverSetup:

Sj = U.

1 Setup ILP:

minimize∑i∈[n]

wi · xi

xi ∈ {0, 1} for i ∈ [n]

48 / 100

Set Cover - Relax...

1 Obtain LP relaxation:

minimize∑i∈[n]

wi · xi

0 ≤ xi ≤ 1 for i ∈ [n]

2 Suppose we end up with fractional solution z ∈ [0, 1]n when we solvethe LP above. Now need to come up with a rounding scheme.

3 Can we just round each coordinate zi to the nearest integer (like invertex cover)?

4 Not really. Say v ∈ U is in 20 sets, and we got zi = 1/20 for each ofthe sets v ∈ Si . Then rounding procedure above would not select anysuch set!

49 / 100

minimize∑i∈[n]

wi · xi

0 ≤ xi ≤ 1 for i ∈ [n]

50 / 100

minimize∑i∈[n]

wi · xi

0 ≤ xi ≤ 1 for i ∈ [n]

51 / 100

minimize∑i∈[n]

wi · xi

0 ≤ xi ≤ 1 for i ∈ [n]

52 / 100

Set Cover - Rounding

1 Think of zi as the “probability” that we would pick set Si .

2 Solution z describes an “optimal probability distribution” over waysto chose the sets Si .

3 Okay, but how do we cover?

Algorithm (Random Pick)

1 Input: values z = (z1, . . . , zn) ∈ [0, 1]n such that z is a solution to our LP

2 Output: a set cover for U

3 Set I = ∅4 for i = 1, . . . n

with probability zi , set I = I ∪ {i}

5 return I

4 Expected cost of the sets is∑n

i=1 wi · zi , which is the optimum forthe LP. But will this process cover U?

53 / 100

3 Set I = ∅4 for i = 1, . . . n

5 return I

54 / 100

3 Set I = ∅4 for i = 1, . . . n

5 return I

55 / 100

3 Set I = ∅4 for i = 1, . . . n

5 return I

56 / 100

3 Set I = ∅

4 for i = 1, . . . n

5 return I

57 / 100

3 Set I = ∅4 for i = 1, . . . n

5 return I

58 / 100

3 Set I = ∅4 for i = 1, . . . n

5 return I

59 / 100

3 Set I = ∅4 for i = 1, . . . n

5 return I

60 / 100

Analyzing Random Pick

Let’s consider the Random Pick process from point of view of v ∈ U.

v ∈ S1, . . . ,Sk (for simplicity)

As long as we select one of Si ’s above we are good (w.r.t. v)

We select Si with probability zi such that

k∑i=1

zi ≥ 1

Because z is a solution to our LP

What is probability that v is covered in Random Pick?

Definitely not 1. Think about case k = 2 and z1 = z2 = 1/2.

If had many elements like that, would expect many elementsuncovered. How to deal with this?

By perseverance! :)

61 / 100

k∑i=1

zi ≥ 1

By perseverance! :)

62 / 100

k∑i=1

zi ≥ 1

By perseverance! :)

63 / 100

k∑i=1

zi ≥ 1

By perseverance! :)

64 / 100

k∑i=1

zi ≥ 1

By perseverance! :)

65 / 100

k∑i=1

zi ≥ 1

By perseverance! :)

66 / 100

k∑i=1

zi ≥ 1

By perseverance! :)

67 / 100

k∑i=1

zi ≥ 1

By perseverance! :)

68 / 100

Probability that Element is Covered

Lemma (Probability of Covering an Element)

In a sequence of k independent experiments, in which the i th experimenthas success probability pi , and

k∑i=1

pi ≥ 1

then there is a probability ≥ 1− 1/e that at least one experiment issuccessful.

Probability that no experiment is successful:

(1− p1) · (1− p2) · · · (1− pk)

1− x ≤ e−x for x ∈ [0, 1]Thus probability of failure is

k∏i=1

(1− pi ) ≤k∏

e−pi = e−p1−···−pk ≤ 1/e

69 / 100

k∑i=1

pi ≥ 1

(1− p1) · (1− p2) · · · (1− pk)

k∏i=1

(1− pi ) ≤k∏

e−pi = e−p1−···−pk ≤ 1/e

70 / 100

k∑i=1

pi ≥ 1

(1− p1) · (1− p2) · · · (1− pk)

1− x ≤ e−x for x ∈ [0, 1]

Thus probability of failure is

k∏i=1

(1− pi ) ≤k∏

e−pi = e−p1−···−pk ≤ 1/e

71 / 100

k∑i=1

pi ≥ 1

(1− p1) · (1− p2) · · · (1− pk)

k∏i=1

(1− pi ) ≤k∏

e−pi = e−p1−···−pk ≤ 1/e72 / 100

Randomized Rounding

Algorithm (Randomized Rounding)

1 Input: values z = (z1, . . . , zn) ∈ [0, 1]n s.t. z is a solution to our LP

3 Set I = ∅4 While there is element v ∈ U uncovered:

For i = 1, . . . , n:

with probability zi , set I = I ∪ {i}5 return I

To analyze this, need to show that we don’t execute the for loop too manytimes.

Lemma (Probability Decay)

Let t ∈ N. The probability that the for loop will be executed more thanln(|U|) + t times is at most e−t .

73 / 100

Randomized Rounding

For i = 1, . . . , n:

74 / 100

Randomized Rounding

For i = 1, . . . , n:

75 / 100

Randomized Rounding

For i = 1, . . . , n:

76 / 100

Proof of Lemma

Probability that for loop is executed more than ln(|U|) + t times isthe probability that there is an uncovered element after theln(|U|) + t iteration.

Let v ∈ U. For each iteration of the loop, there is a probability of 1/ethat v is not covered. (by our previous lemma)

Probability that v not covered after ln(|U|) + t iterations is(1

)ln(|U|)+t

|U|· e−t

Union bound.

77 / 100

Proof of Lemma

)ln(|U|)+t

|U|· e−t

Union bound.

78 / 100

Proof of Lemma

)ln(|U|)+t

|U|· e−t

Union bound.

79 / 100

Proof of Lemma

)ln(|U|)+t

|U|· e−t

Union bound.

80 / 100

Proof of Lemma

)ln(|U|)+t

|U|· e−t

Union bound.

81 / 100

Cost of Rounded SolutionNow that we know we will cover with high probability, we need to boundthe cost of the solution we came up with.

At each implementation of for loop, our expected cover weight is

k∑i=1

wi · zi

After t iterations of for loop, expected weight is

t ·k∑

wi · zi

By Markov:Pr[X ≥ 2 · E[X ]] ≤ 1/2.

Lemma (Cost of Rounding)

Given z optimal for the LP, our randomized rounding outputs, withprobability ≥ 0.45 a feasible solution to set cover with≤ 2 · (ln(|U|) + 3) · OPT (ILP) sets

82 / 100

k∑i=1

wi · zi

t ·k∑

wi · zi

By Markov:Pr[X ≥ 2 · E[X ]] ≤ 1/2.

83 / 100

k∑i=1

wi · zi

t ·k∑

wi · zi

By Markov:Pr[X ≥ 2 · E[X ]] ≤ 1/2.

84 / 100

k∑i=1

wi · zi

t ·k∑

wi · zi

By Markov:Pr[X ≥ 2 · E[X ]] ≤ 1/2.

85 / 100

k∑i=1

wi · zi

t ·k∑

wi · zi

By Markov:Pr[X ≥ 2 · E[X ]] ≤ 1/2.

86 / 100

Cost of Rounding

1 Let t = ln(|U|) + 3. There is a probability at most e−3 < 0.05 thatwhile loop runs for more than t steps.

2 After t steps, expected weight is

ω := t ·∑

wi · zi ≤ t · OPT (ILP)

3 Markov ⇒ probability that our solution has weight ≥ 2 · ω is ≤ 1/2

4 Union bound, with probability ≤ 0.55 either run for more than ttimes, or our solution has weight ≥ 2ω

5 Thus, with probability ≥ 0.45 we stop at t iterations and constructsolution to set cover with cost ≤ 2t · OPT (ILP)

87 / 100

Cost of Rounding

ω := t ·∑

88 / 100

Cost of Rounding

ω := t ·∑

89 / 100

Cost of Rounding

ω := t ·∑

90 / 100

Cost of Rounding

ω := t ·∑

91 / 100

Cost of Rounding

ω := t ·∑

92 / 100

Putting Everything Together

1 Formulate set cover problem as ILP

2 Derive LP from the ILP LP relaxation

3 We are still minimizing the same objective function (weight of cover),but over a (potentially) larger (fractional) set of solutions.

OPT (LP) ≤ OPT (ILP)

2 If have fractional values, rounding procedure

Randomized Rounding algorithm, with probability ≥ 0.45 we get

cost(rounded solution) ≤ 2 · (ln(|U|) + 3) · OPT (ILP)

93 / 100

94 / 100

95 / 100

96 / 100

are done

97 / 100

are done2 If have fractional values, rounding procedure

98 / 100

Conclusion

Integer Linear programming - very general, and pervasive in(combinatorial) algorithmic life

ILP NP-hard

Rounding for the rescue!

Solve LP and round the solution

Deterministic rounding when solutions are niceRandomized rounding when things a bit more complicated

99 / 100

Acknowledgement

Lecture based largely on:

Lectures 7-8 of Luca’s Optimization class

See Luca’s vertex cover notes at https://lucatrevisan.github.io/teaching/cs261-11/lecture07.pdf

See Luca’s set cover notes at https://lucatrevisan.github.io/teaching/cs261-11/lecture08.pdf

100 / 100

Lecture 14: Linear Programming Relaxation and Rounding

Documents

Transcript of Lecture 14: Linear Programming Relaxation and Rounding

Chapter 1 GRASP FOR LINEAR INTEGER …jpp/publications/PDF/mip-grasp.pdfGRASP FOR LINEAR INTEGER PROGRAMMING ... by solving the linear programming (LP) relaxation of the problem. ...

Vertex Cover - Linear Progamming and Approximation Algorithmscrab.rutgers.edu/~rajivg/studentTalks/vertexCover.pdf · Linear Programming Theory Vertex Cover LP-Rounding Primal-Dual

Combining (Integer) Linear Programming … (Integer) Linear Programming ... The linear programming relaxation always provides a lower bound for the original minimization problem, ...Authors:

Hypocoercivity for kinetic equations with linear relaxation terms[Mouhot, Neumann (2006)], [Villani (2007, 2008)] Other related approaches: non-linear Boltzmann and Landau equations:

Citius, Altius, Fortius! - TUM · Relaxation Methods Minimization Methods Non-Linear Equations Multigrid Applications • For large and sparse matrices or systems of linear equations,

CONVEX HULL RELAXATION (CHR) FOR CONVEX AND · PDF fileCONVEX HULL RELAXATION (CHR) FOR CONVEX AND NONCONVEX MINLP PROBLEMS WITH LINEAR CONSTRAINTS by Aykut Ahlatçıoğlu 1

Yongwenhu-A Revised Algorithm for Solving the Sum of Linear Ratios Problem With Lower Dimension Using Linear Relaxation

Uncrossing - unitn.it · Dual uncrossing Primal uncrossing Termination, ﬁniteness, efﬁciency TU and TDI TDI and unimodularity Iterative rounding Iterative relaxation Uncrossing

A Linear Relaxation Technique for the Position Analysis of Multiloop ...

Iterative Rounding and Relaxationkenkyubu/bessatsu/open/B23/pdf/...Iterative Rounding and Relaxation 173 This linear program has exponentially many constraints, but it can be solved

Lecture 2: Linear Programming and Duality - KTH · 2017. 9. 7. · Linear Programming Duality •Part III: Linear Programming as a Problem Relaxation. Part I: Linear Programming,

5 axis local corner rounding of linear tool path discontinuities · 2020. 6. 18. · 5-axis local corner rounding of linear tool path discontinuities Xavier Beudaert, Sylvain Lavernhe,

Lecture 3: Rounding Linear Programs

Non-linear model of stress relaxation curve at high ...€¦ · Non-linear model of stress relaxation curve at high temperatures of Gr. 91 steel *Woo-Gon Kim1), Jae-Young Park2) and

Equivalence of Piecewise-Linear Approximation …...Equivalence of piecewise-linear approximation and Lagrangian relaxation for network revenue management Sumit Kunnumkal Kalyan Talluriy

Analytical Expressions for the Relaxation Moduli of Linear ...

THE RELAXATION METHOD FOR LINEAR INEQUALITIES€¦ · LINEAR INEQUALITIES T. S. MOTZKIN AND I. J. SCHOENBERG I. STATEMENT OF PROBLEM AND MAIN RESULTS 1. The relaxation method. Let

Submodular Set Function Maximization via the Multilinear Relaxation & Dependent Rounding

Iterative Rounding and Iterative Relaxation Some of the slides prepared by Lap Chi Lau.

The Convex Hull Relaxation for Nonlinear Integer … integer minimization ... while solving a linear integer problem ... cannot be directly compared with Lagrangean relaxation, it