2019-07-23 Products of Variables in Mixed Integer ... · Mixed Integer Linear Program and Mixed...

Products of Variables in Mixed Integer Programming

Dr. Tobias Achterberg


Welcome to the Webinar

Speaker Introduction


• R&D Director at Gurobi

• Dr. Achterberg studied mathematics and computer science at the Technical University of Berlin and the Zuse Institute Berlin.

• Author of SCIP, currently the best academic MIP solver. In addition to numerous publications in scientific journals

• Received several awards for his dissertation and for SCIP, such as the Beale-Orchard-Hays Prize

3

OutlineMixed Integer Linear Program and Mixed Integer Bilinear Program

Bilinear Programming and McCormick relaxation• RLT Cuts for Bilinear Programs

Detecting linearization of products with binary variable in MILP formulations

Exploiting product relations for cutting planes

Computational Results

Mixed Integer Linear ProgramA Mixed Integer Linear Program (MILP) is defined as

min 𝑐"𝑥s.t. 𝐴𝑥 ≤ 𝑏

𝑥 ≥ 0𝑥) ∈ ℤ for all 𝑗 ∈ 𝐼

Large number of applications• Accounting• Advertising• Agriculture• Airlines• ATM provisioning• Compilers• Defense• Electrical power • Energy


• Finance • Food service• Forestry• Gas distribution• Government• Internet applications• Logistics/supply chain • Medical• Mining

• National research labs• Online dating• Portfolio management• Railways• Recycling• Revenue management• Semiconductor• Shipping

• Social networking• Sourcing• Sports scheduling• Statistics• Steel manufacturing• Telecommunications• Transportation• Utilities• Workforce scheduling

Mixed Integer Bilinear ProgramA Mixed Integer Bilinear Program is defined as

min 𝑐"𝑥 + 𝑑"𝑦s.t. 𝐴𝑥 + 𝐷𝑦 ≤ 𝑏

−𝑥3𝑥) + 𝑦3) = 0 for all (𝑖, 𝑗) ∈ 𝑆𝑥 ≥ 0𝑥) ∈ ℤ for all 𝑗 ∈ 𝐼

Applications• pooling problem• petrochemical industry• wastewater treatment• emissions regulation• agricultural / food industry• mining


• energy• production planning• logistics• water distribution• engineering design• finance




For LP based solvers bilinear constraints need to be relaxed• McCormick relaxation

−𝑦3) + 𝑙)𝑥3 + 𝑙3𝑥) ≤ 𝑙3 𝑙)−𝑦3) + 𝑢)𝑥3 + 𝑢3𝑥) ≤ 𝑢3 𝑢)𝑦3) − 𝑢)𝑥3 − 𝑙3𝑥) ≤ −𝑙3 𝑢)𝑦3) − 𝑙)𝑥3 − 𝑢3𝑥) ≤ −𝑢3 𝑙) picture from Costa and Liberti: "Relaxations of multilinear

convex envelopes: dual is better than primal"


McCormick Relaxation of Bilinear Constraints

Bilinear constraint: 𝑦3) = 𝑥3𝑥)

Lower McCormick envelope

−𝑦3) + 𝑙)𝑥3 + 𝑙3𝑥) ≤ 𝑙3 𝑙)−𝑦3) + 𝑢)𝑥3 + 𝑢3𝑥) ≤ 𝑢3 𝑢)

Upper McCormick envelope𝑦3) − 𝑢)𝑥3 − 𝑙3𝑥) ≤ −𝑙3 𝑢)𝑦3) − 𝑙)𝑥3 − 𝑢3𝑥) ≤ −𝑢3 𝑙)

Issue:• LP solution will often violate bilinear

constraints


pictures from Costa and Liberti: "Relaxations of multilinearconvex envelopes: dual is better than primal"




RLT Cuts:1. Multiply a linear constraint 𝑎𝑥 + 𝑑𝑦 = 𝑏 by some variable 𝑥) ≥ 02. Linearize resulting quadratic constraint by

a) replacing products 𝑥3𝑥) for (𝑖, 𝑗) ∈ 𝑆 by corresponding product variables 𝑦3)b) relaxing products 𝑥3𝑥) for (𝑖, 𝑗) ∉ 𝑆 by linear term, e.g., McCormick underestimator

How can such a cut be violated?• Replacing product terms 𝑎3𝑥3𝑥) = 𝑎3𝑦3) increases violation if 𝑎3𝑥3∗𝑥)∗ < 𝑎3𝑦3)∗


Products of Variables in Mixed Integer Linear Programs

Some products do not need to be explicitly stated as bilinear constraintsProducts of a binary variable 𝑥3 and an arbitrary variable 𝑥) can be modeled as linear constraints

𝑥3 ∈ 0,1 , 𝑥) ∈ 𝑙), 𝑢) , 𝑦3) = 𝑥3𝑥) ∈ min 𝑙), 0 ,max 𝑢), 0

−𝑦3) + 𝑙)𝑥3 + 𝑙3𝑥) ≤ 𝑙3 𝑙) ⇔ 𝑥3 = 0 → 𝑦3) ≥ 0

−𝑦3) + 𝑢)𝑥3 + 𝑢3𝑥) ≤ 𝑢3𝑢) ⇔ 𝑥3 = 1 → 𝑦3) ≥ 𝑥)𝑦3) − 𝑢)𝑥3 − 𝑙3𝑥) ≤ −𝑙3 𝑢) ⇔ 𝑥3 = 0 → 𝑦3) ≤ 0

𝑦3) − 𝑙)𝑥3 − 𝑢3𝑥) ≤ −𝑢3𝑙) ⇔ 𝑥3 = 1 → 𝑦3) ≤ 𝑥)

𝑦3) ≥ 𝑥3𝑥)

𝑦3) ≤ 𝑥3𝑥)


Products of Variables in Mixed Integer Linear ProgramsSome products do not need to be explicitly stated as bilinear constraintsProducts of a binary variable 𝑥3 and an arbitrary variable 𝑥) can be modeled as linear constraints


−𝑦3) + 𝑙)𝑥3 + 𝑙3𝑥) ≤ 𝑙3 𝑙) ⇔ 𝑥3 = 0 → 𝑦3) ≥ 0


𝑦3) − 𝑙)𝑥3 − 𝑢3𝑥) ≤ −𝑢3𝑙) ⇔ 𝑥3 = 1 → 𝑦3) ≤ 𝑥)

𝑦3) ≥ 𝑥3𝑥)

𝑦3) ≤ 𝑥3𝑥)





−𝑦3) + 𝑙)𝑥3 + 𝑙3𝑥) ≤ 𝑙3 𝑙) ⇔ 𝑥3 = 0 → 𝑦3) ≥ 0


𝑦3) − 𝑙)𝑥3 − 𝑢3𝑥) ≤ −𝑢3𝑙) ⇔ 𝑥3 = 1 → 𝑦3) ≤ 𝑥)

𝑦3) ≥ 𝑥3𝑥)

𝑦3) ≤ 𝑥3𝑥)

0

0





−𝑦3) + 𝑙)𝑥3 + 𝑙3𝑥) ≤ 𝑙3 𝑙) ⇔ 𝑥3 = 0 → 𝑦3) ≥ 0


𝑦3) − 𝑙)𝑥3 − 𝑢3𝑥) ≤ −𝑢3𝑙) ⇔ 𝑥3 = 1 → 𝑦3) ≤ 𝑥)

We can detect these structures in a MILP formulation

𝑦3) ≥ 𝑥3𝑥)

𝑦3) ≤ 𝑥3𝑥)

0

0


Useful MIP Data StructuresImplied bounds• MIP solvers store implied bound relations between binary and non-binary variables in a table

𝑥3 = 0→𝑥) ≤ 𝑏, 𝑥3 = 0→𝑥) ≥ 𝑏, 𝑥3 = 1→𝑥) ≤ 𝑏, 𝑥3 = 1→𝑥) ≥ 𝑏

Cliques• MIP solvers store implications between binary variables in a clique table

𝑥3 = 0→𝑥) = 0, 𝑥3 = 0→𝑥) = 1, 𝑥3 = 1→𝑥) = 0, 𝑥3 = 1→𝑥) = 1−𝑥3 + 𝑥) ≤ 0, −𝑥3 − 𝑥) ≤ −1, 𝑥3 + 𝑥) ≤ 1, 𝑥3 − 𝑥) ≤ 0

• Can aggregate multiple implications into larger cliques, e.g.:

𝑥H + 𝑥I + 𝑥J + 𝑥K + 𝑥L ≤ 1

to represent all implications 𝑥3 = 1→𝑥) = 0 for all 𝑖, 𝑗 = 1, … , 5, 𝑖 ≠ 𝑗


Detection of Products of VariablesMore general version:

𝑥3 = 1→𝑦3) ≤ 𝑎𝑥) + 𝑏 𝑦3) − 𝑎𝑥) + 𝑀H𝑥3 ≤ 𝑏 +𝑀H𝑥3 = 0→𝑦3) ≤ 𝑐𝑥) + 𝑑 𝑦3) − 𝑐𝑥3 − 𝑀I𝑥3 ≤ 𝑑

We can detect these structures in a MILP formulation1. Inspect inequalities with three variables, from which at least one is binary2. For each binary variable in this constraint, assume this is 𝑥3 and search for the other constraint

• 𝑐 ≠ 0: a constraint with 𝑥3 , 𝑥), and 𝑦3)• 𝑐 = 0,𝑀 ≠ 0: a constraint with 𝑥3 and 𝑦3) (i.e., an implied bound if 𝑦3) is non-binary, a clique if 𝑦3) is binary)• 𝑐 = 0,𝑀 = 0: a global upper bound on 𝑦3)

Result: 𝑦3) ≤ 𝑥3 𝑎𝑥) + 𝑏 + 1 − 𝑥3 𝑐𝑥) + 𝑑 = 𝑎 − 𝑐 𝑥3𝑥) + 𝑏 − 𝑑 𝑥3 + 𝑐𝑥) + 𝑑If we also find a product relation in the ≥ direction with the same 𝑎, 𝑏, 𝑐, 𝑑: an equality relationWe store product relations as 𝑥3𝑥) ⋚ R𝑎𝑦3) + S𝑏𝑥3 + �̂�𝑥) + U𝑑


Detection of Products of Variables – DetailsBasic algorithm:

1. Find "implied relations" 𝑥3 = 𝑓 → 𝑦3) ⋚ 𝑎𝑥) + 𝑏 by scanning length 3 constraints with at least one binary2. For each implied relation, try to find other constraint 𝑥3 = 1 − 𝑓 → 𝑦3) ⋚ 𝑐𝑥) + 𝑑 as

• implied relation (binary search for 𝑥3, 𝑥), 𝑦3) in implied relations table)• implied bound (𝑐 = 0, 𝑦3) non-binary; linear scan in implications of 𝑥3 = 1 − 𝑓),• clique (𝑐 = 0, 𝑦3) binary; scan clique table columns of 𝑦3) and W𝑦3) for match with column of 𝑥3 or �̅�3)• unconditional relation (clique, implied bound, or length 2 inequalities for 𝑥) and 𝑦3)), or• global bound of 𝑦3)

3. Also try to find constraints in opposite direction to identify equality product relations

Special case: product relations defined without length 3 constraint

𝑦3) ≤ 𝑎𝑥) + 𝑏 (unconditional relation)𝑥3 = 0 → 𝑦3) ≤ 𝑑 (implied bound)

yields: 𝑦3) ≤ 𝑥3 𝑎𝑥) + 𝑏 + 1 − 𝑥3 𝑑 = 𝑎𝑥3𝑥) + 𝑏 − 𝑑 𝑥3 + 𝑑 YZ[𝑥3𝑥) ≥

HY𝑑 − 𝑏 𝑥3 +

HY𝑦3) −

\Y

Extended algorithm: also consider implied bounds in Step 1• What about cliques? Too expensive!

• And already implicitly stored in clique table𝑦3) + 𝑥3 ≤ 1, 𝑦3) + 𝑥) ≤ 1⇒𝑦3) ≤ 1 − 𝑥3 1 − 𝑥)

Separation of RLT CutsFor each variable 𝑥) consider 𝑥) − 𝑙) and 𝑢) − 𝑥)• For each violated product relation of 𝑥3𝑥) mark constraints of 𝑥3 in which 𝑎3 has the right sign• Multiply each marked constraint by 𝑥) − 𝑙) or 𝑢) − 𝑥) , respectively to yield quadratic constraint• Substitute product terms using known product relations or a linear relaxation• If violated: add to cut table


Substituting Product TermsNeed to get rid of product terms 𝑥3𝑥) to turn quadratic into linear constraint• Substitute 𝑥)I = 𝑥) if 𝑥) is binary• Tangent for 𝑎)𝑥)I if 𝑥) is non-binary and 𝑎) > 0• Secant for 𝑎)𝑥)I if 𝑥) is non-binary and 𝑎) < 0• Clique relationship if 𝑥3 and 𝑥) are binary and clique exists

• 𝑥3 + 𝑥) ≤ 1 ⇒ 𝑥3𝑥) = 0

• 𝑥3 + 1 − 𝑥) ≤ 1 ⇒ 𝑥3𝑥) = 𝑥3• 1 − 𝑥3 + 𝑥) ≤ 1 ⇒ 𝑥3𝑥) = 𝑥)• 1 − 𝑥3 + 1 − 𝑥) ≤ 1 ⇒ 𝑥3𝑥) = 𝑥3 + 𝑥) − 1

• Product relationship, if product relationship exists for 𝑥3𝑥) in the right direction• Product table for binary times non-binary variable, detected in linear constraints• Bilinear constraints

• Else: one of the two lower (𝑎) > 0) or one of the two upper (𝑎) < 0) McCormick envelope constraints• Pick the one that is tighter for the current LP solution


tangent

secant

𝑎) > 0

𝑎) < 0

Tricks to Speed-Up SeparationCheck violation on constraint projected to variables that are not at bound• McCormick constraints are tight if at least one of the variables is at a bound• Replacing product term with product variable will not change violation of cut• Hence:

• Only check violation on projected system• Repeat substitution procedure on full constraint if violated cut has found on projected system

• In basic LP solution, most variables are typically at one of their bounds• Can work on a much smaller system!

Make sure to not flood the solver's cut table with too many RLT cuts• It happens frequently that we can find tons of RLT cuts• Will slow down overall MIP solving process• Solution:

• Add RLT cuts to local cut table• Filter local cut table using regular filtering based on violation and orthogonality• Add surviving cuts to global cut table


𝐴_ 𝐴` 𝑏≤

pictures from Costa and Liberti: "Relaxations of multilinearconvex envelopes: dual is better than primal"

BQP CutsBoolean Quadric Polytope• 𝐵𝑄𝑃d = 𝑥, 𝑦 ∈ ℝd×ℝd×d | 𝑦3) = 𝑥3𝑥) for all i,j ∈ [𝑛], 𝑥) ∈ 0,1 for all j ∈ [𝑛]• BQP is equivalent to the Cut Polytope

Quadratic Programming with Box Constraints• 𝑄𝑃𝐵d = 𝑥, 𝑦 ∈ ℝd×ℝd×d | 𝑦3) = 𝑥3𝑥) for all i,j ∈ [𝑛], 0 ≤ 𝑥) ≤ 1 for all j ∈ [𝑛]

• Just the continuous relaxation of 𝐵𝑄𝑃d• Theorem (Burer and Letchford, 2008):

Every valid inequality for 𝐵𝑄𝑃d is also valid for 𝑄𝑃𝐵d

• Hence, facets of 𝐵𝑄𝑃d provide cuts for bilinear constraints with bounded variables• Padberg (1989) studied facets of 𝐵𝑄𝑃d

• RLT inequalities are facets if 𝑖 ≠ 𝑗• PSD inequalities are facets• "Clique" and "Cut" inequalities are facets• "Cut type" inequalities are facets, generalize "Clique" and "Cut" inequalities (Yajima and Fujie, 1998)


BQP Clique CutsTheorem (Padberg, 1989): For any 𝑆 ⊆ {1,… , 𝑛} with 𝑆 ≥ 3, and any 𝛼 ∈ {1,… , 𝑆 − 2}, the clique inequality

𝛼∑3∈w 𝑥3 − ∑(3,))⊂w 𝑦3) ≤y yzH

I

is valid and facet defining.

Simple case: triangle inequalities (𝛼 = 1, 𝑆 = 3): ∑3∈w 𝑥3 − ∑ 3,) ⊂w 𝑦3) ≤ 1

𝑥H +𝑥I −𝑦HI ≤ 1𝑥H +𝑥J −𝑦HJ ≤ 1

𝑥I +𝑥J −𝑦IJ ≤ 1

𝑥H +𝑥I +𝑥J −𝑦HI −𝑦HJ −𝑦IJ ≤ 1

Facet-defining even if 𝑆 is not a maximal clique!


model inequalities for 𝑥3 = 𝑥) = 1→𝑦3) = 1

zero-half cut

BQP Triangle CutsSwitching (i.e., complementation of 𝑥 and associated changes in 𝑦) yields four triangle inequalities:

𝑥H +𝑥I +𝑥J −𝑦HI −𝑦HJ −𝑦IJ ≤ 1−𝑥H +𝑦HI +𝑦HJ −𝑦IJ ≤ 0

−𝑥I +𝑦HI −𝑦HJ +𝑦IJ ≤ 0−𝑥J −𝑦HI +𝑦HJ +𝑦IJ ≤ 0

Separation• Enumerate all triangles to collect all violated triangle cuts• Filter cuts and pass surviving cuts to cut pool


Computational Results – Nodes (RLT and BQP Cuts)

0.9770.909

0.978

0.555

0.9680.883

0.978

0.473

0.966

0.862

0.955

0.444

0.956

0.8510.896

0.448

0.911

0.413

0

0.2

0.4

0.6

0.8

1

1.2

1.4

MILP MIQP MIQCP Non-convex MIQCP

>0sec >1sec >10sec >100sec >1000sec


not e

noug

h da

ta

not e

noug

h da

ta

MILP: 3924 modelsMIQP: 309 modelsMIQCP: 438 modelsNonConv: 325 modelsTotal time: 338 CPU days

Computational Results – Time (RLT and BQP Cuts)

0.992 0.9620.998

0.778

0.9890.944

0.998

0.645

0.9840.925

0.987

0.583

0.9760.919

0.98

0.583

0.938

0.572

0

0.2

0.4

0.6

0.8

1

1.2

1.4

MILP MIQP MIQCP Non-convex MIQCP

>0sec >1sec >10sec >100sec >1000sec


1.6% speed-up 8.1% speed-up 1.3% speed-up

not e

noug

h da

ta

not e

noug

h da

ta

71.5% speed-up

MILP: 3924 modelsMIQP: 309 modelsMIQCP: 438 modelsNonConv: 325 modelsTotal time: 338 CPU days

Thank You – Questions?

Your Next Steps

• If you haven’t already done so, please register for an account at http://www.gurobi.com

• For questions about Gurobi pricing contact [email protected] or [email protected]

• A recording of this webinar, including the slides, will be available in roughly one week


http://www.gurobi.com/

mailto:[email protected]

mailto:[email protected]

2019-07-23 Products of Variables in Mixed Integer ... · Mixed Integer Linear Program and Mixed...

Documents

Transcript of 2019-07-23 Products of Variables in Mixed Integer ... · Mixed Integer Linear Program and Mixed...