Runseeds and Modeling Assistance in CPLEX 12.7

© 2017 IBM Corporation

Runseeds and Modeling AssistanceROADEF 2017, Metz

Daniel Junglas, PhD, Software Scientist, CPLEX Xavier Nodet, Program Manager, CPLEX Optimization Studio

© 2017 IBM Corporation2

WHAT’S NEW IN 12.7


CPLEX Optimization Studio 12.7

§ (Automatic) Benders decomposition§ Interactive for CP Optimizer, reading .cpo files.§ PWL functions in C and file formats§ …

§ Performance improvements:- Barrier: 7%, with crossover: 21%- MIP: 15%- MIQP (15%), MIQCP (25%), global (MI)QP (> 2x)- Scheduling: 25%

§ C.f. INFORMS 2016 presentations for more details


RUNSEEDS


1

10

100

1000

Performance variability exists and has massive effects

≤ 2 ok 131

]2, 4] shaky 53

> 4 bad 28

rough classification

All solvable instances, run on 5 different seeds, max/min truncated at 1000, log scale

3 instances from MIPLIB2010 test set, run on 5 different random seedsmodel seed 1 seed 2 seed 3 seed 4 seed 5 min max max/min

map14 65.41 61.77 62.82 62.35 64.96 61.77 65.41 1.06

rococoB10-011000

1582.72 811.05 844.39 2223.25 740.78 740.78 2223.25 3.00

neos-911880

315.22 10000.14 354.19 1.71 27.40 1.71 10000.14 5848.04


To assess performance variability with CPLEX:

model seed 1 seed 2 seed 3 seed 4 seed 5 min max max/min

map14 65.41 61.77 62.82 62.35 64.96 61.77 65.41 1.06

rococoB10-011000

1582.72 811.05 844.39 2223.25 740.78 740.78 2223.25 3.00

neos-911880

315.22 10000.14 354.19 1.71 27.40 1.71 10000.14 5848.04

3 instances from MIPLIB2010 test set, run on 5 different random seeds

l run one model with different random seeds

l new command in the interactive: tools→runseeds

Performance variability exists and has massive effects

l surrogate for random perturbation of the model


CPLEX> read beasleyC3.mps 1. Read model


CPLEX> read beasleyC3.mpsCPLEX> set mip tolerance mipgap 0CPLEX> set mip tolerance absmipgap 0

1. Read model

2. Set parameters (if any)

l parameters set here apply to every run!


CPLEX> read beasleyC3.mpsCPLEX> set mip tolerance mipgap 0CPLEX> set mip tolerance absmipgap 0CPLEX> tools runseeds

1. Read model


3. Launch

l runs model with different random seedsl default is 30 random seedsl each solve starts from scratchl CPLEX compiles statistics for each run



1. Read model


3. Launch

4. Wait . . .



1. Read model


3. Launch

4. Wait . . .

====== runseeds statistics of 30 runs

exit sol objective gap iteration node runtime dettimerun code stat value (%) count count seconds ticks

1 0 101 754 0.00 478311 4051 10.68 6941.202 0 101 754 0.00 378060 3991 8.91 5653.563 0 101 754 0.00 421786 4021 10.04 6489.83

. . .30 0 101 754 0.00 413233 3969 9.06 5775.45

Optimization status codes:objective gap iteration node runtime dettime

value (%) count count seconds ticks101 : integer optimal solution (30 times)average: 754 0.00 459225 4009 10.84 7021.98std dev: 1.87205e-10 0.00 66658 43 1.07 779.37

Exit codes:0 : No error

5. Inspect results



1. Read model


3. Launch

4. Wait . . .



1 0 101 754 0.00 478311 4051 10.68 6941.202 0 101 754 0.00 378060 3991 8.91 5653.563 0 101 754 0.00 421786 4021 10.04 6489.83

. . .30 0 101 754 0.00 413233 3969 9.06 5775.45




5. Inspect results

solution information, should be consistentotherwise → modeling assistance



1. Read model


3. Launch

4. Wait . . .



1 0 101 754 0.00 478311 4051 10.68 6941.202 0 101 754 0.00 378060 3991 8.91 5653.563 0 101 754 0.00 421786 4021 10.04 6489.83

. . .30 0 101 754 0.00 413233 3969 9.06 5775.45




5. Inspect results

solution effort,indicates variability



1. Read model


3. Launch

4. Wait . . .



1 0 101 754 0.00 478311 4051 10.68 6941.202 0 101 754 0.00 378060 3991 8.91 5653.563 0 101 754 0.00 421786 4021 10.04 6489.83

. . .30 0 101 754 0.00 413233 3969 9.06 5775.45




5. Inspect results


aflow30a

Some example results

objective gap iteration node runtime dettimevalue (%) count count seconds ticks

101 : integer optimal solution (30 times)average: 1158 0.00 110298 3980 3.19 1883.01std dev: 0 0.00 24585 779 0.30 181.44

l A stable model


gmu-35-40

aflow30a




l A stable model


101 : integer optimal solution (30 times)average: -2.40673e+06 0.00 36824708 4624403 566.02 339839.59std dev: 0 0.00 94077727 12739496 1607.25 935535.50

l A model with high variability


Hurink_edata_mt06_lazy

aflow30a


101 : integer optimal solution (30 times)average: 49.2667 0.00 499151265 72138619 5946.96 3269407.19std dev: 5.52695 0.00 532709392 107808642 7681.00 4400455.20




l A stable model

l A suspicious model

l A model with high variability

standard deviation in objective → indicator for numerical problems→ modeling assistance


101 : integer optimal solution (30 times)average: -2.40673e+06 0.00 36824708 4624403 566.02 339839.59std dev: 0 0.00 94077727 12739496 1607.25 935535.50


MODELING ASSISTANCE


Modeling assistancel detect problems in model data (small numbers, …)l detect bad modeling constructs (big-M, ...)

→ improve numerical robustness

l issues collected from years of CPLEX support and developmentl issues with problem datal issues in the model formulationl issues with parameter settingsl issues during the solution process

l CPLEX scans your model and looks for these issuesl for each issue found a warning is printedl enabled by parameter setting (disabled by default)




l CPLEX scans your model and looks for these issuesl for each issue found a warning is printedl enabled by parameter setting (disabled by default)l CPX_PARAM_DATACHECK 0 – off

solveinput






0 – off

solveinput &check

1 – check for invalid data (dup indices, NaNs, ...)


l CPX_PARAM_DATACHECK





0 – off

2 – check for common modeling issuessolve

modeling assistance

modeling assistance

input &check

1 – check for invalid data (dup indices, NaNs, ...)


l CPX_PARAM_DATACHECK


Modeling assistance

Quick reminder: Assessing numerics in CPLEX (for MIP)


Modeling assistance


l Solution quality: CPLEX> display solution qualityMILP objective 2.1961300000e+05MILP solution norm |x| (Total, Max) 4.22897e+06 5.34130e+05MILP solution error (Ax=b) (Total, Max) 1.02709e-09 4.36557e-11MILP x bound error (Total, Max) 0.00000e+00 0.00000e+00MILP x integrality error (Total, Max) 0.00000e+00 0.00000e+00MILP slack bound error (Total, Max) 3.55271e-15 2.22045e-16


Modeling assistance



l Kappa statistics: track condition number of basis matricesCPLEX> set mip strategy kappa 2CPLEX> optimize…CPLEX> display solution kappaMax condition number: 2.7435e+13Percentage (number) of stable bases: 77.51% (11908)Percentage (number) of suspicious bases: 19.31% (2966)Percentage (number) of unstable bases: 3.18% (489)Percentage (number) of ill-posed bases: 0.00% (0)Attention level: 0.011480


Modeling assistance



l Kappa statistics: track condition number of basis matricesMax condition number:

Percentage (number) of stable bases:

Percentage (number) of suspicious bases:

Percentage (number) of unstable bases:

Percentage (number) of ill-posed bases:

Attention level:

usually no problemdouble checkbe concerned


aTx ≤ My y binary, M huge

y = 0 → aTx ≤ 0y = 1 → aTx ≤ M

y used to model availabilityof a resource

Issues with problem formulation: big-M constraints

resource not availableresource available



y = 0 → aTx ≤ 0y = 1 → aTx ≤ M


Typically M is much larger than ai→ wide range of coefficients in the coefficient matrix

causes numerical trouble





y = 0 → aTx ≤ 0y = 1 → aTx ≤ M


Typically M is much larger than ai→ wide range of coefficients in the coefficient matrix

causes numerical trouble

→ use indicator constraints y = 0 -> aTx <= 0y = 1 -> aTx <= M



often one of the two can be expressed as bound on a variable


FALL_12_3

Max condition number: 1.6814e+13Percentage (number) of stable bases: 8.06% Percentage (number) of suspicious bases: 89.66% Percentage (number) of unstable bases: 2.28% Percentage (number) of ill-posed bases: 0.00% Attention level: 0.015804


A model with big-M constraints


FALL_12_3


CPLEX Warning 1042: In constraint 'c1109', binary variable 'x257' is linked with variable 'x2079' having ratio 1e+06.

c1109: - 1000000 x257 + x2079 ≤ 0

x257 binary, x2079 ≥ 0




FALL_12_3




c1109: - 1000000 x257 + x2079 ≤ 0

x257 binary, x2079 ≥ 0


x257 = 0 -> x2079 = 0x257 = 1 -> x2079 <= 1000000


FALL_12_3




c1109: - 1000000 x257 + x2079 ≤ 0

x257 binary, x2079 ≥ 0


x257 = 0 -> x2079 = 0x2079 <= 1000000


FALL_12_3




c1109: - 1000000 x257 + x2079 ≤ 0

x257 binary, x2079 ≥ 0


2.2289e+1380.15% 19.83% 0.02% 0.00% 0.002048

→ numerics improve significantly


x257 = 0 -> x2079 = 0x2079 <= 1000000


l CPLEX uses IEEE double precision arithmetic (quad precision in some places)l not every rational number has an exact representation→ round-off error will occur

∑i in {1, ..., k} 1/k xi = 1

Issues with problem data: Single precision input data

Consider fraction 1/k and a simple constraint


l CPLEX uses IEEE double precision arithmetic (quad precision in some places)l not every rational number has an exact representation→ round-off error will occur

∑i in {1, ..., k} 1/k xi = 1

single double1/k Δ Δ(∑) Δ Δ(∑)1/3 9.9e-9 3.0e-8 1.9e-17 5.6e-171/5 3.0e-9 1.5e-8 1.1e-17 5.6e-171/6 5.0e-9 3.0e-8 9.3e-18 5.6e-17

only 1/k in singlerest in double!

xi exact integer→infeasible by at least 1e-8 with single precision data→in practice often a lot worse


Consider fraction 1/k and a simple constraint

∆ to exact value


l Obvious warning

→ Fix data generation



CPLEX Warning 1036: Coefficient for %s in %s looks like %d/%d in single precision


CPLEX Warning 1036: Coefficient for %s in %s looks like %d/%d in single precision

l Obvious warning

→ Fix data generation

l But alsoCPLEX Warning 1047: Coefficients in %s are fractions and can be scaled with %d/%d

→ scale row to get exact values

∑i in {1, ..., k} xi = k∑i in {1, ..., k} 1/k xi = 1 →





ns2118727.mps.gz


A model with single precision data




CPLEX Warning 1036: Coefficient for variable 'C7932' in constraint 'R1024' looks like 1/24 in single precision.

R1024: C0017 - C0700 - C0926 + C2957 + C5189 - C5190 - 0.0416667 C7932 = 0

definitely looks truncated/rounded






R1024: C0017 - C0700 - C0926 + C2957 + C5189 - C5190 - 0.0416667 C7932 = 0

8.7299e+06100.00%

0.00% 0.00% 0.00%








R1024: C0017 - C0700 - C0926 + C2957 + C5189 - C5190 - 0.0416667 C7932 = 0

8.7299e+06100.00%

0.00% 0.00% 0.00%

solution time from 120 seconds



90 seconds






R1024: C0017 - C0700 - C0926 + C2957 + C5189 - C5190 - 0.0416667 C7932 = 0

8.7299e+06100.00%

0.00% 0.00% 0.00%

solution time from 120 seconds



90 seconds


Attention

blindly changing coefficients may change feasible region

→ may end up solving a different model

→ change model generation, not model


Issues with problem data: extreme numbers

l feasibility tolerancel optimality tolerancel numerical emphasisl ...

l Numbers below CPLEX default tolerances→ essentially 0

→ Fix numbers or tighten tolerance parameters

l Numbers too small for the solver (smaller than 1e-13)→ essentially 0

→ Fix numbers

l Numbers larger than CPX_INFBOUND (1e20)→ CPLEX will consider them as infinity

→ Fix numbers or, e.g., remove constraints with right-hand side > CPX_INFBOUND


l CPX_PARAM_EPGAP: stop if relative MIP gap drops below a value

l relative MIP gap (in CPLEX): best integer – best objective1e-10 + |best integer|

Issues with parameter settings: relative MIP gap


l CPX_PARAM_EPGAP: stop if relative MIP gap drops below a value (default 1e-4)


l Objective: min aTx + KaTx* + K – (aTx' + K) aTx* – aTx'1e-10 + |aTx* + K| 1e-10 + |aTx* + K|


→ gap:





l K huge → gap small, independent of aTx* and aTx'


→ gap:





l K huge → gap small, independent of aTx* and aTx'

CPLEX Warning 1037: Solution optimal within relative optimality tolerance of %g, but not optimal if objective offset of value %g is removed

l related

l modeling assistance warning

CPLEX Warning 1038: Solution optimal within relative optimality tolerance of %g, but not optimal within absolute optimality tolerance of value %g


→ gap:

→ fix K or adapt CPX_PARAM_EPGAP


l CPLEX tracks condition number of basis matrices in LP solve.l Large number of bases with large condition number

→ potential numerical problems→ inexact/incorrect answers

l Numerical trouble while solving the problem→ Can you trust the results? How much can you trust them?

→ Check/fix data, check solution quality, use numericalemphasis, tighten tolerances

Issues during solve: big condition numbers

CPLEX Warning 1033: Detected 1277 suspicious condition number(s) >= 1e+07.CPLEX Warning 1034: Detected 30 unstable condition number(s) >= 1e+10.

l Solving ns2034125


BIGM_COEF one very big coefficient in row █ █

BIGM_TO_IND constraint can be replaced by indicator █ █

BIGM_VARBOUND variable bound constraint with very large bound █ █

EPGAP_LARGE solution optimal only due to large CPX_PARAM_EPGAP █ █ █

EPGAP_OBJOFFSET solution optimal only due to large objective offset █ █

FRACTION_SCALING rows can be scaled to get exact value █ █

*_LARGE_NUM suspiciously large number █ █

*_SMALL_NUM suspiciously small number █

CANCEL_TOL non-zero less than cancel tolerance █ █ █

FEAS_TOL rhs smaller than feasibility tolerance █ █ █

OPT_TOL objective coefficient smaller than optimality tolerance █ █ █

QOPT_TOL quadratic objective coefficient smaller than optimality tolerance █ █ █

SINGLE_PRECISION single precision data █

SYMMETRY_BREAKING_INEQ chain of symmetry breaking inequalities █ █

WIDE_COEFF_RANGE wide range of coefficients █ █

KAPPA_* bad kappa values detected during solve █ █ █ █

Potential issues in problem dataPotential issues in problem formulation

Potential issues in parameter settingsPotential issues in solve

Overview of modeling assistance warnings


l momentum1.mps.gz (absolute and relative MIP gap 0):

Combining modeling assistance and runseeds

l runseeds on original model:objective gap iteration node runtime dettime

value (%) count count seconds ticks101 : integer optimal solution (30 times)average: 109143 0.00 40986027 1128777 2881.88 1725072.05std dev: 0.000312912 0.00 27028241 726250 1807.30 1071736.66






l big-M constraintsl Single precision datal scaled rowsl wide coefficient ranges

l fix modeling assistance warnings:




→ iteration count, solution time, number of nodes reduced→ variability in iteration count, solution time, number of nodes reduced



l runseeds on model with all warnings fixed:objective gap iteration node runtime dettime


l fix modeling assistance warnings


Be careful when auto-fixing warningsl Model ns2122603 from “unstable” MIPLIB 2010 testsetl gives these warnings:

Detected constraint with wide range of coefficients. In constraint 'R12374' the ratio of largest and smallest (in absolute value) coefficients is 2.4e+09

Coefficient for variable 'C10462' in constraint 'R12374' looks like 1/24 in single precision.





l Solve original model:Max condition number: 1.7025e+20Percentage (number) of stable bases: 0.00% (0)Percentage (number) of suspicious bases: 0.00% (0)Percentage (number) of unstable bases: 0.00% (0)Percentage (number) of ill-posed bases: 100.00% (1)Attention level: 1.000000

CPLEX stops at root with sub-optimal solution





→ replace by indicator

→ recompute coefficients



3.4594e+150.55% (17)

93.47% (2906)5.92% (184)0.06% (2)0.027745

l Solve fixed

not too bad









3.4594e+150.55% (17)

93.47% (2906)5.92% (184)0.06% (2)0.027745

l Solve fixed

not too bad

l but: l fixed model gives super-optimal solutions!l solutions are infeasible in original model!









3.4594e+150.55% (17)

93.47% (2906)5.92% (184)0.06% (2)0.027745

l Solve fixed

not too bad

l but: l fixed model gives super-optimal solutions!l solutions are infeasible in original model! careful!


Conclusion

§ CPLEX 12.7 helps you- Assess performance variability

$ cplex –c ”read mymip.lp” ”tools runseeds”

- Avoid potentially harmful features in your models$ cplex –c ”set read datacheck 2” ”read mymip.lp” ”optimize”

§ Don’t forget to try Benders!$ cplex –c ”read mymip.lp” ”set benders strategy 3” ”optimize”


Conclusion

§ CPLEX 12.7 (free for both students and faculty staffs) helps you- Assess performance variability

$ cplex –c ”read mymip.lp” ”tools runseeds”

- Avoid potentially harmful features in your models$ cplex –c ”set read datacheck 2” ”read mymip.lp” ”optimize”

§ Don’t forget to try Benders!$ cplex –c ”read mymip.lp” ”set benders strategy 3” ”optimize”

Thank You!


Legal Disclaimer

• © IBM Corporation 2017. All Rights Reserved.

• The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.

• References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.

• Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

• IBM, the IBM logo, CPLEX Optimizer, CPLEX Optimization Studio are trademarks of International Business Machines Corporation in the United States, other countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both.

Runseeds and Modeling Assistance in CPLEX 12.7

Software

Transcript of Runseeds and Modeling Assistance in CPLEX 12.7