2006 : Solving Linear Problems with MOSEK (Pittsburgh 2006)

http://www.mosek.com

Solving Linear OptimizationProblems with MOSEK.

Bo Jensen

(Erling D. Andersen)MOSEK ApS,

Fruebjergvej 3, Box 16,2100 Copenhagen,

Denmark.Email: [email protected]

WWW: http://www.mosek.com

Introduction

What is MOSEK

Introduction

What is MOSEK

Topics

The linear optimizer

The simplexoptimizers

Recent work in thesimplex optimizers

Computationalresults

3 / 31

■ A software package.■ Solves large-scale sparse problems.■ Solves linear, conic, and nonlinear convex problems.■ Stand-alone as well as embedded.■ Used to solve problems with up to millions of constraints

and variables.■ Version 4 released spring 2006.■ Version 5 is work in progress.

For details about interfaces, trials, etc. see

http://www.mosek.com.

Topics

Introduction

What is MOSEK

Topics





4 / 31

■ The problem(P ) min cT x

st Ax = b,

x ≥ 0.

■ The linear optimizers.

◆ Homogenius primal-dual interior-point optimizer(Not main focus in this talk).

◆ Simplex optimizer.

■ What is the recent improvements?■ What is the (relative) performance?


Introduction

What is MOSEK

Topics





5 / 31

■ Presolve.■ Form the reduced primal or dual.■ Scale (optimizer specific).■ Optimize (interior-point or simplex).■ Basis identification (interior-point only).■ Undo scaling and dualizing.■ Postsolve.

The simplex optimizers

Introduction

Introduction


Introduction



7 / 31

■ Primal and dual simplex optimizer.

◆ Efficient cold start and warm start.◆ Multiple pricing options. Dantzig, partial,

approximate/exact steepest edge pricing and hybrid.◆ Degeneration handling.◆ Crashes an initial basis.

■ Revised simplex algorithm + many enhancements.■ Many enhancements still possible!.

Recent work in the simplex optimizers

Exploting sparsity aggressively

Introduction



Exploting sparsityaggressively

l

Numerical stability

Primal simplex(Choosing enteringcolumn)

Dual bound flippingidea used moreaggressively

Primal (dual)Degeneracy

Dualizer


9 / 31

■ Simplex algs. require solution of the linear equationsystems

Bf = A:j and BT g = ei.

in each iteration.


Introduction




l

Numerical stability




Dualizer


9 / 31



in each iteration.■ Assume a sparse LU factorization of the basis

B = LU.


Introduction




l

Numerical stability




Dualizer


9 / 31




B = LU.

■ f can be computed as follow. Solve

Lf̄ = A:j

and thenUf = f̄ .


Introduction




l

Numerical stability




Dualizer


9 / 31




B = LU.

■ f can be computed as follow. Solve

Lf̄ = A:j

and thenUf = f̄ .

■ Simple implementation requires O(nz(L) + nz(U)) flops.

Exploting sparsity aggressively (continued)

Introduction




l

Numerical stability




Dualizer


10 / 31

■ Consider the simple example

10 1x 0 1

f̄1

f̄2

f̄3

=

0x

0


Introduction




l

Numerical stability




Dualizer


10 / 31


10 1x 0 1

f̄1

f̄2

f̄3

=

0x

0

■ Clearly sparsity in the RHS can be exploited! (doneextensively in MOSEK).


Introduction




l

Numerical stability




Dualizer


10 / 31


10 1x 0 1

f̄1

f̄2

f̄3

=

0x

0


■ Gilbert and Peierls [GIL:88] demonstrate how to solve thetriangular system in O(minimal number of flops).


Introduction




l

Numerical stability




Dualizer


10 / 31


10 1x 0 1

f̄1

f̄2

f̄3

=

0x

0



■ Aim: Solves with L and U and updates to the LU shouldrun in O(minimal number of flops) and not in O(m) forinstance.


Introduction




l

Numerical stability




Dualizer


10 / 31


10 1x 0 1

f̄1

f̄2

f̄3

=

0x

0



■ Aim: Solves with L and U and updates to the LU shouldrun in O(minimal number of flops) and not in O(m) forinstance.

■ Drawback: Both L and U must be stored row and columnwise because solves with LT and UT are required too.

Numerical stability

Introduction




l

Numerical stability




Dualizer


11 / 31

■ Improving numerical stability.

◆ Moved LU update before updating solution.

Numerical stability

Introduction




l

Numerical stability




Dualizer


11 / 31



■ Saves one solve with L in eiT B−1 [GOL:77].

■ More stable approach.

Numerical stability

Introduction




l

Numerical stability




Dualizer


11 / 31





◆ Better handling of singularities (sing. variables arefixed until later).

Numerical stability

Introduction




l

Numerical stability




Dualizer


11 / 31





◆ Better handling of singularities (sing. variables arefixed until later).

◆ Switch to safe mode if deemed unstable.

Primal simplex (Choosing entering column)

Introduction




l

Numerical stability




Dualizer


12 / 31

Partial pricing (PA).

■ Based on the reduced cost.■ Scans only part of non basic variables.■ Cheap (updated row not needed).■ Often takes many iterations.

Primal simplex (Choosing entering column)(continued)

Introduction




l

Numerical stability




Dualizer


13 / 31

Steepest edge pricing

■ Normalized pricing i.e scale reduced cost sj with acolumn norm Tj .


Introduction




l

Numerical stability




Dualizer


13 / 31



■ Reduced cost are maintained (updated row neededei

T B−1N ).


Introduction




l

Numerical stability




Dualizer


13 / 31




T B−1N ).■ Examples of Steepest edge (SE) and approximations

(ASE):

◆ Exact SE Tj = ||αj = B−1Nj || [FOR:92].


Introduction




l

Numerical stability




Dualizer


13 / 31





(ASE):


■ Work in update one extra btran solvewT = αj

T B−1 and n dot products wT Nj .■ Can be very expensive.


Introduction




l

Numerical stability




Dualizer


13 / 31





(ASE):




◆ Exact DEVEX Tj = ||αj||H [HAR:73].


Introduction




l

Numerical stability




Dualizer


13 / 31





(ASE):




◆ Exact DEVEX Tj = ||αj||H [HAR:73].

■ Only solve positions where basis variable is in H .■ Weights are reset when reference H is changed.


Introduction




l

Numerical stability




Dualizer


14 / 31

■ Examples of Steepest edge (SE) and approximations(ASE): (continued)


Introduction




l

Numerical stability




Dualizer


14 / 31


◆ Cheaper approximation ASE [SWI:98].


Introduction




l

Numerical stability




Dualizer


14 / 31



■ Best for sparse problems.■ Weights are reset.


Introduction




l

Numerical stability




Dualizer


14 / 31




■ DEVEX and ASE can be combined with partial pricing!.


Introduction




l

Numerical stability




Dualizer


14 / 31





◆ Updated row only needed for a subset of N .


Introduction




l

Numerical stability




Dualizer


14 / 31





◆ Updated row only needed for a subset of N .◆ Much less work i.e updated row sparse.


Introduction




l

Numerical stability




Dualizer


14 / 31






■ Hybrid pricing (HY).

◆ Starts with a cheap pricing rule (i.e partial) and shiftsto a more expensive (i.e SE,DEVEX or ASE).


Introduction




l

Numerical stability




Dualizer


14 / 31








◆ Hard to determine when to shift (flop counting).


Introduction




l

Numerical stability




Dualizer


14 / 31








◆ Hard to determine when to shift (flop counting).◆ Default in MOSEK.


Introduction




l

Numerical stability




Dualizer


15 / 31

Pricing examples problem:

Problem Rows Cols Nzdano3mip 3202 13873 79655dbic1 43200 183235 1038761dfl001 6071 12230 35632fome12 24284 48920 142528ken-18 105127 154699 358171lp22 2958 13434 65560mod2 34774 31728 165129nsct2 23003 14981 675156pds-40 66844 212859 462128rail4284 4284 1092610 11279748rlfprim 58866 8052 265927sgpf5y6 246077 308634 828070stormG2-125 66185 157496 418321world 34506 32734 164470


Introduction




l

Numerical stability




Dualizer


16 / 31

Pricing examples iterations:

Problem SE ASE DEVEX PA HYdano3mip 77001 33084 10222 64086 54866dbic1 26138 42343 18634 202450 188402dfl001 34173 50113 50821 635957 50113fome12 144697 220056 346720 2000001 220056ken-18 77061 98707 92409 126829 98707lp22 54089 104082 51179 321998 104082mod2 89687 109460 121352 311964 109460nsct2 11198 10027 10057 22168 10027pds-40 38165 73237 66809 214811 249359rail4284 115852 291826 97428 841251 322663rlfprim 1156 2850 1223 3358 3196sgpf5y6 798 690 535 571 690stormG2-125 45923 52985 54809 142067 52985world 101250 120010 139574 493023 120010Sum 817188 1209470 1061772 5380534 1584616


Introduction




l

Numerical stability




Dualizer


17 / 31

Pricing examples solution times:

Problem SE ASE DEVEX PA HYdano3mip 152.21 51.73 20.43 18.14 25.85dbic1 84.51 121.03 87.85 130.67 129.48dfl001 25.32 24.34 35.29 205.53 24.28fome12 195.92 223.64 462.48 1140.98 223.31ken-18 91.82 37.61 50.25 26.82 37.54lp22 193.76 104.28 54.81 151.75 105.90mod2 714.31 274.01 526.64 273.54 273.17nsct2 4.64 1.76 2.59 3.32 1.76pds-40 135.26 294.34 420.00 55.76 96.40rail4284 689.56 433.12 293.54 461.40 419.32rlfprim 0.70 1.09 0.68 0.89 0.89sgpf5y6 3.18 3.14 3.07 2.96 3.14stormG2-125 33.03 29.01 42.39 179.95 28.89world 800.60 304.71 652.46 506.31 304.37Sum 3124.82 1903.81 2652.48 3158.02 1674.30

Dual bound flipping idea used more aggressively

Introduction




l

Numerical stability




Dualizer


18 / 31

A ranged variable i.e −∞ < lj < xj < uj < ∞ may not bebinding in dual minratio if profitable. This involves flippingnonbasic variables to opposite bound to remain dual feasible .


Introduction




l

Numerical stability




Dualizer


18 / 31


■ Longer dual steplengths.


Introduction




l

Numerical stability




Dualizer


18 / 31


■ Longer dual steplengths.■ Reduces degeneracy.


Introduction




l

Numerical stability




Dualizer


18 / 31


■ Longer dual steplengths.■ Reduces degeneracy.■ Less iterations.


Introduction




l

Numerical stability




Dualizer


18 / 31


■ Longer dual steplengths.■ Reduces degeneracy.■ Less iterations.■ More flexibility in pivot choice (i.e potentially more

stable).


Introduction




l

Numerical stability




Dualizer


18 / 31


■ Longer dual steplengths.■ Reduces degeneracy.■ Less iterations.■ More flexibility in pivot choice (i.e potentially more

stable).■ Improves sparsity of the basis when degenerated! (i.e if

xBi becomes feasible no basis exchange is needed).

Dual bound flipping idea used more aggressively(continued)

Introduction




l

Numerical stability




Dualizer


19 / 31

Bound flipping examples:

Iter Time

Problem Rows Cols NB WB NB WB

osa-60 10280 232966 6938 5111 58.12 8.84

world 34506 32734 54566 32606 218.81 50.03

pds-40 66844 212859 34274 26599 96.51 18.48

ken-18 105127 154699 151203 51452 258.18 13.92

client 27216 20567 80555 63660 208.40 84.09

WB = MOSEK 5 Dual simplex with bound flips

NB = MOSEK 5 Dual simplex with no bound flips

Primal (dual) Degeneracy

Introduction




l

Numerical stability




Dualizer


20 / 31

■ Perturb lj and uj (cj).■ Better and more aggressive perturbation scheme.■ Sparsity issues important.■ Clean up perturbations with dual (primal) simplex.■ Many examples where long solves is substantial reduced.

Dualizer

Introduction




l

Numerical stability




Dualizer


21 / 31

■ Added an automatic dualizer. (Dualizing when deemedworthwhile, also for hotstarts).

Dualizer

Introduction




l

Numerical stability




Dualizer


21 / 31

■ Added an automatic dualizer. (Dualizing when deemedworthwhile, also for hotstarts).Examples with simplex optimizer free :

Primal form

Problem Rows Cols Time

rlfprim 36294 4046 7.43

shuttle100 34817 34819 119.23

test1000 865 865 0.26

neos1 131581 1892 31.72

Dual form

Problem Rows Cols Time Ratio

rlfprim 4046 36294 1.86 3.99

shuttle100 20 34817 2.98 40.01

test1000 65 864 0.33 0.78

neos1 1892 131581 11.18 2.83

Computational results

Test setup

Introduction




Test setup

Primal Simplex

Dual Simplex

Primal, DualSimplex and Interior(no timeouts)

ConclusionsA number openissues exists

References

23 / 31

■ 422 problems (mixed size).■ Max 3000 sec. allowed.■ Windows Server 2003 64 bit, Intel Xeon 3.0 GHz , 4GB

RAM.

Primal Simplex

Introduction




Test setup

Primal Simplex

Dual Simplex



References

24 / 31

MOSEK 5 (alpha) MOSEK 4

First place 293 129

Second place 129 293

Total time 4887.18 11757.24

Arh. Mean 11.98 27.86

Geo. Mean 0.20 0.23

Dual Simplex

Introduction




Test setup

Primal Simplex

Dual Simplex



References

25 / 31

MOSEK 5 (alpha) MOSEK 4

First place 356 66

Second place 66 356

Total time 8385.90 12149.22

Arh. Mean 19.87 28.79

Geo. Mean 0.16 0.22

Primal, Dual Simplex and Interior (no timeouts)

Introduction




Test setup

Primal Simplex

Dual Simplex



References

26 / 31

Primal Dual Interior

First place 102 226 92

Second place 163 147 110

Third place 155 47 218

Total time 4259.77 2916.68 887.36

Arh. Mean 10.14 6.94 2.11

Geo. Mean 0.19 0.16 0.18

Conclusions

Introduction




Test setup

Primal Simplex

Dual Simplex



References

27 / 31

■ Interior overall faster than simplex.■ Simplex:

◆ MOSEK 5 (alpha version) substantial faster thanMOSEK 4.

◆ Dual simplex faster than primal.

A number open issues exists

Introduction




Test setup

Primal Simplex

Dual Simplex



References

28 / 31

■ Simplex:

◆ Degeneracy (non-perturbation method might beneeded in extreme cases).

◆ Improve primal pricing.◆ Better crashing on special problems.◆ Choose more sparse path.

References

Introduction




Test setup

Primal Simplex

Dual Simplex



References

29 / 31

[AND:00] E. D. Andersen and K. D. Andersen, ”The MOSEKinterior point optimizer for linear programming: animplementation of the homogeneous algorithm, In HighPerformance Optimization, pp 197–232, H. Frenk, K. Roos, T.Terlaky and S. Zhang, editor(s), Kluwer Academic Publishers

[AND:02] E. D. Andersen, ”Certificates of primal and dualinfeasibility in linear programming”, ComputationalOptimization and Applications 20(2), 2001, 171–183.

[AND:99] E. D. Andersen, ”On exploiting problem structure in abasis identification procedure for linear programming”,INFORMS Journal on Computing 11(1), 1999, pp 95–103.

[GIL:88] J. R. Gilbert and T. Peierls, ”Sparse partial pivoting in timeproportional to arithmetic operations”, SIAM J. Sci. Statist.Comput., 9, 1988, pp. 862–874.

[CON] http://www.tsp.gatech.edu/concorde.html

[SCIP:A] http://scip.zib.de/

References (continued)

Introduction




Test setup

Primal Simplex

Dual Simplex



References

30 / 31

[SWI:98] A. Swietanowski, ”A New Steepest Edge Approximationfor the Simplex Method for Linear Programming”,Computational Optimization and Applications, 10(3), 1998,271–281.

[KOS:02] E. Kostina, ”The Long Step Rule in the Bounded-VariableDual Simplex Method: Numerical Experiments”,Mathematical Methods of Operations Research, 55 2002, I. 3.

[MAR:03] Maros I, ”A Generalized Dual Phase-2 SimplexAlgorithm”, European Journal of Operational Research, 149,2003, pp. 1–16

[FOR:92] J. Forrest, and D. Goldfarb, ”Steepest-Edge SimplexAlgorithms for Linear Programming”, MathematicalProgramming, 57, 1992, pp. 341–374.

[HAR:73] P.M.J. Harris, ”Pivot Selection Method of the Devex LP

Code”, Mathematical Programming, 5, 1973, pp. 1–28.

References (continued)

Introduction




Test setup

Primal Simplex

Dual Simplex



References

31 / 31

[GOL:77] D. Goldfarb, ”On the Bartels-Golub decomposition for

linear programming bases,” Mathematical. Programming, 13,

1977, pp 272-279

2006 : Solving Linear Problems with MOSEK (Pittsburgh 2006)

Documents

Transcript of 2006 : Solving Linear Problems with MOSEK (Pittsburgh 2006)