SATzilla: Portfolio-based Algorithm Selection for SAT

SATzilla: Portfolio-based Algorithm Selection for SAT

Lin Xu, Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown

Department of Computer ScienceUniversity of British Columbia

Mainly based on [Xu, Hutter, Hoos, Leyton-Brown, JAIR, 2008]

SATzilla

• Excellent performance in SAT competitions [http://www.satcompetition.org/]

– 2 second places, one third place in 2003 SAT competition [Leyton-Brown, Nudelman, Andrew, McFadden & Shoham, 2003]

– 3 Gold, 1 Silver, 1 Bronze in 2007 SAT competition [Xu, Hutter, Hoos, & Leyton-Brown, 2007]

– 3 Gold, 2 Silver in 2009 SAT competition [Xu, Hutter, Hoos & Leyton-Brown, 2009]

• One gold in every category: Industrial/Handmade/Random

• Domain independent– MIPzilla, CSP, Planning …

Xu, Hutter, Hoos, and Leyton-Brown: SATzilla 2

Solvers

InstanceInstance Feature extractor

Algorithm selector

kevinlb

draw a box around "solvers" labeled "portfolio"

kevinlb

may want to cite here again for consistency

Impact on the SAT community

Encouraged research on portfolio-based solvers for SAT

Final 2011 SAT competition results[http://www.satcompetition.org/]

Xu, Hutter, Hoos, and Leyton-Brown: SATzilla

Category Random Handmade Industrial

Medalists

3Sppfolio (2) ppfolio seq

sparrow2011 sattime2011

eagleup march_rw

mphasesat_m

3S (3)Ppfolio (2)

ppfolio seq (2) clasp

glucose

glucose (2)glueminisat (2)

lingeling contrasat hack cirminisat hack mphasesat64

qutersat

Total medals 4/9 7/9 0/9

3

kevinlb

Should point out that in previous years, SATzilla was the only portfolio-based entrant

Outline

• Introduction• Related Work: – Algorithm selection

• SATzilla overview:– Building SATzilla – Running SATzilla

• SATzilla07 and beyond• Results • SATzilla since 2008• Conclusion


kevinlb

Pretty complicated for a 20-min talk. Do you need to tell us all of this?

Related work: algorithm selection

Select the best solver for each instance [Rice, 76]

– Regression-based• Linear regression (SATzilla)

[Leyton-Brown, Nudelman, Andrew, McFadden & Shoham, 2003; Xu, Hutter, Hoos, Leyton-Brown, 2007+]

• Matchbox [Stern et al., 2010]

– Classification-based• Logistic regression [Samulowitz & Memisevic, 2007]

• K-NN [O’Mahony & Hebrard, 2008]

• Decision tree [Guerri & Milano, 2004]


kevinlb

Again I prefer full citations here; e.g., it'll remind people that Frank is involved. And if Galen Andrew is in the audience, he'll see his name.

What is SATzilla?

• Portfolio-based algorithm selection based on empirical hardness models [Leyton-Brown et al., 2002;2009]

– Use regression models to predict each algorithm’s performance

– Select the predicted best

• Many new techniques for improving robustness– Pre-solvers – Backup solver– Solver subset selection


SATzilla: prerequisites

Identify a target distribution of instances

Select a set of candidate solvers, and measure runtime on a training set

Identify a set of features based on domain knowledge and collect feature values– Cheap to compute– Informative for prediction


SATzilla core: predictive modelsGoal: accurately predict solver performance based on cheaply

computable features

Supervised machine learning: learn function f: features → performance

– Input: <features, performance> pairs– Output: f


fw() = wT

Performance(y)Features ()

• 23.34• 7.21• …• …

Algorithm selector

8

SATzilla: selection based on models

• Predict each solver’s performance• Pick solver based on predictions

Feature extractor

Algorithm selector

InstanceInstanceInstanceInstance

Compute features

Make prediction

Run on selected solver


Improving robustness

• Solver subset selection– More candidate solvers can hurt– select subset of solvers that achieves best overall performance

• Pre-solvers– Solve some instances quickly without need for feature computation– Increase model robustness

• Backup solver – In case something goes wrong

Feature extractor

Algorithm selector

Dealing with imperfect predictions:


Algorithm selector

10

kevinlb

Be more specific about what can go wrong

Time to run SATzilla!


Step 1: pre-solving

Run pre-solvers for a fixed short amount of time– If solved, done

InstanceInstance

Feature extractor

Algorithm selector


Step 2: compute features

Run feature extractor to get a set of feature values– If solved, done– If anything fails, simply run backup solver

InstanceInstance

Feature extractor

Algorithm selector


Step 3: performance prediction

Predict performance of solvers and order them according to runtime predictions

Feature extractor

Algorithm selector


Step 4: run top-ranked solver

Run the top-ranked solver. If a run fails, run the next-ranked solver.


Feature extractor

Algorithm selector

InstanceInstance

SATzilla07: construction

Pre-solvers: manually selected

Backup solver: overall best

Model building:– Predict runtime– Hierarchical hardness models [Xu, Hoos, Leyton-Brown, 2007]

– Models handle “censored data” [Schmee & Hahn, 1979]

Solver subset selection: exhaustive searchXu, Hutter, Hoos, and Leyton-Brown: SATzilla 16

kevinlb

define best overall?

kevinlb

It doesn't come across clearly why the audience cares to make a distinction between what we did in 2007 and then what we did later. This needs to be sold in order for the talk to work narratively.

Improving SATzilla07 (1.)

Problem: manual pre-solver selection is not idealSolution: auto pre-solver selection based on validation


Feature extractor

Algorithm selector

kevinlb

well, it's not really suboptimal, if you manually select optimally. It's a lot of work, and it's fragile.

Improving SATzilla07 (2.-5.)

Problem: overall best solver best backup solverSolution: choose the best solver for instances not solved by pre-solvers and with problematic features

Problem: objective is not always runtimeSolution: predict and optimize true objective directly also enables cleaner use of incomplete solvers

Problem: exhaustive search for solver subset selection is infeasible for large portfolios Solution: local search for solver subset selection


kevinlb

This isn't so much a problem as an observation. And again, best overall is not clearly defined.

kevinlb

not clearly defined

SATzilla vs candidate solvers

HANDMADE



HANDMADE


Oracle

20


HANDMADE

Pre-solving

Feat. Comp.


Oracle

SATzilla07

21


HANDMADE


Oracle

SATzilla07

SATzilla07(improved)

22

SATzilla09 [Xu et al., 2009]

New techniques– New features: clause learning, …– Prediction of feature computation cost

Feature time predictor

InstanceInstance

Minimal cost feature

extractor

Results: 3 Gold, 2 Silver in 2009 SAT competition - Gold for Industrial (SAT) category - Gold for Handmade (UNSAT) category - Gold for Random (SAT+UNSAT) category


Feature extractor

Algorithm selector

kevinlb

There's room to name other categories

kevinlb

Doesn't come across clearly why you spent so much time on the 2007 improvements, and much less on the 2009 ones

New Results: Cost sensitive classification Goal: directly optimize the objective [Xu et al., 2011]

– Cost sensitive classification for every pair of candidates– Vote to decide the order of candidates

Improved performance over regression models, particularly with heterogeneous data sets

New SATzilla on SAT competition 2011


Random Gap closed: 80%

55%33%

OracleHandmade Industrial

24

kevinlb

Say "80% of gap closed"

Recent work: Combining algorithm selection and algorithm configuration

Problem: what if you only have one strong solver but it is highly parameterized?

“Hydra”: combine algorithm selection and algorithm configuration [Xu et al., 2010; 2011]

– Automatically find strong, uncorrelated parameter settings– Construct portfolio from these

Results: clear improvement for both SAT and MIP– 2 × to 31 × faster for MIP– 1.2 × to 73 × faster for SAT


kevinlb

e.g., CPLEX. Indeed, might mention we're working with ILOG?

Conclusion

• Portfolio-based algorithm selection can improve performance and robustness

• SATzilla-like portfolio approaches are the state of the art for many problems

• New directions since our JAIR paper:– better predictive models – automatic portfolio construction (“Hydra”)


SATzilla: Portfolio-based Algorithm Selection for SAT

Documents

Transcript of SATzilla: Portfolio-based Algorithm Selection for SAT