Frédéric Saubion LERIA Learning and Intelligent OptimizatioN Conference Autonomous Search.

Frédéric Saubion LERIA

Learning and Intelligent OptimizatioN Conference

Autonomous Search

Based on joint works on this topic with :

G. Di TolloA. Fiahlo Y. Hamadi F. Lardeux J. Maturana E. MonfroyM. SchoenauerM. Sebag

Learning and Intelligent OptimizatioN Conference

1. Introduction2. Main Ideas3. Taxonomy of AS4. Focus on examples5. Conclusion and challenges

Outline

IntroductionGeneric modeling tools for engineers

(Decision) VariablesDomains Constraints Mathematical

Model

Solver

Solving Constraint Optimization and Satifaction Problems

IntroductionMap coloring problem

Satifaction Problems

Introduction

Optimization Problems

Travelling Salesman Problem : finda round trip across cities with minimal cost

IntroductionSearch landscapes are difficult to explore

Many variablesComplex constraints

Problems are more and more complex

IntroductionSearch landscapes are difficult to explore

Exploration vs. Exploitation Balance

Problems are more and more complex

Boolean Variable {0,1}

IntroductionAn illustrative example : solving SAT

SAT CNF instance

Devising more and more complex Solving algorithms

ba

ca

cba

Litterals

Clauses

Assignment

(1 0 0)

Introduction


How to explore the binary search space (assignments) to find a solution ?

Introduction


Use Local Search

Introduction


Basic Local Search

0 1 0 1 1

Choose a random initial assignment

Introduction


Basic Local Search

Compute the number of true and false clauses

Introduction


Basic Local Search

Try to improve by changing a value (flip)

0 1 0 1 1

0 1 1 1 1

Move to a neighbor

Introduction


Basic Local Search

Until finding a solution

Introduction


Short overview of the story : a first greedy version GSATBart Selman, Hector J. Levesque, David G. Mitchell: A New Method for Solving Hard Satisfiability Problems.AAAI 1992: 440-446

A first boat for binary seas

Introduction


10 1 0 1 1 0 0

1 1 1 1

0 0 0

Problem : Many possible moves (many variables)

Introduction


Restrict neighborhoodSelect a false clause C

fda a b c d e f g

0 1 0 1 1 0 0

Introduction


Get stuck in local optima

Introduction


Add pertubationsSelect a false clause C

With a random probability p Perform a random flip for C

With (1-p) Select the variable with best IMP Perform best move

If solution then stopElse go on

Parameter !

Introduction


Use restarts

False Clauses

Iterations

Parameter !

Introduction


WalkSAT : adding a noise and random restart

Henry A. Kautz, Bart Selman: Noise Strategies for Improving Local Search..AAAI 1994

Introduction


How to break ties ?

0 1 0 1 1 0 0

+3 +3

+3

Introduction


Add more sophisticated heuristics Compute the age of the variableIf the best variable is not the most recent then flipElseWith a random probability p’

Perform a random flip the second best

With (1-p’) Flip the best

Parameter !

Introduction


Novelty : using more strategies to perform improvements (age of the variable)D.A. McAllester, B. Selman and H. Kautz. Evidence for invariant in local search.In Proceedings of AAAI-97, AAAI Press 1997, pages 321-326.

Introduction


And improvements go on …

Novelty +,Novelty ++, …, TNM, Sattime…

Introduction


Captain Jack : many indicators and thus selection strategies

Dave A. D. Tompkins, Adrian Balint, Holger H. Hoos: Captain Jack: New Variable Selection Heuristics in Local Search for SAT. SAT 2011: 302-316

IntroductionAdding more parameters and heuristics


More flexible algorithms Fit to different instances

Set parameters/heuristics values Understand the behavior

John Rice. The algorithm selection problem. Technical Report CSD-TR152, Computer science department, Purdue University, 1975.

The Algorithm Selection Problem

Main ideas

Tuning the parameters

Related Questions

Main ideas

Using several algorithms for solving a classof problems


Related Questions

Main ideas

Adjusting the parametersof one algorithm


Main Objectives

Main ideas

Need for more autonomous solving tools

Increasing number of works in this trend : LION, Specialsessions in EA conferences (GECCO,…) …


Ideas for More Autonomous Solvers

How to use an algorithm that includes

•Many parameters

•Many possible heuristics or components

Ideas


Ideas for More Autonomous Solvers

How to use an algorithm that include

•Many parameters

•Many possible heuristics or components

How to automate all these choices ?

Ideas


Off-line Automated Tuning

Ideas

Run your solver on some problems

Experiment automatically parameters values



Ideas

Run your solver on new problems with these parameters values



Ideas

Question : Generality of the parameters ?


On-line Parameter Control

Ideas

Try to react during the resolution by changing the parameter



Ideas

Example : try to increase some parameter when possible



Ideas

Question : How to react efficiently ?


Hyper Heuristics

Ideas

Combine basic solving heuristics


Hyper Heuristics

Ideas

Get new solvers


Hyper Heuristics

Ideas

Question : How to learn the suitable solver ?


Portfolios Based Solvers

Ideas

Use different types of solvers



Ideas

Learn how to select the right solver for a given problem



Ideas

Question : Reliability of the learning process ?

Evolutionary ComputationA. E. Eiben, R. Hinterding, and Z. Michalewicz. Parameter control in

evolutionary algorithms. IEEE Trans. Evolutionary Computation,

3(2) :124141, 1999.

Reactive Search Battiti R, Brunato M, Mascia F (2008) Reactive Search and Intelligent

Optimization, ORCS interf., vol 45. Springer

Hyper-HeuristicsBurke EK, Hyde M, Kendall G, Ochoa G, Ozcan E, Woodward J

Handbook of Meta-heuristics, chap A Classification of Hyper-heuristics Approaches

Portfolios methodsGagliolo M, Schmidhuber J (2008) Algorithm selection as a bandit problem with unbounded losses. Tech. rep., Tech.

report IDSIA - 07

Why introducing the concept of Autonomous Search ?

Taxonomy

Taxonomies

Classification : Solving

Solving MethodsTree-Based Search

MetaheuristicsSLS EA

On-

line

Off-

line Au

to

Complete/incomplete search,Model representationOther optimization paradigms (e.g., ACO )

Taxonomy

Classification : Parameters

Solving MethodsTree-Based Search SLS EA

Para

met

er s

etting

met

hod

On-

line

Off-

line Au

to

Parameter type

StructuralBehavioral

Numerical/discrete valuesComponents of the solverVs. Configuration of the solver

Taxonomy

Classification : Settings

Solving MethodsTree-Based Search EA

Para

met

er s

etting

met

hod

On-

line

Off-

line Au

to

Parameter type

•Experiment-based•Feedback Control •Measures and learning techniques(reinforcement learning, statistical learning, case-base reasonning…)

Taxonomy

Related Approaches



Para

met

er s

etting

met

hod

On-

line

Off-

line Au

to

Parameter type


Taxonomy

Parameter Setting in Evolutionary Computation

Related Approaches



Para

met

er s

etting

met

hod

On-

line

Off-

line Au

to

Parameter type


Taxonomy

Parameter Setting

ParameterTuning

ParameterControl

Deterministic

Adaptive Self-adaptive

Related Approaches



Para

met

er s

etting

met

hod

On-

line

Off-

line Au

to

Parameter type


Taxonomy

Optimization of algorithms (automated tuning)

Related Approaches



Para

met

er s

etting

met

hod

On-

line

Off-

line Au

to

Parameter type


Taxonomy

Optimization of algorithms (automated tuning)

SLS Based (ParamILS)Frank Hutter, Holger H. Hoos, Kevin Leyton-Brown, Thomas Stützle: ParamILS: An Automatic Algorithm Configuration Framework. J. Artif. Intell. Res. (JAIR) 36: 267-306 (2009)

GA Based (Revac)Volker Nannen, A. E. Eiben: Efficient relevance estimation and value calibration of evolutionary algorithm parameters. IEEE Congress on Evolutionary Computation 2007: 103-110

Racing techniquesMauro Birattari, Thomas Stützle, Luis Paquete, Klaus Varrentrapp: A Racing Algorithm for Configuring Metaheuristics. GECCO 2002: 11-18

…

Related Approaches



Para

met

er s

etting

met

hod

On-

line

Off-

line Au

to

Parameter type


Taxonomy

Reactive SearchLearning for SLS

Related Approaches



Para

met

er s

etting

met

hod

On-

line

Off-

line Au

to

Parameter type


Taxonomy

Hyper heuristics

Hyper Heuristics

Taxonomy

Two possible views

• heuristics to choose heuristics

• heuristics to generate heuristics

Burke EK, Hyde M, Kendall G, Ochoa G, Ozcan E, Woodward JHandbook of Meta-heuristics, A Classification of Hyper-heuristicsApproaches

Proposing a general view of AS



Para

met

er s

etting

met

hod

On-

line

Off-

line Au

to

Parameter type


Taxonomy

Autonomous Search

Autonomous Search Genesis

Taxonomy

Gather works from the CSP solving community

Workshop “Autonomous Search” CP 2007, Providence (RI)

Autonomous Search Genesis

Taxonomy

Identify common concepts, goals andchallenges for future works


Requirements for an Autonomous Search SystemParameters

Taxonomy

Modify its internal components

•Parameters•Fine grain heuristics•Coarse grain solving techniques•Model representation

React to external forces and opportunities

•Search landscape analysis (quality, diversity,...)•External knowledge (prediction models, rules, ...)


CS Related Areas

Taxonomy

Solving Techniques Point of View

•Constraint Programming•Operation Research•Evolutionary Computation

Adjustment Techniques Point of View

•Reinforcement Learning•Statistical Learning•Information Theory


CS Related Areas

Taxonomy

Solving Techniques Point of View

•Constraint Programming•Operation Research•Evolutionary Computation

Adjustment Techniques Point of View

•Reinforcement Learning•Statistical Learning•Information Theory

Not limited to…

Examples of works



Para

met

er s

etting

met

hod

On-

line

Off-

line Au

to

Parameter type

StructuralBehavioralExamples

Examples of works

Examples



Para

met

er s

etting

met

hod

On-

line

Off-

line Au

to

Parameter type


Manual Empirical TuningDeciding the Size of a Tabu List

Mazure B, Sais L, Gregoire E, Tabu search for sat. In : AAAI/IAAI,pp 281285, 1997

Examples of works

Examples



Para

met

er s

etting

met

hod

On-

line

Off-

line Au

to

Parameter type


Automated Parameter Tuning(SLS based)Hutter F, Hoos H, Stutzle T Automatic algorithm configuration based on local search. AAAI 2007

Examples of works

Examples



Para

met

er s

etting

met

hod

On-

line

Off-

line Au

to

Parameter type


Portfolios ApproachesFeatures based linear regression and classiers

Xu L, Hutter F, Hoos HH, Leyton-Brown K Satzilla : portfolio-based algorithm selection for sat. JAIR 2008

Examples of works

Examples



Para

met

er s

etting

met

hod

On-

line

Off-

line Au

to

Parameter type


Learning Combinations of Well-known Heuristics

Epstein S, Freuder E, Wallace R Learning to support constraint programmers. Comput Intell 2005

Examples of works

Examples



Para

met

er s

etting

met

hod

On-

line

Off-

line Au

to

Parameter type


Discovering heuristics(variable selection in SAT SLS)

Alex S. Fukunaga : Automated Discovery of Local Search Heuristics forSatisability Testing. Evolutionary Computation 2008

Examples of works

Examples



Para

met

er s

etting

met

hod

On-

line

Off-

line Au

to

Parameter type


Automated TuningAdjusting the size of a Tabu List

R. Battiti, G. Tecchiolli : The Reactive Tabu Search. INFORMS Journal on Computing 6(2): 126-140 (1994)

Examples of works

Examples



Para

met

er s

etting

met

hod

On-

line

Off-

line Au

to

Parameter type


Controlling Variable Orderings and Values Selection inSearch Heuristics

Boussemart F, Hemery F, Lecoutre C, Sais L Boosting systematic search by weighting constraints. ECAI2004 2004

Examples of works

Examples



Para

met

er s

etting

met

hod

On-

line

Off-

line Au

to

Parameter type


Adaptive Genetic Algorithms

A.E. Eiben, Z. Michalewicz, M. Schoenauer, J. E. SmithParameter Control in Evolutionary Algorithms.In Parameter Setting in Evolutionary Algorithms

2007

Focus on an example : how to design an AS system ?

Generic Evolutionary Algorithm for Constraint Satisfaction Problem

Focus

Population

Apply variation operator

New Population

Solution ?

Focus on an example

Generic Search Algorithm for ConstraintSatisfaction Problem

Focus

Configuration

Apply variation operator

New configuration

Solution ?

Question : How to choose the suitable operator at each iteration ?

Focus on an example

Idea

Associate a probability of application to each operator (parameter)

Select an operator according to thisprobability scheme

Focus

Focus on an example

Question : how to set the probabilities(parameters of the algorithm) ?

Focus

Focus on an example

Use a principled approach to tune yourparameter

•Search in the parameters space

•Assess the performance of each setting through runs of the algorithm on selected Instances

ParamILS, REVAC, F-Race …

Focus

Focus on an example

Second idea:

Control the probability during the run

•Evaluate the performance of each operator after application

•Adjust the parameters according to the performances

Focus

General process for control(Automated Operator Selection)

Focus

Focus on an example

What are the suitable criteria ?

-Quality -Fitness diversity -Genotypic diversity -Time -…

Focus

Different performnce mesearues

Focus

1 2 3 4 5 6 70

2

4

6

8

10

12

14

16

Op1Op2

Sliding Windows

Mean or Max ?

How to measure the impact ?

Focus on an example

What is the performance of the operators ?

•Fix a search policy•Dynamic policy •Values against rank •…

Focus

Focus on an example

What is the performance of the operators ?

Fix a search policy

Focus

Focus on an example

What is the performance of the operators ?No values :

Pareto rank of the operatorsArea under the curve

Focus

Estimating efficience of operators

How to reward the operators ?

Proportionally to their performance

Focus


Using UCB (Upper Confidence Bound)(reinforcement learning technique)

Exploration + Exploitation of the operators

Choosing the operator having the best UCB

Focus

to

ktk

to n

nCr

,

,

,

)log(


Warning : UCB converge asymptotically toGain for the MAB

But here we have dynamic changes

Use of statistical test to restart learning.

Focus

Different selection processes

Focus

UCB PM Uniform0

2

4

6

8

10

12

14

16

18

Op1Op2Op3Op4

Instances ?

Comparisons ?

Induced new parameters ? ?

Reliability ?

How to assess the performances of your system ?

Focus

What’s next ?



Para

met

er s

etting

met

hod

On-

line

Off-

line Au

to

Parameter type


Focus

Conclusion

Many different possible approaches

Guidelines for designing new autonomousSolvers

Off-line/On-lineBehavioural parameters/componentsControl of the efficient heuristics/discovering new heuristics

…

Conclusion

Challenges

Comparing performances

•Autonomous vs. ad-hoc

•Off-line Tuning vs. On-line control

•Representative benchmarking

Conclusion

Challenges

Comparing performances

•Methodologies for comparisons

•New competitions Chesc (Cross-domain Heuristic Search Challenge)

G. Ochoa and her team

•Related to No Free-Lunch TheoremsMore reliable on more problems

Conclusion

Challenges

Parameters induced by the AS system

•Abstract parameters should be more easy to control (e.g., EvE balance)

•New parameters should be less sensitive than original ones

•Fewer paramaters are easier to adjust

Conclusion

Challenges

Learning

•Interactions solving-learning

•Improving learning off-line

•Short term (react) vs. long term (prediction)

•Continuous search (Arbelaez, Hamadi & Sebag)

Conclusion

Challenges

Distributed and parallel computing

•Improving algorithm’s space exploration

•Sharing information on parameters

•Sharing information on problems

Conclusion

Challenges

Towards more generic on-line control tools

•Identify generic control techniques andmeasures

•Control various components type (behavioral parameters, objective functions, heuristics…)

Conclusion

Of course all LION Proceedings …

Some books to read

Conclusion

And ;-)

Conclusion

So

Sorry for missing references and works

Conclusion

I will not forget important works and references I will not forget important works and references I will not forget important works and references I will not forget important works and references I will not forget important works and references I will not forget important works and references I will not forget important works and references I will not forget important works and references

So

Questions

Conclusion

I will not forget important works and references I will not forget important works and references I will not forget important works and references I will not forget important works and references I will not forget important works and references I will not forget important works and references I will not forget important works and references I will not forget important works and references

Frédéric Saubion LERIA Learning and Intelligent OptimizatioN Conference Autonomous Search.

Documents

Transcript of Frédéric Saubion LERIA Learning and Intelligent OptimizatioN Conference Autonomous Search.