slides from the defense

Automating the Configuration of Algorithms for Solving Hard Computational Problems Ph.D. Thesis Defence Frank Hutter Supervisory committee: Prof. Prof. Prof. Prof.

Holger Hoos (supervisor) Kevin Leyton-Brown (co-supervisor) Kevin Murphy (co-supervisor) Alan Mackworth

University Examiners: Prof. Michael Friedlander (CS) Prof. Lutz Lampe (ECE)

External Examiner: Prof. ? Chair: Prof. John Nelson (Forestry)

Parameters in Algorithms

Most algorithms have parameters I

Decisions that are left open during algorithm design – numerical parameters (e.g., real-valued thresholds) – categorical parameters (e.g., which heuristic to use)

2

Parameters in Algorithms

Most algorithms have parameters I

Decisions that are left open during algorithm design – numerical parameters (e.g., real-valued thresholds) – categorical parameters (e.g., which heuristic to use)

I

Set to maximize empirical performance

2

Real-world example for parameterized algorithms: commercial optimization tool CPLEX I

State of the art for mixed integer programming (MIP)

3

Real-world example for parameterized algorithms: commercial optimization tool CPLEX I I

State of the art for mixed integer programming (MIP) Large user base – Over 1 300 corporations and over 1 000 universities

3



I

63 parameters that affect search trajectory

3



I

63 parameters that affect search trajectory “Integer programming problems are more sensitive to specific parameter settings, so you may need to experiment with them.” [CPLEX 10.0 user manual, page 130]

3



I


I

“Experiment with them”

3



I


I

“Experiment with them” – Perform manual optimization in 63-dimensional space – Complex, unintuitive interactions between parameters

3



I


I

“Experiment with them” – Perform manual optimization in 63-dimensional space – Complex, unintuitive interactions between parameters – Humans are not good at that

3



I


I

“Experiment with them” – Perform manual optimization in 63-dimensional space – Complex, unintuitive interactions between parameters – Humans are not good at that developed the first automated tools for this type of problem 3

Automated Algorithm Configuration

Automate the setting of algorithm parameters I

Eliminate most tedious part of algorithm design and end use

I

Save development time

I

Improve performance

4

Automated Algorithm Configuration

Automate the setting of algorithm parameters I

Eliminate most tedious part of algorithm design and end use

I

Save development time

I

Improve performance

I

First to consider the general problem, in particular many categorical parameters – E.g. 50/63 CPLEX parameters are categorical Algorithm configuration

4

Main Contribution of this thesis

Comprehensive study of the algorithm configuration problem

5


Comprehensive study of the algorithm configuration problem I

Empirical analysis of configuration scenarios

5




I

Two fundamentally different solution approaches

5




I


I

Demonstrated practical relevance of algorithm configuration

5




I

Two fundamentally different solution approaches – 1st and 2nd approach to configure algorithms with many categorical parameters

I


5




I


I

Demonstrated practical relevance of algorithm configuration – CPLEX: up to 23-fold speedup

5




I


I

Demonstrated practical relevance of algorithm configuration – CPLEX: up to 23-fold speedup – SAT solver: 500-fold speedup for software verification

5

Outline

1. Problem Definition & Intuition 2. Model-Free Search for Algorithm Configuration 3. Model-Based Search for Algorithm Configuration 4. Conclusions

6

Outline


7

Algorithm Configuration as Function Optimization Deterministic algorithm with continuous parameters – “Blackbox function” f : Rn → R – Can query function at arbitrary points θ ∈ Rn Find minn f (θ) θ∈R

8


Randomized algorithm with continuous parameters – For each θ: distribution Dθ – Optimize statistical parameter τ (e.g., expected value)

8


Randomized algorithm with continuous parameters – For each θ: distribution Dθ – Optimize statistical parameter τ (e.g., expected value) – Can sample from distribution Dθ at arbitrary points θ ∈ Θ Find minn τ (Dθ ) θ∈R

8

Algorithm Configuration: General Case

Difference to “standard” blackbox optimization I

Categorical parameters

9


Difference to “standard” blackbox optimization I I

Categorical parameters Distribution of costs – across multiple repeated runs for randomized algorithms – across problem instances

9


Difference to “standard” blackbox optimization I I

Categorical parameters Distribution of costs – across multiple repeated runs for randomized algorithms – across problem instances

I

Can terminate unsuccessful runs early

9

Outline

1. Problem Definition & Intuition 2. Model-Free Search for Algorithm Configuration ParamILS: Iterated Local Search in Configuration Space “Real-World” Applications of ParamILS 3. Model-Based Search for Algorithm Configuration 4. Conclusions

10

Outline


11

Simple manual approach for configuration

Start with some parameter configuration

12


Start with some parameter configuration Modify a single parameter

12


Start with some parameter configuration Modify a single parameter if results on benchmark set improve then keep new configuration

12


Start with some parameter configuration repeat Modify a single parameter if results on benchmark set improve then keep new configuration until no more improvement possible (or “good enough”)

12


Start with some parameter configuration repeat Modify a single parameter if results on benchmark set improve then keep new configuration until no more improvement possible (or “good enough”) Manually-executed local search

12

The ParamILS Framework

Iterated Local Serach in parameter configuration space: Choose initial parameter configuration θ Perform subsidiary local search on θ

13


Iterated Local Serach in parameter configuration space: Choose initial parameter configuration θ Perform subsidiary local search on θ While tuning time left: | θ0 := θ | Perform perturbation on θ | Perform subsidiary local search on θ

13


Iterated Local Serach in parameter configuration space: Choose initial parameter configuration θ Perform subsidiary local search on θ While tuning time left: | θ0 := θ | Perform perturbation on θ | Perform subsidiary local search on θ | | Based on acceptance criterion, keep θ or revert to θ := θ0 |

13


Iterated Local Serach in parameter configuration space: Choose initial parameter configuration θ Perform subsidiary local search on θ While tuning time left: | θ0 := θ | Perform perturbation on θ | Perform subsidiary local search on θ | | Based on acceptance criterion, keep θ or revert to θ := θ0 | | b With probability p randomly pick new θ restart

Performs biased random walk over local optima 13

Instantiations of ParamILS Framework

How to evaluate each configuration? I

BasicILS(N): perform fixed number of N runs to evaluate a configuration θ – Blocking: use same N (instance, seed) pairs for each θ

14




I

FocusedILS: adaptive choice of N(θ) – small N(θ) for poor configurations θ – large N(θ) only for good θ

14




I

FocusedILS: adaptive choice of N(θ) – small N(θ) for poor configurations θ – large N(θ) only for good θ – typically outperforms BasicILS

14

Empirical Comparison to Previous Configuration Procedure

CALIBRA system

[Adenso-Diaz & Laguna, ’06]

I

Based on fractional factorial designs

I

Limited to continuous parameters

I

Limited to 5 parameters

15

Empirical Comparison to Previous Configuration Procedure

CALIBRA system

[Adenso-Diaz & Laguna, ’06]

I

Based on fractional factorial designs

I

Limited to continuous parameters

I

Limited to 5 parameters

Empirical comparison I

FocusedILS typically did better, never worse

I

More importantly, much more general

15

Adaptive Choice of Cutoff Time

I

Evaluation of poor configurations takes especially long

16


I I

Evaluation of poor configurations takes especially long Can terminate evaluations early I I

Incumbent solution provides bound Can stop evaluation once bound is reached

16


I I

Evaluation of poor configurations takes especially long Can terminate evaluations early I I

I

Incumbent solution provides bound Can stop evaluation once bound is reached

Results – Provably never hurts – Sometimes substantial speedups (factor 10)

16

Outline


17

Configuration of ILOG CPLEX I

Recall: 63 parameters, 1.78 × 1038 possible configurations

I

Ran FocusedILS for 2 days on 10 machines

18



I


I

Compared against default “A great deal of algorithmic development effort has been devoted to establishing default ILOG CPLEX parameter settings that achieve good performance on a wide variety of MIP models.” [CPLEX 10.0 user manual, page 247]

18



I


I

Compared against default “A great deal of algorithmic development effort has been devoted to establishing default ILOG CPLEX parameter settings that achieve good performance on a wide variety of MIP models.” [CPLEX 10.0 user manual, page 247]

4

10

3

10

Auto−tuned

2

10

1

10

0

10

−1

10

−2

10

−2

10

−1

10

0

10

1

10

Default

2

10

3

10

4

10

Combinatorial auctions: 7-fold speedup 18



I


I

Compared against default “A great deal of algorithmic development effort has been devoted to establishing default ILOG CPLEX parameter settings that achieve good performance on a wide variety of MIP models.” [CPLEX 10.0 user manual, page 247] 4

4

10

10

3

3

10

2

10

10

Auto−tuned

Auto−tuned

10

1

10

0

10

2

1

10

0

10

−1

−1

10

10

−2

−2

10

10

−2

10

−1

10

0

10

1

10

Default

2

10

3

10

4

10

Combinatorial auctions: 7-fold speedup

−2

10

−1

10

0

10

1

10

Default

2

10

3

10

4

10

Mixed integer knapsack: 23-fold speedup 18

Configuration of SAT Solver for Verification SAT (propositional satisfiability problem) – Prototypical N P-hard problem – Interesting theoretically and in practical applications

19


Formal verification – Bounded model checking – Software verification – Recent progress based on SAT solvers

19


Formal verification – Bounded model checking – Software verification – Recent progress based on SAT solvers

Spear, tree search solver for industrial SAT instances – 26 parameters, 8.34 × 1017 configurations

19

Configuration of SAT Solver for Verification I


20

Configuration of SAT Solver for Verification I I

Ran FocusedILS for 2 days on 10 machines Compared to manually-engineered default – 1 week of performance tuning – competitive with the state of the art

20

Configuration of SAT Solver for Verification Ran FocusedILS for 2 days on 10 machines Compared to manually-engineered default

I I

SPEAR, optimized for IBM−BMC (s)

– 1 week of performance tuning – competitive with the state of the art

4

10

3

10

2

10

1

10

0

10

−1

10

−2

10

−2

10

−1

0

1

2

3

4

10 10 10 10 10 10 SPEAR, original default (s)

IBM Bounded Model Checking: 4.5-fold speedup 20

Configuration of SAT Solver for Verification Ran FocusedILS for 2 days on 10 machines Compared to manually-engineered default

I I

4

10

SPEAR, optimized for SWV (s)

SPEAR, optimized for IBM−BMC (s)

– 1 week of performance tuning – competitive with the state of the art

3

10

2

10

1

10

0

10

−1

10

−2

4

10

3

10

2

10

1

10

0

10

−1

10

−2

10

10 −2

10

−1

0

1

2

3

4


IBM Bounded Model Checking: 4.5-fold speedup

−2

10

−1

0

1

2

3

4


Software verification: 500-fold speedup won 2007 SMT competition 20

Other Fielded Applications of ParamILS I

SAPS, local search for SAT 8-fold and 130-fold speedup

21



I

SAT4J, tree search for SAT 11-fold speedup

21



I


I

GLS+ for Most Probable Explanation (MPE) problem > 360-fold speedup

21



I


I


I

Applications by others – Protein folding [Thatchuk, Shmygelska & Hoos ’07] – Time-tabling [Fawcett, Hoos & Chiarandini ’09] – Local Search for SAT [Khudabukhsh, Xu, Hoos, & Leyton-Brown ’09]

21



I


I


I

Applications by others – Protein folding [Thatchuk, Shmygelska & Hoos ’07] – Time-tabling [Fawcett, Hoos & Chiarandini ’09] – Local Search for SAT [Khudabukhsh, Xu, Hoos, & Leyton-Brown ’09] demonstrates versatility & maturity

21

Outline

1. Problem Definition & Intuition 2. Model-Free Search for Algorithm Configuration 3. Model-Based Search for Algorithm Configuration State of the Art Improvements for Stochastic Blackbox Optimization Beyond Stochastic Blackbox Optimization 4. Conclusions

22

Model-Based Optimization: Motivation

Fundamentally different approach for algorithm configuration I I

So far: discussed local search approach Now: alternative choice, based on predictive models

23



So far: discussed local search approach Now: alternative choice, based on predictive models – Model-based optimization was less well developed emphasis on methodological improvements

23



So far: discussed local search approach Now: alternative choice, based on predictive models – Model-based optimization was less well developed emphasis on methodological improvements

I

In then end: state-of-the-art configuration tool

23

Outline


24

Model-Based Deterministic Blackbox Optimization (BBO) EGO algorithm

[Jones, Schonlau & Welch ’98]

30 . . True function . .

response y

25

20

15

10

5

0

−5

0

0.2

0.4

0.6

0.8

1

parameter x 25



1. Get response values at initial design points

30 .

response y

25

True function Function evaluations .

20

15

10

5

0

−5

0

0.2

0.4

0.6

0.8

1

parameter x 25



1. Get response values at initial design points

30 . . . Function evaluations .

response y

25

20

15

10

5

0

−5

0

0.2

0.4

0.6

0.8

1

parameter x 25



1. Get response values at initial design points 2. Fit a model to the data

30 DACE mean prediction DACE mean +/− 2*stddev . Function evaluations .

response y

25

20

15

10

5

0

−5

0

0.2

0.4

0.6

0.8

1

parameter x 25



1. Get response values at initial design points 2. Fit a model to the data 3. Use model to pick most promising next design point

30 DACE mean prediction DACE mean +/− 2*stddev . Function evaluations EI (scaled)

response y

25

20

15

10

5

0

−5

0

0.2

0.4

0.6

0.8

1

parameter x 25

Model-Based Deterministic Blackbox Optimization (BBO) EGO algorithm 1. 2. 3. 4.


Get response values at initial design points Fit a model to the data Use model to pick most promising next design point Repeat 2. and 3. until time is up

30 DACE mean prediction DACE mean +/− 2*stddev True function Function evaluations EI (scaled)

25

response y

20

15

10

5

0

−5

0

0.2

0.4

0.6

0.8

1

parameter x 25

Model-Based Deterministic Blackbox Optimization (BBO) EGO algorithm 1. 2. 3. 4.


Get response values at initial design points Fit a model to the data Use model to pick most promising next design point Repeat 2. and 3. until time is up 30

30 DACE mean prediction DACE mean +/− 2*stddev True function Function evaluations EI (scaled)

25

20

response y

20

response y

DACE mean prediction DACE mean +/− 2*stddev True function Function evaluations EI (scaled)

25

15

10

15

10

5

5

0

0

−5

−5

0

0.2

0.4

0.6

parameter x

First step

0.8

1

0

0.2

0.4

0.6

0.8

1

parameter x

Second step 25

Stochastic Blackbox Optimization (BBO): State of the Art Extensions of EGO algorithm for stochastic case – Sequential Parameter Optimization (SPO) [Bartz-Beielstein, Preuss, Lasarczyk, ’05-’09]

– Sequential Kriging Optimization (SKO) [Huang, Allen, Notz & Zeng, ’06]

26



Application domain for stochastic BBO I

Randomized algorithms with continuous parameters

I

Optimization for single instances

26



Application domain for stochastic BBO I

Randomized algorithms with continuous parameters

I

Optimization for single instances

Empirical Evaluation I

SPO more robust 26

Outline


27

Improvements for stochastic BBO I: Studied SPO components I

Improved component: “intensification mechanism” – Increase N(θ) similarly as in FocusedILS – Improved robustness

28



II: Better Models I

Compared various probabilistic models – Model SPO uses – Approximate Gaussian process (GP) – Random forest (RF)

28



II: Better Models I


I

New models much better – Resulting configuration procedure: ActiveConfigurator – Improved state of the art for model-based stochastic BBO

28



II: Better Models I


I

New models much better – – – –

Resulting configuration procedure: ActiveConfigurator Improved state of the art for model-based stochastic BBO Randomized algorithm with continuous parameters Optimization for single instances 28

Outline


29

Extension I: Categorical Parameters Models that can handle categorical inputs I I

Random forests: out of the box Extended (approximate) Gaussian processes – new kernel based on weighted Hamming distance

30



Application domain I

Algorithms with categorical parameters

I

Single instances

30



Application domain I


I

Single instances

Empirical evaluation I

ActiveConfigurator outperformed FocusedILS 30

Extension II: Multiple Instances Models incorporating multiple instances I I

Can still learn probabilistic models of algorithm performance Model inputs: I I

algorithm parameters instance features

31




General algorithm configuration I


I

Multiple instances

31




General algorithm configuration I


I

Multiple instances

Empirical evaluation I

ActiveConfigurator never worse than FocusedILS

I

Overall: model-based approaches very promising

31

Outline


32

Conclusions Algorithm configuration I

Is a high-dimensional optimization problem – Can be solved by automated approaches – Sometimes much better than by human experts

33



I

Can cut development time & improve results

33



I


Scaling to very complex problems allows us to I I

Build very flexible algorithm frameworks Apply automated tool to instantiate framework Generate custom algorithms for different problem types

33



I


Scaling to very complex problems allows us to I I

Build very flexible algorithm frameworks Apply automated tool to instantiate framework Generate custom algorithms for different problem types

Blackbox approaches I

Very general

I

Can be used to optimize your parameters 33


Comprehensive study of the algorithm configuration problem

34




I


I


34




I


I


34




I

Two fundamentally different solution approaches – Model-free Iterated Local Search approach

I


34




I

Two fundamentally different solution approaches – Model-free Iterated Local Search approach – Improved & Extended Sequential Model-Based Optimization

I


34




I


I


34




I


I

Demonstrated practical relevance of algorithm configuration – CPLEX: up to 23-fold speedup – SPEAR: 500-fold speedup for software verification

34



Empirical analysis of configuration scenarios [Ready for submission]

I

Two fundamentally different solution approaches – Model-free Iterated Local Search approach [AAAI’07] – Improved & Extended Sequential Model-Based Optimization [GECCO’09; EMAA’09]

I

Demonstrated practical relevance of algorithm configuration – CPLEX: up to 23-fold speedup [JAIR’09] – SPEAR: 500-fold speedup for software verification [FMCAD’07]

34

Important Directions for the Next Few Years

I

Improve configuration procedures from practical point of view – Mixed categorical/numerical optimization – Make easier to use off the shelf

35


I


I

More sophisticated model-based methods – Use model to select most informative instance – Use model to select best cutoff time – Per-instance setting of parameters

35


I


I

More sophisticated model-based methods – Use model to select most informative instance – Use model to select best cutoff time – Per-instance setting of parameters

I

Explore other fields of applications

35

Thanks to I

Supervisory committee – – – –

I

Further collaborators – – – – – – –

I

Holger Hoos (supervisor) Kevin Leyton-Brown (co-supervisor) Kevin Murphy (co-supervisor) Alan Mackworth

Domagoj Babi´c Thomas Bartz-Beielstein Youssef Hamadi Alan Hu Thomas St¨ utzle Dave Tompkins Lin Xu

LCI and BETA lab faculty and students

36