Advanced Analytics:

Plant a (decision) TREE and save the world*!

Vivek NairNorth Carolina State University

vivekaxl@gmail.comvivekaxl.com

* Configure software using less resources

Most Valuable Point

“Information is a source of learning. But unless it is organized, processed, and available to the right people in a format for decision making, it is a burden, not a benefit” -- Dr. William Pollard

Decision Trees - Use Cases

TAR(ZAN)2 [1]

2002

[1] Menzies, Tim, and Ying Hu. "Just enough learning (of association rules): the TAR2 “Treatment” learner." Artificial Intelligence Review 25.3 (2006): 211-229.

TAR(ZAN)2 Learns Small Theories

Problem: Find picture on a page from 11 features

TAR(ZAN)2 Learns Small Theories

Problem: Find picture on a page from 11 features

TAR(ZAN)2 Learns Small Theories

Find picture on a page from 11 features

TAR(ZAN)2 Learns Small Theories

Find picture on a page from 11 features

(34 ≤ height ≤ 86) ⋀ (3.9 ≤ mean_tr < 9.5)

TAR(ZAN)2 Learns Small Theories

Find picture on a page from 11 features

(34 ≤ height ≤ 86) ⋀ (3.9 ≤ mean_tr < 9.5)

Decision Trees - Use Cases

TAR(ZAN)2 [1]

2002

[1] Menzies, Tim, and Ying Hu. "Just enough learning (of association rules): the TAR2 “Treatment” learner." Artificial Intelligence Review 25.3 (2006): 211-229.

Decision Trees - Use Cases

TAR(ZAN)2 [1]

2002SWAY[2]

2016

XTREE[4]

2017Performance Optimization[3]

2017

[1] Menzies, Tim, and Ying Hu. "Just enough learning (of association rules): the TAR2 “Treatment” learner." Artificial Intelligence Review 25.3 (2006): 211-229.[2] Nair et al. "An (accidental) exploration of alternatives to evolutionary algorithms for sbse." SSBSE- 2016.[3] Guo et al. “Variability-aware performance prediction: A statistical learning approach." ASE-2013.[4] Krishna et al.. "Less is more: Minimizing code reorganization using XTREE." IST-2017

Decision Trees - Use Cases

Optimization

PlanningSoftware Variability

[1] Menzies, Tim, and Ying Hu. "Just enough learning (of association rules): the TAR2 “Treatment” learner." Artificial Intelligence Review 25.3 (2006): 211-229.[2] Nair et al. "An (accidental) exploration of alternatives to evolutionary algorithms for sbse." SSBSE- 2016.[3] Guo et al. “Variability-aware performance prediction: A statistical learning approach." ASE-2013.[4] Krishna et al.. "Less is more: Minimizing code reorganization using XTREE." IST-2017

TAR(ZAN)2 [1]

2002SWAY[2]

2016

XTREE[4]

2017Performance Optimization[3]

2017

Decision Trees - Use Cases

Optimization

PlanningSoftware Variability

TAR(ZAN)2 [1]

2002SWAY[2]

2016

XTREE[4]

2017Performance Optimization[3]

2017

[1] Menzies, Tim, and Ying Hu. "Just enough learning (of association rules): the TAR2 “Treatment” learner." Artificial Intelligence Review 25.3 (2006): 211-229.[2] Nair et al. "An (accidental) exploration of alternatives to evolutionary algorithms for sbse." SSBSE- 2016.[3] Guo et al. “Variability-aware performance prediction: A statistical learning approach." ASE-2013.[4] Krishna et al.. "Less is more: Minimizing code reorganization using XTREE." IST-2017

Configurable Systems and Variability

System

Configurable Systems and Variability

Systeminput.y4m

Configurable Systems and Variability

System output.x264input.y4m

Configurable Systems and Variability

System output.x264

c1 c2 c3 c4 cn. . .

Configuration options

input.y4m

Configurable Systems and Variability

System output.x264

Non-functional behavior: response time, throughput, etc.

c1 c2 c3 c4 cn. . .

Configuration options

input.y4m

Configurable Systems and Variability

System output.x264

c1 c2 c3 c4 cn. . .

Non-functional behavior: response time, throughput, etc.

Configuration options

input.y4m

Performance Optimization is Necessary!

Performance Optimization is getting more Complex!

[1] Xu et. al. 2015. Hey, you have given me too many knobs!: understanding and dealing with over-designed configuration in system software.FSE 2015[2] Van Aken, Dana, et al. "Automatic Database Management System Tuning Through Large-scale Machine Learning." International Conference on Management of Data. ACM, 2017.

● Necessary

Performance Optimization is required since Default Configuration is Bad!

[1] Van Aken, Dana, et al. "Automatic Database Management System Tuning Through Large-scale Machine Learning." International Conference on Management of Data. ACM, 2017.[2] Herodotou, Herodotos, et al. "Starfish: A Self-tuning System for Big Data Analytics." CIDR

● Necessary

● Complex

Performance Optimization can be Expensive!

• Evaluation of single instance of software/hardware co-design problem can take weeks[1]

• Rolling Sort use-case required 21 days, within a total experimental time of about 2.5 months[2]

• Test suite generation using Evolutionary Algorithm can take weeks[3]

• Image recognition workload and speech recognition workload, jobs ran for many hours or days[4]

[1] Zuluaga, Marcela, et al. "Active learning for multi-objective optimization." International Conference on Machine Learning. 2013.[2] Jamshidi, Pooyan, and Giuliano Casale. "An uncertainty-aware approach to optimal configuration of stream processing systems."MASCOTS-2016[3] Wang, Tiantian, et al. "Searching for better configurations: a rigorous approach to clone evaluation." FSE-2013[4] Venkataraman, Shivaram, et al. "Ernest: Efficient Performance Prediction for Large-Scale Advanced Analytics." NSDI. 2016.

● Necessary

● Complex

● Default is bad

Database- otter-tune- ituned

Is it pervasive?

Machine Learning- Hyperparameter Tuning- Random search - SMBO- Fabolas

Software Engineering- Tuning or Default Values?- Tuning for Software Analytics- Tuning for Defect Prediction- Topic Modelling

Cloud Computing- Ernest- Cherrypick- PARIS

● Necessary

● Complex

● Default is bad

Expensive

Performance Optimization!

● Necessary

● Complex

● Default is bad

Expensive

Pervasive

Performance Optimization!

● Necessary

● Complex

● Default is bad

Expensive

Pervasive

Road Map

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Road Map

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Residual-based Methods

Road Map

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Rank-based Method

Road Map

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Bayesian-based Method

Road Map

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Surrogate is a cheap(er) version of the actual system

APACHEhttp server

Request

Surrogate APACHE

http server

Response Time

Request

Response Time

Progressive Sampling

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Guo, Jianmei, et al. "Variability-aware performance prediction: A statistical learning approach." ASE-2013.

Residual-based MethodsProgressive Sampling

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

How to find the ‘best performing’ configuration for any given system?

Residual-based MethodsProgressive Sampling

Configuration Space

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Residual-based MethodsProgressive Sampling

Configuration Space

Training Pool Testing Pool Validation Pool

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

40%40% 20%

Residual-based MethodsProgressive Sampling

Configuration Space

Training Pool Testing Pool Validation Pool

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Residual-based MethodsProgressive Sampling

Configuration Space

Training Pool Testing Pool Validation Pool

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Residual-based MethodsProgressive Sampling

Configuration Space

Training Pool Testing Pool Validation Pool

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Train

Residual-based MethodsProgressive Sampling

Configuration Space

Training Pool Testing Pool Validation Pool

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

TrainIF MRE < τ: ExitELSE: continue

Check Performance

Residual-based MethodsProgressive Sampling

Configuration Space

Training Pool Testing Pool Validation Pool

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Train Check Performance

Mean Relative Error

MRE = |actual - predicted|actual

IF MRE < τ: ExitELSE: continue

Residual-based MethodsProgressive Sampling

Configuration Space

Training Pool Testing Pool Validation Pool

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

TrainIF MRE < τ: ExitELSE: continue

Check Performance

Residual-based MethodsProgressive Sampling

Configuration Space

Training Pool Testing Pool Validation Pool

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

TrainIF MRE < τ: ExitELSE: continue

Check Performance

Residual-based MethodsProgressive Sampling

Configuration Space

Training Pool Testing Pool Validation Pool

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

TrainIF MRE < τ: ExitELSE: continue

Check Performance

Residual-based MethodsProgressive Sampling

Configuration Space

Training Pool Testing Pool Validation Pool

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

TrainIF MRE < τ: ExitELSE: continue

Check Performance

Residual-based MethodsProgressive Sampling

Configuration Space

Training Pool Testing Pool Validation Pool

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Residual-based MethodsProgressive Sampling

Configuration Space

Training Pool Testing Pool Validation Pool

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Use the model to find the best configuration

Residual-based MethodsProgressive Sampling - Limitation

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

• The stopping condition is arbitrary

• Cannot estimate cost required to build a surrogate

Projective Sampling

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Sarkar, Atri, et al. "Cost-efficient sampling for performance prediction of configurable systems." ASE 2015.

Residual-based MethodsProjective Sampling

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Given an acceptable accuracy estimate (τ), how many samples is required to build a ‘quality’ surrogate?

Residual-based MethodsProjective Sampling

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Learning Curve

Residual-based MethodsProjective Sampling

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Learning Curve

Residual-based MethodsProjective Sampling

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Learning Curve

Residual-based MethodsProjective Sampling

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Learning Curve

Steep Incline

Residual-based MethodsProjective Sampling

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Steep Incline

Gradual Incline

Learning Curve

Residual-based MethodsProjective Sampling

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Steep Incline

Gradual Incline

Pleatue

Learning Curve

Residual-based MethodsProjective Sampling

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Steep Incline

Gradual Incline

Plateau

Optimal Sample SizeLearning Curve

Estimates the Learning Curve

Residual-based MethodsProjective Sampling

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Generate initial data points for the learning curve

Find best fit projective function

Calculate optimal training set

Build surrogate

Estimates the Learning Curve

Residual-based MethodsProjective Sampling

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Generate initial data points for the learning curve

Find best fit projective function

Calculate optimal training set

Build surrogate

Estimates the Learning Curve

Residual-based MethodsProjective Sampling

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Requirement: Initial samples should reflect relationship between all configuration options

Intuition: Performance depends if configuration option is selected or deselected

Heuristic: Feature Frequency - initial samples have each option selected or deselected, at least, δ times

Generate initial data points for the learning curve

Find best fit projective function

Calculate optimal training set

Build surrogate

Residual-based MethodsProjective Sampling

Configuration Space

Training Pool Testing Pool

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Residual-based MethodsProjective Sampling

Configuration Space

Training Pool Testing Pool

Selected 0 0 0 0

Deselected 0 0 0 0

c1 c2 c3 c4

Feature frequency table (δ=2)

#Samples Accuracy

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Residual-based MethodsProjective Sampling

Configuration Space

Training Pool Testing Pool

Selected 0 0 0 0

Deselected 0 0 0 0

c1 c2 c3 c4

Train

c1 c2 c3 c4

1 0 1 1

#Samples Accuracy

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Feature frequency table (δ=2)

Residual-based MethodsProjective Sampling

Configuration Space

Training Pool Testing Pool

Selected 1 0 1 1

Deselected 0 1 0 0

c1 c2 c3 c4

Train

c1 c2 c3 c4

1 0 1 1

1 5%

#Samples Accuracy

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Feature frequency table (δ=2)

Residual-based MethodsProjective Sampling

Configuration Space

Training Pool Testing Pool

Selected 1 0 1 1

Deselected 0 1 0 0

c1 c2 c3 c4

Train

c1 c2 c3 c4

1 0 1 1

0 1 0 0

1 5%

#Samples Accuracy

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Feature frequency table (δ=2)

Residual-based MethodsProjective Sampling

Configuration Space

Training Pool Testing Pool

Selected 1 1 2 1

Deselected 1 1 0 1

c1 c2 c3 c4

Train

c1 c2 c3 c4

1 0 1 1

0 1 1 0

1 5%

2 17%

#Samples Accuracy

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Feature frequency table (δ=2)

Residual-based MethodsProjective Sampling

Configuration Space

Training Pool Testing Pool

Selected 1 1 2 1

Deselected 1 1 0 1

c1 c2 c3 c4

Train

c1 c2 c3 c4

1 0 1 1

0 1 1 0

1 1 0 0

1 5%

2 17%

#Samples Accuracy

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Feature frequency table (δ=2)

Residual-based MethodsProjective Sampling

Configuration Space

Training Pool Testing Pool

Selected 2 2 2 1

Deselected 1 1 1 2

c1 c2 c3 c4

Train

c1 c2 c3 c4

1 0 1 1

0 1 1 0

1 1 0 0

1 5%

2 17%

3 29%

#Samples Accuracy

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Feature frequency table (δ=2)

Residual-based MethodsProjective Sampling

Configuration Space

Training Pool Testing Pool

Selected 2 2 2 1

Deselected 1 1 1 2

c1 c2 c3 c4

Train

c1 c2 c3 c4

1 0 1 1

0 1 1 0

1 1 0 0

0 0 0 1

1 5%

2 17%

3 29%

#Samples Accuracy

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Feature frequency table (δ=2)

Residual-based MethodsProjective Sampling

Configuration Space

Training Pool Testing Pool

Selected 2 2 2 2

Deselected 2 2 2 2

c1 c2 c3 c4

Train

c1 c2 c3 c4

1 0 1 1

0 1 1 0

1 1 0 0

0 0 0 1

1 5%

2 17%

3 29%

4 35%

#Samples Accuracy

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Feature frequency table (δ=2)

Estimates the Learning Curve

Residual-based MethodsProjective Sampling

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Generate initial data points for the learning curve

Find best fit projective function

Calculate optimal training set

Build surrogate

Residual-based MethodsProjective Sampling

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

1 5%

2 17%

3 29%

4 35%

#Samples Accuracy

Residual-based MethodsProjective Sampling

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Exponential

Power

Log

Estimates the Learning Curve

Residual-based MethodsProjective Sampling

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Generate initial data points for the learning curve

Find best fit projective function

Calculate optimal training set

Build surrogate

Residual-based MethodsProjective Sampling

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Coefficients of projective functionNumber of configurations whose performance value will be predicted

Penalty factor

Taken from Sarkar et al.

Estimates the Learning Curve

Residual-based MethodsProjective Sampling

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Generate initial data points for the learning curve

Find best fit projective function

Calculate optimal training set

Build surrogate

Residual-based MethodsProjective Sampling

Configuration Space

Training Pool Testing Pool Validation Pool

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Use the model to find the best configuration

Optimal Sample Size

Assumes an accurate model can be built

Residual-based MethodsProjective Sampling - Limitation

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Rank-based Method

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Nair, Vivek, et al. "Using Bad Learners to find Good Configurations." FSE 2017

Rank-based Method

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Rank-based Method

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Rank-based Method

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Rank-based Method

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Rank-based Method

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

What happens when an accurate model cannot be built?

Rank-based MethodCore Insights

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Rank Preserving Model

Rank Preserving Model

Rank Preserving Model

Rank Preserving Model

Rank Preserving Model

Rank Preserving Model

Rank Preserving Model

Rank Preserving Model

Configuration Space

Training Pool Testing Pool Validation Pool

TrainIF accuracy < τ: ExitELSE: continue

Check Performance

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Rank Preserving Model

Configuration Space

Training Pool Testing Pool Validation Pool

TrainIF accuracy < τ: ExitELSE: continue

Check Performance

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Rank Preserving Model - Limitation

Requires Testing Pool - 20% of configuration space

Configuration Space

Training Pool Testing Pool Validation Pool

40%40% 20%

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Future - Bayesian based Method

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Nair, Vivek et al. "FLASH: A Faster Optimizer for SBSE Tasks." preprint

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Bayesian-based Method

Can we avoid measurement of configurations in the testing pool?

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Bayesian-based MethodBayesian Optimization

Taken from Dr. Nando de Freitas (tiny.cc/4tgeny)

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Bayesian-based MethodBayesian Optimization

True Performance Distribution

Taken from Dr. Nando de Freitas (tiny.cc/4tgeny)

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Bayesian-based MethodBayesian Optimization

True Performance Distribution

Predicted Performance Distribution

Taken from Dr. Nando de Freitas (tiny.cc/4tgeny)

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Bayesian-based MethodBayesian Optimization

True Performance Distribution

Predicted Performance Distribution

Taken from Dr. Nando de Freitas (tiny.cc/4tgeny)

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

True Performance Distribution

Bayesian-based MethodBayesian Optimization

Predicted Performance Distribution

Acquisition Function

Which configuration should I evaluate next?

Taken from Dr. Nando de Freitas (tiny.cc/4tgeny)

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Bayesian-based MethodBayesian Optimization

True Performance Distribution

Predicted Performance Distribution

Acquisition Function

Which configuration should I evaluate next?

Taken from Dr. Nando de Freitas (tiny.cc/4tgeny)

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Bayesian-based MethodBayesian Optimization

Tradeoff between Exploration vs Exploitation

True Performance Distribution

Predicted Performance Distribution

Acquisition Function

Taken from Dr. Nando de Freitas (tiny.cc/4tgeny)

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Bayesian-based MethodBayesian Optimization

Surrogate of choice: Gaussian Processes (GP)

True Performance Distribution

Predicted Performance Distribution

Acquisition Function

Taken from Dr. Nando de Freitas (tiny.cc/4tgeny)

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Bayesian-based MethodBayesian Optimization

Taken from Dr. Nando de Freitas (tiny.cc/4tgeny)

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Bayesian-based MethodBayesian Optimization

Taken from Dr. Nando de Freitas (tiny.cc/4tgeny)

GP lose efficiency in high dimensional spaces i.e. number of features exceeds a dozen

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Bayesian-based MethodBayesian Optimization

Bayesian-based Method - FLASH

GP

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Bayesian-based Method - FLASH

GP

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Bayesian-based Method - FLASH

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Bayesian-based Method - FLASH

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Bayesian-based Method - FLASH

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Bayesian-based Method - FLASH

2013 2015

[Sarkar’15][Guo’13]

2017

[Nair’17]

Future

Bayesian-based Method - FLASH

Bayesian-based Method - FLASH

● Fast

● Effective

● Comprehensible

Bayesian-based Method - FLASH

Conclusion• ML Algorithms are not a black box

– How to use Decision Tree in Planning?– Can I explain the results to the Decision

Makers?

• Lazy is good– Only do what is required– Optimization does not require an accurate

model

• Easy over Hard – Try simplest first– Tuning SVM outperforms DL

Rank-based Method: http://tiny.cc/wnhenyFlash: http://tiny.cc/hohenyePAL: http://www.spiral.net/software/pal.htmlBayesian Optimization: https://youtu.be/vz3D36VXefI

"Look for me...beneath the tree...North!"

―Three-eyed raven