Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y...
Transcript of Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y...
![Page 1: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/1.jpg)
Machine Learning with R at LRZIntroduction to mlr
www.essentialds.de2019-10-11
![Page 2: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/2.jpg)
![Page 3: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/3.jpg)
WHAT IS MACHINE LEARNING
![Page 4: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/4.jpg)
Typical Workflow
WHAT IS MACHINE LEARNING www.essentialds.de 3/76
![Page 5: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/5.jpg)
Machine Learning Tasks
Machine learning (ML) can be seen as the intersection betweencomputer science and computational statistics in which computeralgorithms learn to solve different tasks based on data (e.g. makingpredictions, finding groups, . . . ).
WHAT IS MACHINE LEARNING www.essentialds.de 4/76
![Page 6: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/6.jpg)
Binary Classification Task
• y is a categorical variable (with two values)• E.g., sick/healthy, or credit/no credit• Goal: Predict a class (or membership probabilities)
30
40
50
60
4000 6000 8000monthly salary
age
class
employee
manager
WHAT IS MACHINE LEARNING www.essentialds.de 5/76
![Page 7: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/7.jpg)
Multiclass Classification Task
• y is a categorical variable with > 2 unordered values• Each instance belongs to only one class• Goal: Predict a class (or membership probabilities)
0.0
0.5
1.0
1.5
2.0
2.5
2 4 6Petal.Length
Pet
al.W
idth Species
setosa
versicolor
virginica
Iris flower data
WHAT IS MACHINE LEARNING www.essentialds.de 6/76
![Page 8: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/8.jpg)
Regression Task
• Goal: Predict a continuous output• y is a metric variable (with values in R)• Regression model can be constructed by different methods,
e.g., linear regression, trees or splines
2000
4000
6000
20 25 30 35 40 45 50 55 60 65 70age
mon
thly
sal
ary
WHAT IS MACHINE LEARNING www.essentialds.de 7/76
![Page 9: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/9.jpg)
INTRODUCTION
![Page 10: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/10.jpg)
Motivation: Machine Learning in R
The good news:
• CRAN serves hundreds of packages for machine learning• Often compliant to the unwritten interface definition:
model = fit(target ~ ., data = train.data, ...)predictions = predict(model, newdata = test.data, ...)
The bad news:
• Some packages’ API is “just different”• Functionality is always package or model-dependent, even
though the procedure might be general• No meta-information available or buried in docs
Our goal: A domain-specific language for ML concepts!
INTRODUCTION www.essentialds.de 8/76
![Page 11: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/11.jpg)
Motivation: MLR
• Project home page: https://github.com/mlr-org/mlr• Cheatsheet for an quick overview• Tutorial for mlr documentation with many code examples• Ask questions in the GitHub issue tracker
• 8-10 main developers, quite a few contributors, 4 GSOCprojects in 2015/16 and one in 2017
• About 30K lines of code, 8K lines of unit tests
INTRODUCTION www.essentialds.de 9/76
![Page 12: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/12.jpg)
Motivation: MLR
• Unified interface for the basic building blocks: tasks, learners,hyperparameters, . . .
INTRODUCTION www.essentialds.de 10/76
![Page 13: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/13.jpg)
Basic Features of MLR
• Tasks and Learners• Train, Test, Resample• Performance• Benchmarking• Hyperparameter Tuning• Nested Resampling• Parallelization
INTRODUCTION www.essentialds.de 11/76
![Page 14: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/14.jpg)
Building blocks
Training Set
Learner
Performance Measure
Test Set
Task / Data Performance
Model Prediction
Repeat = Resample
• Core objects: tasks, learners, measures, resampling instances.
INTRODUCTION www.essentialds.de 12/76
![Page 15: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/15.jpg)
Task Abstraction
• Tasks encapsulate data and meta-information about it.• Regression, classification, clustering, survival tasks.
data(BostonHousing, package = "mlbench")task = makeRegrTask(data = BostonHousing, target = "medv")print(task)## Supervised task: BostonHousing## Type: regr## Target: medv## Observations: 506## Features:## numerics factors ordered functionals## 12 1 0 0## Missings: FALSE## Has weights: FALSE## Has blocking: FALSE## Has coordinates: FALSE
INTRODUCTION www.essentialds.de 13/76
![Page 16: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/16.jpg)
Learner Abstraction
• Internal structure of learners:• wrappers around fit() and predict() of the package• description of the parameter set• annotations
• Naming convention: <tasktype>.<functionname>
makeLearner("regr.lm")makeLearner("classif.randomForest")makeLearner("classif.knn", k = 2)
• Adding custom learners is covered in the tutorial
INTRODUCTION www.essentialds.de 14/76
![Page 17: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/17.jpg)
Learner Abstraction
lrn = makeLearner("classif.knn", k = 2)print(lrn)## Learner classif.knn from package class## Type: classif## Name: k-Nearest Neighbor; Short name: knn## Class: classif.knn## Properties: twoclass,multiclass,numerics## Predict-Type: response## Hyperparameters: k=2
getParamSet(lrn)## Type len Def Constr Req Tunable Trafo## k integer - 1 1 to Inf - TRUE -## l numeric - 0 0 to Inf - TRUE -## prob logical - FALSE - - FALSE -## use.all logical - TRUE - - TRUE -
INTRODUCTION www.essentialds.de 15/76
![Page 18: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/18.jpg)
Learn more
• Extensive Tutorial covers all features in mlr:https://mlr-org.github.io/mlr/
• Tuning• Resampling (with blocking)• Visualization Topics• Multilabel Classification, Survival Analysis, Clustering• Handling Spatial Data• Functional Data• Create Custom Learners and Measures• . . .
INTRODUCTION www.essentialds.de 16/76
![Page 19: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/19.jpg)
Getting Help
• Ask questions on Stackoverflow:https://stackoverflow.com/questions/tagged/mlr
• Found bugs? Report them:https://github.com/mlr-org/mlr/issues
You want to contribute? - Open a PR on github and join our slack:https://mlr-org.slack.com/
INTRODUCTION www.essentialds.de 17/76
![Page 20: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/20.jpg)
FIRST DATA ANALYSIS
![Page 21: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/21.jpg)
First Classification Analysis
1. Peek into the iris data set2. Define a classification task3. Fit a k-NN classification model4. Predict labels
FIRST DATA ANALYSIS www.essentialds.de 18/76
![Page 22: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/22.jpg)
Iris Data Set
The iris dataset was introduced by the statistician Ronald Fisherand is one of the most frequent used datasets. Originally it wasdesigned for linear discriminant analysis.
The set is a typical test case for many statistical classificationtechniques and has its own wikipedia page.
Setosa Versicolor Virginica
Source: https://en.wikipedia.org/wiki/Iris_flower_data_set
FIRST DATA ANALYSIS www.essentialds.de 19/76
![Page 23: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/23.jpg)
Iris Data Set
• 150 iris flowers• 3 different species (50 setosa, 50 versicolor, 50 virginica) to be
predicted.• Sepal length / width and petal length / width in [cm].
Source: https://holgerbrandl.github.io/kotlin4ds_kotlin_night_frankfurt//krangl_example_report.html
FIRST DATA ANALYSIS www.essentialds.de 20/76
![Page 24: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/24.jpg)
Iris Data Set
A peek into the data:
data("iris", package = "datasets")str(iris)## 'data.frame': 150 obs. of 5 variables:## $ Sepal.Length: num 5.1 4.9 4.7 4.6 5 5.4 4.6 5 ...## $ Sepal.Width : num 3.5 3 3.2 3.1 3.6 3.9 3.4 3.4 ...## $ Petal.Length: num 1.4 1.4 1.3 1.5 1.4 1.7 1.4 1.5 ...## $ Petal.Width : num 0.2 0.2 0.2 0.2 0.2 0.4 0.3 0.2 ...## $ Species : Factor w/ 3 levels "setosa","versicolor",..: 1..
FIRST DATA ANALYSIS www.essentialds.de 21/76
![Page 25: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/25.jpg)
Iris Data Set
Cor : −0.118setosa: 0.743
versicolor: 0.526virginica: 0.457
Cor : 0.872setosa: 0.267
versicolor: 0.754virginica: 0.864
Cor : −0.428setosa: 0.178
versicolor: 0.561virginica: 0.401
Cor : 0.818setosa: 0.278
versicolor: 0.546virginica: 0.281
Cor : −0.366setosa: 0.233
versicolor: 0.664virginica: 0.538
Cor : 0.963setosa: 0.332
versicolor: 0.787virginica: 0.322
Sepal.Length Sepal.Width Petal.Length Petal.Width SpeciesS
epal.LengthS
epal.Width
Petal.Length
Petal.W
idthS
pecies
5 6 7 8 2.0 2.5 3.0 3.5 4.0 4.5 2 4 6 0.0 0.5 1.0 1.5 2.0 2.5 setosaversicolorvirginica
0.0
0.4
0.8
1.2
2.0
2.5
3.0
3.5
4.0
4.5
2
4
6
0.0
0.5
1.0
1.5
2.0
2.5
0.02.55.07.5
0.02.55.07.5
0.02.55.07.5
FIRST DATA ANALYSIS www.essentialds.de 22/76
![Page 26: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/26.jpg)
First Classification Analysis: Task
Define a mlr task for the iris data using the target column Species:
library(mlr)task.iris = makeClassifTask(id = "iris", data = iris, target = "Species")print(task.iris) # Gives you an overview of the task## Supervised task: iris## Type: classif## Target: Species## Observations: 150## Features:## numerics factors ordered functionals## 4 0 0 0## Missings: FALSE## Has weights: FALSE## Has blocking: FALSE## Has coordinates: FALSE## Classes: 3## setosa versicolor virginica## 50 50 50## Positive class: NA
Functions prefixed with getTask[...] allow to extract information,i.e., getTaskData extracts the data.frame.
FIRST DATA ANALYSIS www.essentialds.de 23/76
![Page 27: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/27.jpg)
What Learners are available?
Classification• LDA, QDA, RDA, MDA• Trees and forests• Boosting (different variants)• SVMs (different variants)• (Deep) Neural Networks• . . .
Clustering• K-Means• EM• DBscan• X-Means• . . .
Regression• Linear, lasso and ridge• Boosting• Trees and forests• Gaussian processes• (Deep) Neural Networks• . . .
Survival• Cox-PH• Cox-Boost• Random survival forest• Penalized regression• . . .
FIRST DATA ANALYSIS www.essentialds.de 24/76
![Page 28: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/28.jpg)
What Learners are available?
We can explore them on the webpage, e.g. Classification learners:
FIRST DATA ANALYSIS www.essentialds.de 25/76
![Page 29: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/29.jpg)
What Learners are available?
Or use listLearners to find appropriate learners for the giventask:
tab = listLearners(task.iris, warn.missing.packages = FALSE)tab[1:5, c("class", "package")]## class package## 1 classif.adaboostm1 RWeka## 2 classif.boosting adabag,rpart## 3 classif.C50 C50## 4 classif.cforest party## 5 classif.ctree party
This is possible because
• the task contains relevant data/task characteristics (e.g.,missings) and
• the learner checks if it can handle data/tasks with thesecharacteristics (learner properties)
FIRST DATA ANALYSIS www.essentialds.de 26/76
![Page 30: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/30.jpg)
First Classification Analysis: Learner
Define the classification learner:
lrn.knn = makeLearner("classif.kknn", k = 30, predict.type = "prob")## Loading required package: kknn
print(lrn.knn) # learner will predict classes and probabilities## Learner classif.kknn from package kknn## Type: classif## Name: k-Nearest Neighbor; Short name: kknn## Class: classif.kknn## Properties: twoclass,multiclass,numerics,factors,prob## Predict-Type: prob## Hyperparameters: k=30
FIRST DATA ANALYSIS www.essentialds.de 27/76
![Page 31: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/31.jpg)
First Classification Analysis: Learner
The learner contains information about all parameters that can bespecified:
# list available hyperparameters + defaults, constraints, dependencies, ...getParamSet(lrn.knn)
## Type len Def Constr Req Tunable Trafo## k integer - 7 1 to Inf - TRUE -## distance numeric - 2 0 to Inf - TRUE -## kernel discrete - optimal rectangular,trian... - TRUE -## scale logical - TRUE - - TRUE -
FIRST DATA ANALYSIS www.essentialds.de 28/76
![Page 32: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/32.jpg)
First Classification Analysis: Training
A model is usually trained on a subset of the data - the remainingpart is used to evaluate its performance.
n = getTaskSize(task.iris)train.ind = sample(n, n/2)test.ind = setdiff(1:n, train.ind)str(train.ind)## int [1:75] 14 50 118 43 150 148 90 91 ...
str(test.ind)## int [1:75] 1 2 3 5 8 10 11 12 ...
FIRST DATA ANALYSIS www.essentialds.de 29/76
![Page 33: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/33.jpg)
First Classification Analysis: Training
# train model with mlrmod = train(lrn.knn, task = task.iris, subset = train.ind)print(mod)## Model for learner.id=classif.kknn; learner.class=classif.kknn## Trained on: task.id = iris; obs = 75; features = 4## Hyperparameters: k=30
# retrieve model as returned from the third party packageknn.mod = getLearnerModel(mod)
FIRST DATA ANALYSIS www.essentialds.de 30/76
![Page 34: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/34.jpg)
First Classification Analysis: Predictions
The prediction is then applied on the unseen test data.
# predict using the taskpreds = predict(mod, task = task.iris, subset = test.ind)head(as.data.frame(preds), 3)## id truth prob.setosa prob.versicolor prob.virginica response## 1 1 setosa 0.957 0.0426 0 setosa## 2 2 setosa 0.927 0.0733 0 setosa## 3 3 setosa 0.938 0.0617 0 setosa
# predict using data set observationspreds = predict(mod, newdata = iris[test.ind, ])head(as.data.frame(preds), 3)## truth prob.setosa prob.versicolor prob.virginica response## 1 setosa 0.957 0.0426 0 setosa## 2 setosa 0.927 0.0733 0 setosa## 3 setosa 0.938 0.0617 0 setosa
FIRST DATA ANALYSIS www.essentialds.de 31/76
![Page 35: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/35.jpg)
First Classification Analysis: Predictions
pred.class = getPredictionResponse(preds) # predicted classespred.prob = getPredictionProbabilities(preds) # predicted probabilitiestruth = getPredictionTruth(preds) # true classeshead(pred.class, 3)## [1] setosa setosa setosa## Levels: setosa versicolor virginica
head(pred.prob, 3)## setosa versicolor virginica## 1 0.957 0.0426 0## 2 0.927 0.0733 0## 3 0.938 0.0617 0
head(truth, 3)## [1] setosa setosa setosa## Levels: setosa versicolor virginica
FIRST DATA ANALYSIS www.essentialds.de 32/76
![Page 36: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/36.jpg)
First Classification Analysis: Evaluation
# total number of errorssum(pred.class != truth)## [1] 7
# mean misclassification error (MMCE)mean(pred.class != truth)## [1] 0.0933
# percentage of accurate predictions (ACC, accuracy)mean(pred.class == truth)## [1] 0.907
FIRST DATA ANALYSIS www.essentialds.de 33/76
![Page 37: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/37.jpg)
First Classification Analysis: Evaluation
calculateConfusionMatrix(preds)## predicted## true setosa versicolor virginica -err.-## setosa 27 0 0 0## versicolor 0 22 6 6## virginica 0 1 19 1## -err.- 0 1 6 7
FIRST DATA ANALYSIS www.essentialds.de 34/76
![Page 38: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/38.jpg)
First Classification Analysis: Evaluation
performance(preds, measures = mmce) # mean misclassification error## mmce## 0.0933
performance(preds, measures = list(mmce, acc))## mmce acc## 0.0933 0.9067
listMeasures(task.iris)## [1] "featperc" "mmce" "lsr"## [4] "bac" "qsr" "timeboth"## [7] "multiclass.aunp" "timetrain" "multiclass.aunu"## [10] "ber" "timepredict" "multiclass.brier"## [13] "ssr" "acc" "logloss"## [16] "wkappa" "multiclass.au1p" "multiclass.au1u"## [19] "kappa"
FIRST DATA ANALYSIS www.essentialds.de 35/76
![Page 39: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/39.jpg)
First Classification Analysis: Evaluation
plotLearnerPrediction(lrn.knn, task.iris,features = c("Petal.Width", "Petal.Length"))
2
4
6
0.0 0.5 1.0 1.5 2.0 2.5Petal.Width
Pet
al.L
engt
h Species
setosa
versicolor
virginica
kknn: k=30Train: mmce=0.040; CV: mmce.test.mean=0.040
Predictions for learner fitted on two features.
FIRST DATA ANALYSIS www.essentialds.de 36/76
![Page 40: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/40.jpg)
EXERCISE 1
![Page 41: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/41.jpg)
RESAMPLING
![Page 42: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/42.jpg)
Hold-out in mlr
task = iris.taskn = getTaskSize(task)ratio = 2/3
set.seed(123)train.inds = sample(1:n, n * ratio)test.inds = setdiff(1:n, train.inds)
lrn.knn1 = makeLearner("classif.knn", k = 1)mod = train(lrn.knn1, task, subset = train.inds)preds = predict(mod, task, subset = test.inds)preds = predict(mod, newdata = iris[test.inds, ]) # alternativemlr::performance(preds, mmce)## mmce## 0.04
RESAMPLING www.essentialds.de 37/76
![Page 43: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/43.jpg)
Cross-validation in mlr
# Define learner:lrn = makeLearner("classif.randomForest", predict.type = "prob")
# Define resampling strategy:rdesc = makeResampleDesc("CV", iters = 3, stratify = TRUE)r = resample(lrn, spam.task, rdesc,
measure = list(mlr::acc, mlr::auc))print(r)## Resample Result## Task: spam-example## Learner: classif.randomForest## Aggr perf: acc.test.mean=0.950,auc.test.mean=0.986## Runtime: 16.4351
RESAMPLING www.essentialds.de 38/76
![Page 44: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/44.jpg)
Cross-validation in mlr
head(r$measures.test)## iter acc auc## 1 1 0.950 0.985## 2 2 0.947 0.984## 3 3 0.952 0.990
head(as.data.frame(r$pred))## id truth prob.nonspam prob.spam response iter set## 1 1816 nonspam 1.000 0.000 nonspam 1 test## 2 1819 nonspam 0.990 0.010 nonspam 1 test## 3 1820 nonspam 0.992 0.008 nonspam 1 test## 4 1824 nonspam 0.902 0.098 nonspam 1 test## 5 1828 nonspam 0.988 0.012 nonspam 1 test## 6 1829 nonspam 0.978 0.022 nonspam 1 test
RESAMPLING www.essentialds.de 39/76
![Page 45: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/45.jpg)
Parameter of makeResamplingDesc
Methods Parameter Description
CV iters Number of iterationsLOORepCV reps Repeats for repeated CV
folds Folds in the repeated CVBootstrap iters Number of iterationsSubsample iters Number of iterations
split Proportion of training casesHoldout split Proportion of training cases
RESAMPLING www.essentialds.de 40/76
![Page 46: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/46.jpg)
Possible Ways to use Cross Validation
1. Explicitly define resampling:
rdesc = makeResampleDesc("CV", iters = 10)rdesc = cv10
res1 = resample("classif.randomForest", iris.task, resampling = rdesc,show.info = FALSE)
res2 = resample("classif.randomForest", iris.task, resampling = cv10,show.info = FALSE)
res1
## Resample Result## Task: iris-example## Learner: classif.randomForest## Aggr perf: mmce.test.mean=0.053## Runtime: 0.297201
Other pre defined objects are cv2, cv3 and cv5.
RESAMPLING www.essentialds.de 41/76
![Page 47: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/47.jpg)
Possible Ways to use Cross Validation
2. Use crossval:
res3 = crossval("classif.randomForest", iris.task, iters = 10,show.info = FALSE)
res3
## Resample Result## Task: iris-example## Learner: classif.randomForest## Aggr perf: mmce.test.mean=0.047## Runtime: 0.315572
Similar functions are repcv, holdout, subsample,bootstrapOOB, bootstrapB632 and bootstrapB632plus.
RESAMPLING www.essentialds.de 42/76
![Page 48: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/48.jpg)
Classification Analysis: Benchmarking
# quick way to compare learners with identical train/test splitstask = iris.tasklearners = list(
makeLearner("classif.knn", k = 3),makeLearner("classif.lda"),makeLearner("classif.naiveBayes")
)benchmark(learners, task, resamplings = cv3)## task.id learner.id mmce.test.mean## 1 iris-example classif.knn 0.04## 2 iris-example classif.lda 0.02## 3 iris-example classif.naiveBayes 0.04
RESAMPLING www.essentialds.de 43/76
![Page 49: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/49.jpg)
Classification Analysis: Benchmarking
tasks = list(iris.task, sonar.task, pid.task)bm = benchmark(learners, tasks, resampling = cv3)print(bm)## task.id learner.id mmce.test.mean## 1 iris-example classif.knn 0.0400## 2 iris-example classif.lda 0.0200## 3 iris-example classif.naiveBayes 0.0467## 4 PimaIndiansDiabetes-example classif.knn 0.3034## 5 PimaIndiansDiabetes-example classif.lda 0.2331## 6 PimaIndiansDiabetes-example classif.naiveBayes 0.2500## 7 Sonar-example classif.knn 0.1734## 8 Sonar-example classif.lda 0.2883## 9 Sonar-example classif.naiveBayes 0.2981
RESAMPLING www.essentialds.de 44/76
![Page 50: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/50.jpg)
Classification Analysis: Benchmarking
# aggregated data:getBMRAggrPerformances(bm, as.df = TRUE)## task.id learner.id mmce.test.mean## 1 iris-example classif.knn 0.0400## 2 iris-example classif.lda 0.0200## 3 iris-example classif.naiveBayes 0.0467## 4 PimaIndiansDiabetes-example classif.knn 0.3034## 5 PimaIndiansDiabetes-example classif.lda 0.2331## 6 PimaIndiansDiabetes-example classif.naiveBayes 0.2500## 7 Sonar-example classif.knn 0.1734## 8 Sonar-example classif.lda 0.2883## 9 Sonar-example classif.naiveBayes 0.2981
RESAMPLING www.essentialds.de 45/76
![Page 51: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/51.jpg)
Classification Analysis: Benchmarking
# complete data:head(as.data.frame(bm), 10)## task.id learner.id iter mmce## 1 iris-example classif.knn 1 0.040## 2 iris-example classif.knn 2 0.080## 3 iris-example classif.knn 3 0.000## 4 iris-example classif.lda 1 0.040## 5 iris-example classif.lda 2 0.020## 6 iris-example classif.lda 3 0.000## 7 iris-example classif.naiveBayes 1 0.040## 8 iris-example classif.naiveBayes 2 0.080## 9 iris-example classif.naiveBayes 3 0.020## 10 PimaIndiansDiabetes-example classif.knn 1 0.324
RESAMPLING www.essentialds.de 46/76
![Page 52: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/52.jpg)
Classification Analysis: Benchmarking
plotBMRBoxplots(bm, pretty.names = FALSE)
iris−example PimaIndiansDiabetes−example Sonar−example
classif.knn
classif.lda
classif.naiveBayes
classif.knn
classif.lda
classif.naiveBayes
classif.knn
classif.lda
classif.naiveBayes
0.0
0.1
0.2
0.3
mm
ce
RESAMPLING www.essentialds.de 47/76
![Page 53: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/53.jpg)
Classification Analysis: Benchmarking
plotBMRSummary(bm, pretty.names = FALSE)
iris−example
PimaIndiansDiabetes−example
Sonar−example
0.1 0.2 0.3mmce.test.mean
learner.id
classif.knn
classif.lda
classif.naiveBayes
RESAMPLING www.essentialds.de 48/76
![Page 54: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/54.jpg)
TUNING
![Page 55: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/55.jpg)
Hyperparameter Tuning
• Many parameters or decisions for an ML algorithm are notdecided by the fitting procedure
• Model parameters are optimized during training, by someform of loss minimization. They are an output of the training.E.g., the coefficients of a linear model or the optimal splits of atree learner.
• Hyperparameters must be specified before the training phase.They are an input of the training. E.g., how small a leaf canbecome for a tree; k and which distance measure to use forkNN
TUNING www.essentialds.de 49/76
![Page 56: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/56.jpg)
Hyperparameter Tuning
• HPs have to be set either by the user or by (smart) defaultvalues
• Our goal is to optimize these w.r.t. the estimated predictionerror; this implies an independent test set, or cross-validation
• The same applies to preprocessing, feature construction andother model-relevant operations. In general we might beinterested in optimizing an entire ML “pipeline”
TUNING www.essentialds.de 50/76
![Page 57: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/57.jpg)
ML Pipeline
TUNING www.essentialds.de 51/76
![Page 58: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/58.jpg)
Why tuning is important
• Hyperparameters control the complexity of a model, i.e., howflexible the model is
• If a model is too flexible so that it simply “memorizes” thetraining data, we will face the dreaded problem of overfitting
• Hence, control of capacity, i.e., proper setting ofhyperparameters can prevent overfitting the model on thetraining set
• Many other factors like optimization control settings, distancefunctions, scaling, algorithmic variants in the fitting procedurecan heavily influence model performance in non-trivial ways. Itis extremely hard to guess the correct choices here.
TUNING www.essentialds.de 52/76
![Page 59: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/59.jpg)
Types of hyperparameters
• Numerical parameters (real valued / integers)• mtry in a random forest• Neighborhood size k for kNN
• Categorical parameters:• Which split criterion for classification trees?• Which distance measure for kNN?
• Ordinal parameters:• {low, medium, high}
• Dependent parameters:• If we use the Gaussian kernel for the SVM, what is its width?
TUNING www.essentialds.de 53/76
![Page 60: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/60.jpg)
Components of tuning problem
• The learner (possibly: several competing learners?)• The performance measure. Determined by the application. Not
necessarily identical to the loss function that the learner tries tominimize. We could even be interested in multiple measuressimultaneously, e.g., accuracy and sparseness of our model,TPR and PPV, etc.
• A (resampling) procedure for estimating the predictiveperformance
• The learner’s hyperparameters and their respectiveregions-of-interest over which we optimize
TUNING www.essentialds.de 54/76
![Page 61: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/61.jpg)
Hyperparameter Tuning
Tuner proposes configuration, eval by resampling, tuner receivesperformance, iterate
TUNING www.essentialds.de 55/76
![Page 62: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/62.jpg)
Hyperparameter Tuning
Tuner proposes configuration, eval by resampling, tuner receivesperformance, iterate
TUNING www.essentialds.de 56/76
![Page 63: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/63.jpg)
Hyperparameter Tuning
Tuner proposes configuration, eval by resampling, tuner receivesperformance, iterate
TUNING www.essentialds.de 57/76
![Page 64: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/64.jpg)
Why is tuning so hard?
• Tuning is derivative-free (“black box problem”): It is usuallyimpossible to compute derivatives of the objective (i.e., theresampled performance measure) that we optimize with regardto the HPs. All we can do is evaluate the performance for agiven hyperparameter configuration.
• Every evaluation requires one or multiple train and predictsteps of the learner, i.e., every evaluation is very expensive.
• Even worse: the answer we get from that evaluation is notexact, but stochastic in most settings, as we use resampling.
TUNING www.essentialds.de 58/76
![Page 65: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/65.jpg)
Why is tuning so hard?
• Categorical and dependent hyperparameters aggravate ourdifficulties: the space of hyperparameters we optimize over hasa non-metric, complicated structure.
• For large and difficult problems parallelizing the computationseems relevant, to evaluate multiple HP configurations inparallel or to speed up the resampling-based performanceevaluation
TUNING www.essentialds.de 59/76
![Page 66: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/66.jpg)
Grid search
• Simple technique which is still quite popular, tries all HPcombinations on a multi-dimensional discretized grid
• For each hyperparameter a finite set of candidates is predefined• We simply search all possible combinations in arbitrary order
−10
−5
0
5
10
−10 −5 0 5 10Parameter 1
Par
amet
er 2
0.5
0.6
0.7
0.8
TestAccuracy
TUNING www.essentialds.de 60/76
![Page 67: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/67.jpg)
Grid search
Advantages
• Very easy to implement, therefore very popular• All parameter types possible• Parallelization is trivial
Disadvantages
• Combinatorial explosion, inefficient• Searches large irrelevant areas• Which values / discretization?
TUNING www.essentialds.de 61/76
![Page 68: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/68.jpg)
Random search
• Small variation of grid search• Uniformly sample from the region-of-interest
−10
−5
0
5
10
−10 −5 0 5 10Parameter 1
Par
amet
er 2
0.5
0.6
0.7
0.8
TestAccuracy
TUNING www.essentialds.de 62/76
![Page 69: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/69.jpg)
Random search
Advantages
• Very easy to implement, therefore very popular• All parameter types possible• Parallelization is trivial• Anytime algorithm - we can always increase the budget when
we are not satisfied• Often better than grid search, as each individual parameter has
been tried with m different values, when the search budget wasm. Mitigates the problem of discretization
Disadvantages
• As for grid search, many evaluations in areas with lowlikelihood for improvement
TUNING www.essentialds.de 63/76
![Page 70: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/70.jpg)
Tuning Example
Tuning gradient boosting with random search and 5CV on the spamdata set for AUC
Parameter Type Min Maxn.trees integer 3 500shrinkage numeric 0 1interaction integer 1 5bag.fraction numeric 0.2 0.9
0.90
0.91
0.92
0.93
0.94
0 10 20 30 40 50iter
auc
TUNING www.essentialds.de 64/76
![Page 71: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/71.jpg)
Advanced Tuning Techniques
• Stochastic local search, e.g. simulated annealing• Genetic algorithms / CMAES• Iterated F-Racing• Model-based Optimization / Bayesian Optimization• Hyperband• . . .
TUNING www.essentialds.de 65/76
![Page 72: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/72.jpg)
Hyperparameters in mlr
lrn = makeLearner("classif.gbm", predict.type="prob")ps = getParamSet(makeLearner("classif.gbm"))print(ps, constr.clip=10)## Type len Def Constr Req Tunable Trafo## distribution discrete - - bernoul... - TRUE -## n.trees integer - 100 1 to Inf - TRUE -## cv.folds integer - 0 -Inf to Inf - TRUE -## interaction.depth integer - 1 1 to Inf - TRUE -## n.minobsinnode integer - 10 1 to Inf - TRUE -## shrinkage numeric - 0.001 0 to Inf - TRUE -## bag.fraction numeric - 0.5 0 to 1 - TRUE -## train.fraction numeric - 1 0 to 1 - TRUE -## keep.data logical - TRUE - - FALSE -## verbose logical - FALSE - - FALSE -
TUNING www.essentialds.de 66/76
![Page 73: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/73.jpg)
Hyperparameters in mlr
• Either set them in constructor, or change them later:
lrn = makeLearner("classif.gbm", predict.type="prob", shrinkage = 0.1)lrn = setHyperPars(lrn, distribution = "bernoulli", shrinkage = 0.2)
TUNING www.essentialds.de 67/76
![Page 74: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/74.jpg)
Tuning in mlr
• Create a set of parameters• Here we optimize boosting
par.set = makeParamSet(makeIntegerParam("n.trees", lower = 3, upper = 20),makeNumericParam("shrinkage", lower = 0, upper = 0.2)
)
TUNING www.essentialds.de 68/76
![Page 75: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/75.jpg)
Tuning in mlr
Optimize the hyperparameter of learner
tune.ctrl = makeTuneControlRandom(maxit = 50L)tr = tuneParams(lrn, task = sonar.task, par.set = par.set,
resampling = hout, control = tune.ctrl,measures = mlr::auc)
TUNING www.essentialds.de 69/76
![Page 76: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/76.jpg)
Tuning in mlr
tr$x## $n.trees## [1] 5#### $shrinkage## [1] 0.2
tr$y## auc.test.mean## 0.849
head(as.data.frame(tr$opt.path), 3L)[, c(1,2,3,7)]## n.trees shrinkage auc.test.mean exec.time## 1 6 0.181 0.792 0.05## 2 6 0.121 0.806 0.03## 3 10 0.134 0.813 0.06
TUNING www.essentialds.de 70/76
![Page 77: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/77.jpg)
Parameter Types
makeNumericParam("x" ,lower = -1, upper = 1)makeIntegerParam("x" ,lower = -1L, upper = 1L)makeDiscreteParam("x" ,values = c("a", "b", "c"))makeLogicalParam("x")
and vector-types exist for all param types
makeNumericVectorParam("x" , len = 3L, lower = -1, upper = 1)## Type len Def Constr Req Tunable Trafo## 1 numericvector 3 - -1 to 1 - TRUE -
TUNING www.essentialds.de 71/76
![Page 78: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/78.jpg)
PARALLEL MLR
![Page 79: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/79.jpg)
Parallelization
• Many tasks in statistics are embarrassingly parallel(independence assumptions, resampling, . . . )
• R is mostly single-threaded (matrix operations may be parallel,depending on your installation)
• Multiple backends for explicit parallelization available:• Multicore (packages parallel/multicore)• Socket and MPI cluster (packages parallel/snow/Rmpi)• HPC-Clusters (package batchtools): SLURM, Torque/PBS,
SGE, LSF, Docker, SSH makeshift clusters, . . .
PARALLEL MLR www.essentialds.de 72/76
![Page 80: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/80.jpg)
Parallelization
• We use parallelMap in mlr an abstraction for all backends• Initialize with parallelStart()• Parallelize function call with
parallelMap()/parallelLapply()/. . .• Stop with parallelStop()
parallelStartSocket(4)parallelMap(function(x) x^2, 1:10)parallelStop()
PARALLEL MLR www.essentialds.de 73/76
![Page 81: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/81.jpg)
Parallelization
• The first loop which is marked as parallel executable will beautomatically parallelized
• Which loop is suited best for parallelization depends on thenumber of iterations
• Levels allow fine grained control over the parallelization• mlr.resample: Each resampling iteration (a train / test step)
is a parallel job.• mlr.benchmark: Each experiment “run this learner on this
data set” is a parallel job.• mlr.tuneParams: Each evaluation in hyperparameter space
“resample with these parameter settings” is a parallel job. Howmany of these can be run independently in parallel depends onthe tuning algorithm.
• mlr.selectFeatures: Each evaluation in feature space“resample with this feature subset” is a parallel job.
PARALLEL MLR www.essentialds.de 74/76
![Page 82: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/82.jpg)
Parallelization
lrns = list(makeLearner("classif.rpart"), makeLearner("classif.svm"))rdesc = makeResampleDesc("Bootstrap", iters = 100)
parallelStartSocket(4)## Starting parallelization in mode=socket with cpus=4.
bm = benchmark(learners = lrns, tasks = iris.task, resamplings = rdesc)## Exporting objects to slaves for mode socket: .mlr.slave.options## Mapping in parallel: mode = socket; level = mlr.benchmark; cpus = 4; elements = 2.
parallelStop()## Stopped parallelization. All cleaned up.
PARALLEL MLR www.essentialds.de 75/76
![Page 83: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/83.jpg)
Parallelization
Parallelize the bootstrap instead:
parallelStartSocket(4, level = "mlr.resample")## Starting parallelization in mode=socket with cpus=4.
bm = benchmark(learners = lrns, tasks = iris.task, resamplings = rdesc)## Task: iris-example, Learner: classif.rpart## Exporting objects to slaves for mode socket: .mlr.slave.options## Resampling: OOB bootstrapping## Measures: mmce## Mapping in parallel: mode = socket; level = mlr.resample; cpus = 4; elements = 100.#### Aggregated Result: mmce.test.mean=0.063#### Task: iris-example, Learner: classif.svm## Exporting objects to slaves for mode socket: .mlr.slave.options## Resampling: OOB bootstrapping## Measures: mmce## Mapping in parallel: mode = socket; level = mlr.resample; cpus = 4; elements = 100.#### Aggregated Result: mmce.test.mean=0.042##
parallelStop()## Stopped parallelization. All cleaned up.
PARALLEL MLR www.essentialds.de 76/76
![Page 84: Machine Learning with R at LRZ - Introduction to mlr · MulticlassClassificationTask • y isacategoricalvariablewith>2unorderedvalues • Eachinstancebelongstoonlyoneclass • Goal:](https://reader033.fdocuments.in/reader033/viewer/2022060318/5f0c7e347e708231d435ad45/html5/thumbnails/84.jpg)
EXERCISE 2