COMP24111: Machine Learning Ensemble Models Gavin Brown gbrown.
-
Upload
damon-clifford-stewart -
Category
Documents
-
view
220 -
download
0
description
Transcript of COMP24111: Machine Learning Ensemble Models Gavin Brown gbrown.
![Page 1: COMP24111: Machine Learning Ensemble Models Gavin Brown gbrown.](https://reader033.fdocuments.in/reader033/viewer/2022051521/5a4d1ace7f8b9ab059970517/html5/thumbnails/1.jpg)
COMP24111: Machine Learning
Ensemble Models
Gavin Brownwww.cs.man.ac.uk/~gbrown
![Page 2: COMP24111: Machine Learning Ensemble Models Gavin Brown gbrown.](https://reader033.fdocuments.in/reader033/viewer/2022051521/5a4d1ace7f8b9ab059970517/html5/thumbnails/2.jpg)
![Page 3: COMP24111: Machine Learning Ensemble Models Gavin Brown gbrown.](https://reader033.fdocuments.in/reader033/viewer/2022051521/5a4d1ace7f8b9ab059970517/html5/thumbnails/3.jpg)
The Usual Supervised Learning Approach
data + labels
Learning Algorithm
Model
Testing data Predicted label
![Page 4: COMP24111: Machine Learning Ensemble Models Gavin Brown gbrown.](https://reader033.fdocuments.in/reader033/viewer/2022051521/5a4d1ace7f8b9ab059970517/html5/thumbnails/4.jpg)
Ensemble Methods
data + labels
Learning Algorithm
“Committee” of Models
Testing data Predicted label
(voting or averaging)
Model 1
Model 2
Model m
![Page 5: COMP24111: Machine Learning Ensemble Models Gavin Brown gbrown.](https://reader033.fdocuments.in/reader033/viewer/2022051521/5a4d1ace7f8b9ab059970517/html5/thumbnails/5.jpg)
Ensemble Methods
Like a “committee”….
- don’t want all models to be identical.- don’t want them to be different for the sake of it – sacrificing individual performance
This is the “Diversity” trade-off.
Equivalent to underfitting/overfitting – need the right level, just enough.
“Committee” of Models
Model 1
Model 2
Model m
![Page 6: COMP24111: Machine Learning Ensemble Models Gavin Brown gbrown.](https://reader033.fdocuments.in/reader033/viewer/2022051521/5a4d1ace7f8b9ab059970517/html5/thumbnails/6.jpg)
Parallel Ensemble Methods
data + labels
Vote
Model 1 Model 2 Model m
Dataset 1 Dataset 2 Dataset m
Ensemble
![Page 7: COMP24111: Machine Learning Ensemble Models Gavin Brown gbrown.](https://reader033.fdocuments.in/reader033/viewer/2022051521/5a4d1ace7f8b9ab059970517/html5/thumbnails/7.jpg)
Generating a new dataset by “Bootstrapping”
- sample N items with replacement from the original N
![Page 8: COMP24111: Machine Learning Ensemble Models Gavin Brown gbrown.](https://reader033.fdocuments.in/reader033/viewer/2022051521/5a4d1ace7f8b9ab059970517/html5/thumbnails/8.jpg)
“Bagging” : Bootstrap AGGregatING
data + labels
Model 1
Dataset 1 Bootstrap: Sample N examples,with replacement, from the original N.
![Page 9: COMP24111: Machine Learning Ensemble Models Gavin Brown gbrown.](https://reader033.fdocuments.in/reader033/viewer/2022051521/5a4d1ace7f8b9ab059970517/html5/thumbnails/9.jpg)
“Bagging” : Bootstrap AGGregatING
data + labels
Vote
Model 1 Model 2 Model m
Bootstrap 1 Bootstrap 2 Bootstrap m
Bagged Ensemble
![Page 10: COMP24111: Machine Learning Ensemble Models Gavin Brown gbrown.](https://reader033.fdocuments.in/reader033/viewer/2022051521/5a4d1ace7f8b9ab059970517/html5/thumbnails/10.jpg)
The “Bagging” algorithm
![Page 11: COMP24111: Machine Learning Ensemble Models Gavin Brown gbrown.](https://reader033.fdocuments.in/reader033/viewer/2022051521/5a4d1ace7f8b9ab059970517/html5/thumbnails/11.jpg)
Model Stability
Some models are almost completely unaffected by bootstrapping.
These are stable models. Not so suitable for ensemble methods.
Highly Stable
Highly Unstable
KNN
Naïve Bayes
Logistic Regression / Perceptrons
Neural Networks
Decision Trees
![Page 12: COMP24111: Machine Learning Ensemble Models Gavin Brown gbrown.](https://reader033.fdocuments.in/reader033/viewer/2022051521/5a4d1ace7f8b9ab059970517/html5/thumbnails/12.jpg)
![Page 13: COMP24111: Machine Learning Ensemble Models Gavin Brown gbrown.](https://reader033.fdocuments.in/reader033/viewer/2022051521/5a4d1ace7f8b9ab059970517/html5/thumbnails/13.jpg)
Use bagging on a set of decision trees.
How can we make them even more diverse?
![Page 14: COMP24111: Machine Learning Ensemble Models Gavin Brown gbrown.](https://reader033.fdocuments.in/reader033/viewer/2022051521/5a4d1ace7f8b9ab059970517/html5/thumbnails/14.jpg)
Random Forests
![Page 15: COMP24111: Machine Learning Ensemble Models Gavin Brown gbrown.](https://reader033.fdocuments.in/reader033/viewer/2022051521/5a4d1ace7f8b9ab059970517/html5/thumbnails/15.jpg)
Real-time human pose recognition in parts from single depth imagesComputer Vision and Pattern Recognition 2011Shotton et al, Microsoft Research
• Basis of Kinect controller
• Features are simple image properties
• Test phase: 200 frames per sec on GPU
• Train phase more complex but still parallel
![Page 16: COMP24111: Machine Learning Ensemble Models Gavin Brown gbrown.](https://reader033.fdocuments.in/reader033/viewer/2022051521/5a4d1ace7f8b9ab059970517/html5/thumbnails/16.jpg)
Sequential Ensemble Methods
data + labels
Model 1 Model 2
Dataset 2
Model 3
Dataset 3
Model 4
Dataset 4
Each model corrects the mistakes of its predecessor.
![Page 17: COMP24111: Machine Learning Ensemble Models Gavin Brown gbrown.](https://reader033.fdocuments.in/reader033/viewer/2022051521/5a4d1ace7f8b9ab059970517/html5/thumbnails/17.jpg)
Boosting – an informal description
![Page 18: COMP24111: Machine Learning Ensemble Models Gavin Brown gbrown.](https://reader033.fdocuments.in/reader033/viewer/2022051521/5a4d1ace7f8b9ab059970517/html5/thumbnails/18.jpg)
Boosting
![Page 19: COMP24111: Machine Learning Ensemble Models Gavin Brown gbrown.](https://reader033.fdocuments.in/reader033/viewer/2022051521/5a4d1ace7f8b9ab059970517/html5/thumbnails/19.jpg)
Ensemble Methods: Summary
• Treats models as a committee• Two main types – dependent and independent§(boosting and bagging are two examples)
• Responsible for major advances in industrial MLRandom Forests = KinectBoosting = Face Recognition in Phone Cameras
• Tend to work extremely well “off-the-shelf” – no parameter tuning.
![Page 20: COMP24111: Machine Learning Ensemble Models Gavin Brown gbrown.](https://reader033.fdocuments.in/reader033/viewer/2022051521/5a4d1ace7f8b9ab059970517/html5/thumbnails/20.jpg)