A Brief Introduction to Adaboost - Middle East Technical...
-
Upload
vuongduong -
Category
Documents
-
view
217 -
download
4
Transcript of A Brief Introduction to Adaboost - Middle East Technical...
![Page 1: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/1.jpg)
1
A Brief Introduction
to Adaboost
Hongbo Deng
6 Feb, 2007
Some of the slides are borrowed from Derek Hoiem & Jan ˇSochman.
![Page 2: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/2.jpg)
2
Outline
Background
Adaboost Algorithm
Theory/Interpretations
![Page 3: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/3.jpg)
3
What’s So Good About Adaboost
Can be used with many different classifiers
Improves classification accuracy
Commonly used in many areas
Simple to implement
Not prone to overfitting
![Page 4: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/4.jpg)
4
Bootstrapping
Bagging
Boosting (Schapire 1989)
Adaboost (Schapire 1995)
A Brief History Resampling for
estimating statistic
Resampling for
classifier
design
![Page 5: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/5.jpg)
5
Bootstrap Estimation
Repeatedly draw n samples from D
For each set of samples, estimate a
statistic
The bootstrap estimate is the mean of the
individual estimates
Used to estimate a statistic (parameter)
and its variance
![Page 6: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/6.jpg)
6
Bagging - Aggregate Bootstrapping
For i = 1 .. M
Draw n*<n samples from D with replacement
Learn classifier Ci
Final classifier is a vote of C1 .. CM
Increases classifier stability/reduces
variance D1
D2
D3 D
![Page 7: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/7.jpg)
7
Boosting (Schapire 1989)
Consider creating three component classifiers for a two-category problem through boosting.
Randomly select n1 < n samples from D without replacement to obtain D1
Train weak learner C1
Select n2 < n samples from D with half of the samples misclassified by C1 to obtain D2 Train weak learner C2
Select all remaining samples from D that C1 and C2 disagree on Train weak learner C3
Final classifier is vote of weak learners D
D1
D2
D3
+ - -
+
![Page 8: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/8.jpg)
8
Adaboost - Adaptive Boosting
Instead of resampling, uses training set re-weighting Each training sample uses a weight to determine the probability
of being selected for a training set.
AdaBoost is an algorithm for constructing a “strong” classifier as linear combination of “simple” “weak” classifier
Final classification based on weighted vote of weak classifiers
![Page 9: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/9.jpg)
9
Adaboost Terminology
ht(x) … “weak” or basis classifier (Classifier =
Learner = Hypothesis)
… “strong” or final classifier
Weak Classifier: < 50% error over any
distribution
Strong Classifier: thresholded linear combination
of weak classifier outputs
![Page 10: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/10.jpg)
10
Discrete Adaboost Algorithm Each training sample has a
weight, which determines the
probability of being selected for
training the component classifier
![Page 11: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/11.jpg)
11
Find the Weak Classifier
![Page 12: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/12.jpg)
12
Find the Weak Classifier
![Page 13: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/13.jpg)
13
The algorithm core
![Page 14: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/14.jpg)
14
Reweighting
y * h(x) = 1
y * h(x) = -1
![Page 15: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/15.jpg)
15
Reweighting
In this way, AdaBoost “focused on” the
informative or “difficult” examples.
![Page 16: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/16.jpg)
16
Reweighting
In this way, AdaBoost “focused on” the
informative or “difficult” examples.
![Page 17: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/17.jpg)
17
Algorithm recapitulation
t = 1
![Page 18: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/18.jpg)
18
Algorithm recapitulation
![Page 19: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/19.jpg)
19
Algorithm recapitulation
![Page 20: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/20.jpg)
20
Algorithm recapitulation
![Page 21: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/21.jpg)
21
Algorithm recapitulation
![Page 22: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/22.jpg)
22
Algorithm recapitulation
![Page 23: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/23.jpg)
23
Algorithm recapitulation
![Page 24: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/24.jpg)
24
Algorithm recapitulation
![Page 25: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/25.jpg)
25
Pros and cons of AdaBoost
Advantages
Very simple to implement
Does feature selection resulting in relatively simple classifier
Fairly good generalization
Disadvantages
Suboptimal solution
Sensitive to noisy data and outliers
![Page 26: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/26.jpg)
26
References Duda, Hart, ect – Pattern Classification
Freund – “An adaptive version of the boost by majority algorithm”
Freund – “Experiments with a new boosting algorithm”
Freund, Schapire – “A decision-theoretic generalization of on-line learning and an application to boosting”
Friedman, Hastie, etc – “Additive Logistic Regression: A Statistical View of Boosting”
Jin, Liu, etc (CMU) – “A New Boosting Algorithm Using Input-Dependent Regularizer”
Li, Zhang, etc – “Floatboost Learning for Classification”
Opitz, Maclin – “Popular Ensemble Methods: An Empirical Study”
Ratsch, Warmuth – “Efficient Margin Maximization with Boosting”
Schapire, Freund, etc – “Boosting the Margin: A New Explanation for the Effectiveness of Voting Methods”
Schapire, Singer – “Improved Boosting Algorithms Using Confidence-Weighted Predictions”
Schapire – “The Boosting Approach to Machine Learning: An overview”
Zhang, Li, etc – “Multi-view Face Detection with Floatboost”
![Page 27: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/27.jpg)
27
Appendix
Bound on training error
Adaboost Variants
![Page 28: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/28.jpg)
28
Bound on Training Error (Schapire)
![Page 29: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/29.jpg)
29
Discrete Adaboost (DiscreteAB)
(Friedman’s wording)
![Page 30: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/30.jpg)
30
Discrete Adaboost (DiscreteAB)
(Freund and Schapire’s wording)
![Page 31: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/31.jpg)
31
Adaboost with Confidence
Weighted Predictions (RealAB)
![Page 32: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/32.jpg)
32
Adaboost Variants Proposed By
Friedman
LogitBoost
Solves
Requires care to avoid numerical problems
GentleBoost
Update is fm(x) = P(y=1 | x) – P(y=0 | x) instead of Bounded [0 1]
![Page 33: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/33.jpg)
33
Adaboost Variants Proposed By
Friedman
LogitBoost
![Page 34: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/34.jpg)
34
Adaboost Variants Proposed By
Friedman
GentleBoost
![Page 35: A Brief Introduction to Adaboost - Middle East Technical ...user.ceng.metu.edu.tr/~tcan/ceng734_f1112/Schedule/adaboost.pdf · 1 A Brief Introduction to Adaboost Hongbo Deng 6 Feb,](https://reader033.fdocuments.in/reader033/viewer/2022051801/5ae63d3c7f8b9acc268d1e06/html5/thumbnails/35.jpg)
35
Thanks!!!
Any comments or questions?