Adaboost Matas

7/31/2019 Adaboost Matas

1/136

AdaBoost

Jiri Matas and JanSochman

Centre for Machine PerceptionCzech Technical University, Prague

http://cmp.felk.cvut.cz
http://prevpage/


2/136

Presentation

Outline:

AdaBoost algorithm

Why is of interest? How it works? Why it works?

AdaBoost variants

AdaBoost with a Totally Corrective Step (TCS)

Experiments with a Totally Corrective Step

1

2

3

4

5

6

78

9

10

1112

13

14

15

16
http://cmp.felk.cvut.cz/http://cmp.felk.cvut.cz/http://prevpage/


3/136

Introduction

1990 Boost-by-majority algorithm (Freund)

1995 AdaBoost (Freund & Schapire)

1997 Generalized version of AdaBoost (Schapire & Singer)

2001 AdaBoost in Face Detection (Viola & Jones)

Interesting properties:

AB is a linear classifier with all its desirable properties.

AB output converges to the logarithm of likelihood ratio.

AB has good generalization properties.

AB is a feature selector with a principled strategy (minimisation of upperbound on empirical error).

AB close to sequential decision making (it produces a sequence of graduallymore complex classifiers).

1

2

3

4

5

6

78

9

10

1112

13

14

15

16
http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://cmp.felk.cvut.cz/http://prevpage/http://cmp.felk.cvut.cz/http://prevpage/


4/136

What is AdaBoost?

AdaBoost is an algorithm for constructing a strong classifier as linear

combination

f(x) =

Tt=1

tht(x)

of simple weak classifiers ht(x).

1

2

3

4

5

6

78

9

10

1112

13

14

15

16
http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://cmp.felk.cvut.cz/http://prevpage/http://cmp.felk.cvut.cz/


5/136

What is AdaBoost?


combination

f(x) =

Tt=1

tht(x)


Terminology

ht(x) ... weak or basis classifier, hypothesis, feature

H(x) = sign(f(x)) ... strong or final classifier/hypothesis

1

2

3

4

5

6

78

9

10

1112

13

14

15

16
http://cmp.felk.cvut.cz/http://cmp.felk.cvut.cz/


6/136

What is AdaBoost?


combination

f(x) =

Tt=1

tht(x)


Terminology

ht(x) ... weak or basis classifier, hypothesis, feature

H(x) = sign(f(x)) ... strong or final classifier/hypothesis

Comments

The ht(x)s can be thought of as features.

Often (typically) the set H = {h(x)} is infinite.

1

2

3

4

5

6

78

9

10

1112

13

14

15

16


7/136

(Discrete) AdaBoost Algorithm Singer & Schapire (1997)

Given: (x1, y1), ..., (xm, ym); xi X, yi {1, 1}Initialize weights D1(i) = 1/m

For t = 1,...,T:

1. (Call WeakLearn), which returns the weak classifier ht : X {1, 1} withminimum error w.r.t. distribution Dt;

2. Choose t

R,

3. Update

Dt+1(i) =Dt(i)exp(tyiht(xi))

Zt

where Zt

is a normalization factor chosen so that Dt+1

is a distribution

Output the strong classifier:

H(x) = sign

T

t=1

tht(x)

1

2

34

5

6

78

9

10

1112

13

14

15

16


8/136

(Discrete) AdaBoost Algorithm Singer & Schapire (1997)


For t = 1,...,T:


2. Choose t R,3. Update


Zt

where Zt is a normalization factor chosen so that Dt+1 is a distribution

Output the strong classifier:

H(x) = sign

Tt=1

tht(x)

Comments

The computational complexity of selecting ht is independent of t. All information about previously selected features is captured in Dt!

1

2

34

5

6

78

9

10

11

12

13

14

15

16
http://cmp.felk.cvut.cz/


9/136


10/136

WeakLearn

Loop step: Call WeakLearn, given distribution Dt;returns weak classifier ht : X {1, 1} from H = {h(x)}

Select a weak classifier with the smallest weighted errorht = arg min

hjHj =

mi=1 Dt(i)[yi = hj(xi)]

Prerequisite: t < 1/2 (otherwise stop)

WeakLearnexamples:

Decision tree builder, perceptron learning rule H infinite Selecting the best one from given finiteset H

Demonstration example

Weak classifier = perceptron

N(0, 1) 1r83

e1/2(r4)2

1

2

34

5

6

78

9

10

11

12

13

14

15

16


11/136

WeakLearn



hjHj =



WeakLearnexamples:



Training set Weak classifier = perceptron

N(0, 1) 1r83e1/2(r4)2

1

2

34

5

6

78

9

10

11

12

13

14

15

16
http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://cmp.felk.cvut.cz/http://prevpage/http://cmp.felk.cvut.cz/


12/136

WeakLearn



hjHj =



WeakLearnexamples:



Training set Weak classifier = perceptron

N(0, 1) 1r83e1/2(r4)2

1

2

34

5

6

78

9

10

11

12

13

14

15

16


13/136

AdaBoost as a Minimiser of an Upper Bound on the Empirical Error

The main objective is to minimize tr =1m|{i : H(xi) = yi}|

It can be upper bounded by tr(H) Tt=1 Zt

1

2

34

5

6

78

9

10

11

12

13

14

15

16

Ad B Mi i i f U B d h E i i l E
http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://cmp.felk.cvut.cz/http://prevpage/http://cmp.felk.cvut.cz/


14/136



It can be upper bounded by tr(H) Tt=1 Zt

How to set t?

Select t to greedily minimize Zt() in each step

Zt() is convex differentiable function with one extremum

ht(x) {1, 1} then optimal t = 12 log(1+rt1rt)where rt =

mi=1 Dt(i)ht(xi)yi

Zt = 2t(1 t) 1 for optimal t Justification of selection of ht according to t

1

2

34

5

6

78

9

10

11

12

13

14

15

16

Ad B t Mi i i f U B d th E i i l E


15/136



It can be upper bounded by tr(H) T

t=1 ZtHow to set t?

Select t to greedily minimize Zt() in each step

Zt() is convex differentiable function with one extremum

ht(x)

{1, 1

}then optimal t =

12 log(

1+rt1rt

)

where rt =m

i=1 Dt(i)ht(xi)yi

Zt = 2

t(1 t) 1 for optimal t Justification of selection of ht according to t

Comments

The process of selecting t and ht(x) can be interpreted as a singleoptimization step minimising the upper bound on the empirical error.Improvement of the bound is guaranteed, provided that t < 1/2.

The process can be interpreted as a component-wise local optimization(Gauss-Southwell iteration) in the (possibly infinite dimensional!) space of

= (1, 2, . . . ) starting from. 0 = (0, 0, . . . ).

1

2

34

5

6

78

9

10

11

12

13

14

15

16

R i hti
http://cmp.felk.cvut.cz/http://prevpage/http://cmp.felk.cvut.cz/http://prevpage/


16/136

Reweighting

Effect on the training set

Reweighting formula:


Zt=

exp(yitq=1 qhq(xi))mt

q=1 Zq

exp(tyiht(xi))

< 1, yi = ht(xi)> 1, yi = ht(xi)

} Increase (decrease) weight of wrongly (correctly)

classified examples. The weight is the upper bound on the

error of a given example!

1

2

34

5

6

78

9

10

11

12

13

14

15

16

Reweighting
http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://cmp.felk.cvut.cz/http://prevpage/http://cmp.felk.cvut.cz/


17/136

Reweighting




Zt=


q=1 Zq

exp(tyiht(xi))

< 1, yi = ht(xi)> 1, yi = ht(xi)




1

2

34

5

6

78

9

10

11

12

13

14

15

16

Reweighting


18/136

Reweighting




Zt=


q=1 Zq

exp(tyiht(xi))

< 1, yi = ht(xi)> 1, yi = ht(xi)




1

2

34

5

6

78

9

10

11

12

13

14

15

16

Reweighting


19/136

Reweighting




Zt=


q=1 Zq

exp(tyiht(xi))

< 1, yi = ht(xi)> 1, yi = ht(xi)




1

2

34

5

6

7

8

9

10

11

12

13

14

15

16

Reweighting


20/136

Reweighting




Zt=


q=1 Zq

exp(tyiht(xi))

< 1, yi = ht(xi)> 1, yi = ht(xi)




2 1.5 1 0.5 0 0.5 1 1.5 2

0

0.5

1

1.5

2

2.5

3

yf(x)

err

1

2

34

5

6

7

8

9

10

11

12

13

14

15

16

Reweighting
http://cmp.felk.cvut.cz/http://prevpage/http://cmp.felk.cvut.cz/http://prevpage/


21/136

Reweighting




Zt=


q=1 Zq

exp(tyiht(xi))

< 1, yi = ht(xi)> 1, yi = ht(xi)




2 1.5 1 0.5 0 0.5 1 1.5 2

0

0.5

1

1.5

2

2.5

3

yf(x)

err

Effect on ht

t minimize Zt i:ht(xi)=yi

Dt+1(i) =

i:ht(xi)=yiDt+1(i)

Error of ht on Dt+1 is 1/2

Next weak classifier is the most

independent one1 2 3 4

e

0.5

t

1

2

34

5

6

7

8

9

10

11

12

13

14

15

16

Summary of the Algorithm
http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://cmp.felk.cvut.cz/http://prevpage/http://prevpage/http://cmp.felk.cvut.cz/http://prevpage/


22/136


1

2

34

5

6

7

8

9

10

11

12

13

14

15

16

http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://cmp.felk.cvut.cz/http://prevpage/http://prevpage/http://cmp.felk.cvut.cz/http://nextpage/http://prevpage/


23/136


Initialization...1

2

34

5

6

7

8

9

10

11

12

13

14

15

16

http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://cmp.felk.cvut.cz/http://nextpage/http://prevpage/http://prevpage/http://cmp.felk.cvut.cz/http://prevpage/


24/136

y g

Initialization...For t = 1,...,T:

1

2

34

5

6

7

8

9

10

11

12

13

14

15

16

http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://cmp.felk.cvut.cz/http://prevpage/http://prevpage/http://cmp.felk.cvut.cz/http://prevpage/


25/136

y g


Find ht = arg minhjH

j =mi=1

Dt(i)[yi = hj(xi)]

t = 11

2

34

5

6

7

8

9

10

11

12

13

14

15

16

http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://cmp.felk.cvut.cz/http://prevpage/http://prevpage/http://cmp.felk.cvut.cz/http://prevpage/


26/136

y g



j =mi=1

Dt(i)[yi = hj(xi)]

If t 1/2 then stop

t = 11

2

34

5

6

7

8

9

10

11

12

13

14

15

16

http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://cmp.felk.cvut.cz/http://prevpage/http://prevpage/http://cmp.felk.cvut.cz/http://prevpage/


27/136



j =mi=1

Dt(i)[yi = hj(xi)]

If t 1/2 then stop Set t =

12

log(1+rt1rt)

t = 11

2

34

5

6

7

8

9

10

11

12

13

14

15

16

http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://cmp.felk.cvut.cz/http://prevpage/http://prevpage/http://cmp.felk.cvut.cz/http://prevpage/


28/136


Find ht = arg minhjH j =

mi=1

Dt(i)[yi = hj(xi)]


12

log(1+rt1rt)

Update


Zt

t = 11

2

34

5

6

7

8

9

10

11

12

13

14

15

16

http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://cmp.felk.cvut.cz/http://prevpage/http://cmp.felk.cvut.cz/


29/136



mi=1

Dt(i)[yi = hj(xi)]


12

log(1+rt1rt)

Update


Zt

Output the final classifier:

H(x) = sign

Tt=1

tht(x)

t = 1

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

1

2

34

5

6

7

8

9

10

11

12

13

14

15

16



30/136



mi=1

Dt(i)[yi = hj(xi)]


12

log(1+rt1rt)

Update


Zt


H(x) = sign

Tt=1

tht(x)

t = 2

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16



31/136



mi=1

Dt(i)[yi = hj(xi)]


12

log(1+rt1rt)

Update


Zt


H(x) = sign

Tt=1

tht(x)

t = 3

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16



32/136



mi=1

Dt(i)[yi = hj(xi)]


12

log(1+rt1rt)

Update


Zt


H(x) = sign

Tt=1

tht(x)

t = 4

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16



33/136



mi=1

Dt(i)[yi = hj(xi)]


12

log(1+rt1rt)

Update


Zt


H(x) = sign

Tt=1

tht(x)

t = 5

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16



34/136



mi=1

Dt(i)[yi = hj(xi)] If t 1/2 then stop Set t =

12

log(1+rt1rt)

Update


Zt


H(x) = sign

Tt=1

tht(x)

t = 6

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16



35/136



mi=1


12

log(1+rt1rt)

Update


Zt


H(x) = sign

Tt=1

tht(x)

t = 7

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16



36/136



mi=1


12

log(1+rt1rt)

Update


Zt


H(x) = sign

Tt=1

tht(x)

t = 40

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

Does AdaBoost generalize?


37/136

Margins in SVM

max min(x,y)S

y( h(x))

2

Margins in AdaBoost

max min(x,y)S

y( h(x))1

Maximizing margins in AdaBoost

PS[yf(x) ] 2TTt=1

1t (1 t)1+ where f(x) =

h(x)1

Upper bounds based on margin

PD[yf(x) 0] PS[yf(x) ] + O 1

m

d log2(m/d)

2+ log(1/)

1/2

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

AdaBoost variants


38/136

Freund & Schapire 1995

Discrete (h : X {0, 1}) Multiclass AdaBoost.M1 (h : X {0, 1,...,k}) Multiclass AdaBoost.M2 (h : X [0, 1]k) Real valued AdaBoost.R (Y = [0, 1], h : X [0, 1])

Schapire & Singer 1997

Confidence rated prediction (h : X R, two-class) Multilabel AdaBoost.MR, AdaBoost.MH (different formulation of minimized

loss)

... Many other modifications since then (Totally Corrective AB, Cascaded AB)

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16


39/136

Adaboost with a Totally Corrective Step (TCA)
http://prevpage/http://cmp.felk.cvut.cz/http://prevpage/


40/136


For t = 1,...,T:


2. Choose t R,3. Update Dt+1

4. (Call WeakLearn) on the set of hms with non zero s . Update ..Update Dt+1. Repeat till |t 1/2| < , t.

Comments

All weak classifiers have t 1/2, therefore the classifier selected at t + 1 isindependent of all classifiers selected so far.

It can be easily shown, that the totally corrective step reduces the upperbound on the empirical error without increasing classifier complexity.

The TCA was first proposed by Kivinen and Warmuth, but their t is set as instadard Adaboost.

Generalization of TCA is an open question.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

Experiments with TCA on the IDA Database
http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://cmp.felk.cvut.cz/


41/136

Discrete AdaBoost, Real AdaBoost, and Discrete and Real TCA evaluated

Weak learner: stumps. Data from the IDA repository (Ratsch:2000):

Input Training Testing Number ofdimension patterns patterns realizations

Banana 2 400 4900 100Breast cancer 9 200 77 100

Diabetes 8 468 300 100German 20 700 300 100Heart 13 170 100 100Image segment 18 1300 1010 20Ringnorm 20 400 7000 100Flare solar 9 666 400 100

Splice 60 1000 2175 20Thyroid 5 140 75 100Titanic 3 150 2051 100Twonorm 20 400 7000 100Waveform 21 400 4600 100

Note that the training sets are fairly small

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

Results with TCA on the IDA Database


42/136

Training error (dashed line), test error (solid line)

Discrete AdaBoost (blue), Real AdaBoost (green),

Discrete AdaBoost with TCA (red), Real AdaBoost with TCA (cyan)

the black horizontal line: the error of AdaBoost with RBF network weak classifiers from (Ratsch-ML:2000)

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16



43/136





100

101

102

103

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

Length of the strong classifier

IMAGE

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

http://prevpage/http://prevpage/http://cmp.felk.cvut.cz/http://prevpage/


44/136





100

101

102

103

0.32

0.34

0.36

0.38

0.4

0.42

0.44

0.46


FLARE

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16


1
http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://cmp.felk.cvut.cz/


45/136





100

101

102

103

0.16

0.18

0.2

0.22

0.24

0.26

0.28

0.3

0.32


GERMAN

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16


1
http://prevpage/http://prevpage/http://cmp.felk.cvut.cz/http://prevpage/


46/136





100

101

102

103

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4


RINGNORM

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16


1
http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://prevpage/http://cmp.felk.cvut.cz/


47/136





100

101

102

103

0

0.05

0.1

0.15

0.2

0.25


SPLICE

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16


1


48/136





100

101

102

103

0

0.05

0.1

0.15

0.2

0.25


THYROID

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16


1


49/136





100

101

102

103

0.21

0.215

0.22

0.225

0.23

0.235

0.24


TITANIC

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16


1


50/136





100

101

102

103

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45


BANANA

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16


1


51/136





100

101

102

103

0.24

0.25

0.26

0.27

0.28

0.29

0.3

0.31


BREAST

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16


1


52/136





100

101

102

103

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35


DIABETIS

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16


1


53/136





100

101

102

103

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35


HEART

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

Conclusions

1


54/136

The AdaBoost algorithm was presented and analysed

A modification of the Totally Corrective AdaBoost was introduced

Initial test show that the TCA outperforms AB on some standard data sets.

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16


55/136
http://prevpage/


56/136
http://prevpage/http://prevpage/


57/136


58/136
http://prevpage/


59/136
http://prevpage/


60/136


61/136


62/136

3


63/136

2 1.5 1 0.5 0 0.5 1 1.5 2

0

0.5

1

1.5

2

2.5

yf(x)

err

e

0.5
http://prevpage/


64/136

1 2 3 4 t
http://prevpage/


65/136


66/136


67/136

3


68/136

2 1.5 1 0.5 0 0.5 1 1.5 2

0

0.5

1

1.5

2

2.5

yf(x)

err

3
http://prevpage/


69/136

2 1.5 1 0.5 0 0.5 1 1.5 2

0

0.5

1

1.5

2

2.5

yf(x)

err

e

0.5


70/136

1 2 3 4 t


71/136


72/136


73/136


74/136


75/136


76/136


77/136

0.3

0.35


78/136

0 5 10 15 20 25 30 35 40

0

0.05

0.1

0.15

0.2

0.25


79/136

0.3

0.35


80/136

0 5 10 15 20 25 30 35 40

0

0.05

0.1

0.15

0.2

0.25


81/136

0.3

0.35


82/136

0 5 10 15 20 25 30 35 40

0

0.05

0.1

0.15

0.2

0.25


83/136

0.3

0.35


84/136

0 5 10 15 20 25 30 35 40

0

0.05

0.1

0.15

0.2

0.25


85/136

0.3

0.35


86/136

0 5 10 15 20 25 30 35 40

0

0.05

0.1

0.15

0.2

0.25


87/136

0.3

0.35


88/136

0 5 10 15 20 25 30 35 40

0

0.05

0.1

0.15

0.2

0.25


89/136

0.3

0.35


90/136

0 5 10 15 20 25 30 35 40

0

0.05

0.1

0.15

0.2

0.25


91/136

0.3

0.35


92/136

0 5 10 15 20 25 30 35 40

0

0.05

0.1

0.15

0.2

0.25


93/136


94/136


95/136


96/136


97/136


98/136


99/136

0.3

0.35


100/136

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25


101/136

0.3

0.35


102/136

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25


103/136

0.3

0.35


104/136

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25


105/136

0.3

0.35


106/136

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25


107/136

0.3

0.35


108/136

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25


109/136

0.3

0.35


110/136

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25


111/136

0.3

0.35


112/136

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25
http://nextpage/http://prevpage/


113/136

0.3

0.35
http://nextpage/http://prevpage/http://prevpage/


114/136

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

IMAGE
http://prevpage/


115/136

100

101

102

103

0

0.05

0.1

0.15

0.2

0.25


0.44

0.46

FLARE


116/136

100

101

102

103

0.32

0.34

0.36

0.38

0.4

0.42


0.28

0.3

0.32

GERMAN
http://prevpage/


117/136

100

101

102

103

0.16

0.18

0.2

0.22

0.24

0.26


0.3

0.35

0.4

RINGNORM
http://prevpage/http://nextpage/http://prevpage/


118/136

100

101

102

103

0

0.05

0.1

0.15

0.2

0.25



119/136

0.2

0.25THYROID


120/136

100

101

102

103

0

0.05

0.1

0.15


0.235

0.24TITANIC
http://nextpage/http://prevpage/http://nextpage/http://prevpage/


121/136

100

101

102

103

0.21

0.215

0.22

0.225

0.23


0.35

0.4

0.45BANANA
http://nextpage/http://prevpage/http://nextpage/http://prevpage/


122/136

100

101

102

103

0.05

0.1

0.15

0.2

0.25

0.3



123/136

0 25

0.3

0.35DIABETIS


124/136

100

101

102

103

0

0.05

0.1

0.15

0.2

0.25


0 25

0.3

0.35HEART


125/136

100

101

102

103

0

0.05

0.1

0.15

0.2

0.25


0.3

0.35

0.4IMAGE


126/136

100

101

102

103

0

0.05

0.1

0.15

0.2

0.25


0 42

0.44

0.46FLARE


127/136

100

101

102

103

0.32

0.34

0.36

0.38

0.4

0.42


0.28

0.3

0.32GERMAN


128/136

100

101

102

103

0.16

0.18

0.2

0.22

0.24

0.26


0.3

0.35

0.4RINGNORM


129/136

100

101

102

103

0

0.05

0.1

0.15

0.2

0.25


0.2

0.25SPLICE


130/136

100

101

102

103

0

0.05

0.1

0.15


0.2

0.25THYROID


131/136

100

101

102

103

0

0.05

0.1

0.15


0.235

0.24TITANIC


132/136

100

101

102

103

0.21

0.215

0.22

0.225

0.23


0.35

0.4

0.45BANANA


133/136

100

101

102

103

0.05

0.1

0.15

0.2

0.25

0.3


0.29

0.3

0.31BREAST


134/136

100

101

102

103

0.24

0.25

0.26

0.27

0.28


0.25

0.3

0.35DIABETIS


135/136

100

101

102

103

0

0.05

0.1

0.15

0.2


0.25

0.3

0.35HEART


136/136

100

101

102

103

0

0.05

0.1

0.15

0.2


Adaboost Matas

Documents

Transcript of Adaboost Matas