1

A PAC-Bayes Risk Bound for General Loss Functions

NIPS 2006NIPS 2006

Pascal Germain, Alexandre Lacasse, François Laviolette, Mario Marchand Université Laval, Québec, Canada

2

Summary

We provide a (tight) PAC-Bayesian bound for the expected loss of convex combinations of classifiers under a wide class of loss functions like the exponential loss and the logistic loss.

Experiments with Adaboost indicate that the upper bound (computed on the training set) behaves very similarly as the true loss (estimated on the testing set).

3

Convex Combinations of Classifiers

Consider any set H of {-1, +1}-valued classifiers and any posterior Q on H .

For any input example x, the [-1,+1]-valued output fQ(x) of a convex combination of classifiers is given by

4

The Margin and WQ(x,y) WQ(x,y) is the fraction, under measure Q, of

classifiers that err on example (x,y)

It is relate to the margin y fQ(x) by

5

General Loss Functions Q(x,y)

Hence, we consider any loss function Q(x,y) that can be written as a Taylor series

and our task is to provide tight bounds for the expected loss Q that depend on the empirical loss measured on a training set of m examples, where

6

Bounds for the Majority Vote A bound on Q also provides a bound on the majority

vote since

7

A PAC-Bayes Bound on Q

8

Proof

where h1-k denotes the product of k classifiers. Hence

9

Proof (cnt.) Let us define the “error rate” R(h1-k ) as

to relate Q to the error rate of a new Gibbs classifier:

10

Proof (ctn.) Where is a distribution over products of

classifiers that works as follows: A number k is chosen according to

k classifiers in H are chosen according to Qk

So denotes the risk of this Gibbs classifier:

11

Proof (ctn.) The standard PAC-Bayes theorem implies that for

any prior on H* = [k2 N+ Hk , we have

Our theorem follows for any having the same structure of (i.e: k is first chosen according to |g(k)|/c, then k

classifiers are chosen accord. to Pk) since, in that case, we have

12

Remark Since we have

any looseness in the bound for R(GQ) will be amplified by c on the bound for Q.

Hence, the bound on Q can be tight only for small c.

This is the case for Q(x,y) = |fQ(x) – y|r since we have c = 1 for r = 1 and c = 3 for r = 2.

13

Bound Behavior During Adaboost

Here H is the set of decision stumps. The output h(x) of decision stump h on attribute x with threshold t is given by h(x) = sgn(x-t) .

If P(h) = 1/|H| hH, then

H(Q) generally increases at each boosting round

14

Results for the Exponential Loss

For this loss function, we have

Since c increases exponentially rapidly with , so will the risk bound.

15

Exponential Loss Results (ctn.)

16

Exponential Loss Results (ctn.)

17

Results for the Sigmoid Loss

For this loss function, we have

The Taylor series for tanh(x) converges only for |x| < /2. We are thus limited to < /2.

18

Sigmoid Loss Results (ctn.)

19

Conclusion We have obtained PAC-Bayesian risk bounds for

any loss function Q having a convergent Taylor expansion around WQ = ½.

The bound is tight only for small c.

On Adaboost, the loss bound is basically parallel to the true loss.