Slide 1

An Introduction to Support Vector Machines Martin Law

Slide 2

08/11/05CSE 802. Prepared by Martin Law2 Outline History of support vector machines (SVM) Two classes, linearly separable What is a good decision boundary? Two classes, not linearly separable How to make SVM non-linear: kernel trick Demo of SVM Epsilon support vector regression ( -SVR) Conclusion

Slide 3

08/11/05CSE 802. Prepared by Martin Law3 History of SVM SVM is a classifier derived from statistical learning theory by Vapnik and Chervonenkis SVM was first introduced in COLT-92 SVM becomes famous when, using pixel maps as input, it gives accuracy comparable to sophisticated neural networks with elaborated features in a handwriting recognition task Currently, SVM is closely related to: Kernel methods, large margin classifiers, reproducing kernel Hilbert space, Gaussian process

Slide 4

08/11/05CSE 802. Prepared by Martin Law4 Two Class Problem: Linear Separable Case Class 1 Class 2 Many decision boundaries can separate these two classes Which one should we choose?

Slide 5

08/11/05CSE 802. Prepared by Martin Law5 Example of Bad Decision Boundaries Class 1 Class 2 Class 1 Class 2

Slide 6

08/11/05CSE 802. Prepared by Martin Law6 Good Decision Boundary: Margin Should Be Large The decision boundary should be as far away from the data of both classes as possible We should maximize the margin, m Class 1 Class 2 m

Slide 7

08/11/05CSE 802. Prepared by Martin Law7 The Optimization Problem Let {x 1,..., x n } be our data set and let y i {1,-1} be the class label of x i The decision boundary should classify all points correctly A constrained optimization problem

Slide 8

08/11/05CSE 802. Prepared by Martin Law8 The Optimization Problem We can transform the problem to its dual This is a quadratic programming (QP) problem Global maximum of i can always be found w can be recovered by

Slide 9

08/11/05CSE 802. Prepared by Martin Law9 Characteristics of the Solution Many of the i are zero w is a linear combination of a small number of data Sparse representation x i with non-zero i are called support vectors (SV) The decision boundary is determined only by the SV Let t j (j=1,..., s) be the indices of the s support vectors. We can write For testing with a new data z Compute and classify z as class 1 if the sum is positive, and class 2 otherwise

Slide 10

08/11/05CSE 802. Prepared by Martin Law10 6 =1.4 A Geometrical Interpretation Class 1 Class 2 1 =0.8 2 =0 3 =0 4 =0 5 =0 7 =0 8 =0.6 9 =0 10 =0

Slide 11

08/11/05CSE 802. Prepared by Martin Law11 Some Notes There are theoretical upper bounds on the error on unseen data for SVM The larger the margin, the smaller the bound The smaller the number of SV, the smaller the bound Note that in both training and testing, the data are referenced only as inner product, x T y This is important for generalizing to the non-linear case

Slide 12

08/11/05CSE 802. Prepared by Martin Law12 How About Not Linearly Separable We allow error i in classification Class 1 Class 2

Slide 13

08/11/05CSE 802. Prepared by Martin Law13 Soft Margin Hyperplane Define i =0 if there is no error for x i i are just slack variables in optimization theory We want to minimize C : tradeoff parameter between error and margin The optimization problem becomes

Slide 14

08/11/05CSE 802. Prepared by Martin Law14 The Optimization Problem The dual of the problem is w is also recovered as The only difference with the linear separable case is that there is an upper bound C on i Once again, a QP solver can be used to find i

Slide 15

08/11/05CSE 802. Prepared by Martin Law15 Extension to Non-linear Decision Boundary Key idea: transform x i to a higher dimensional space to make life easier Input space: the space x i are in Feature space: the space of (x i ) after transformation Why transform? Linear operation in the feature space is equivalent to non-linear operation in input space The classification task can be easier with a proper transformation. Example: XOR

Slide 16

08/11/05CSE 802. Prepared by Martin Law16 Extension to Non-linear Decision Boundary Possible problem of the transformation High computation burden and hard to get a good estimate SVM solves these two issues simultaneously Kernel tricks for efficient computation Minimize ||w|| 2 can lead to a good classifier ( ) (.) ( ) Feature space Input space

Slide 17

08/11/05CSE 802. Prepared by Martin Law17 Example Transformation Define the kernel function K (x,y) as Consider the following transformation The inner product can be computed by K without going through the map (.)

Slide 18

08/11/05CSE 802. Prepared by Martin Law18 Kernel Trick The relationship between the kernel function K and the mapping (.) is This is known as the kernel trick In practice, we specify K, thereby specifying (.) indirectly, instead of choosing (.) Intuitively, K (x,y) represents our desired notion of similarity between data x and y and this is from our prior knowledge K (x,y) needs to satisfy a technical condition (Mercer condition) in order for (.) to exist

Slide 19

08/11/05CSE 802. Prepared by Martin Law19 Examples of Kernel Functions Polynomial kernel with degree d Radial basis function kernel with width Closely related to radial basis function neural networks Sigmoid with parameter and It does not satisfy the Mercer condition on all and Research on different kernel functions in different applications is very active

Slide 20

08/11/05CSE 802. Prepared by Martin Law20 Example of SVM Applications: Handwriting Recognition

Slide 21

08/11/05CSE 802. Prepared by Martin Law21 Modification Due to Kernel Function Change all inner products to kernel functions For training, Original With kernel function

Slide 22

08/11/05CSE 802. Prepared by Martin Law22 Modification Due to Kernel Function For testing, the new data z is classified as class 1 if f 0, and as class 2 if f