BY :-

Support Vector Machine

CONTENTS

Classifiers

Difference b/w Classification and Clustering

What is SVM

Supervised learning

Linear SVM

NON Linear SVM

Features and Application

CLASSIFIERS

The the goal of Classifiers is to use an object's characteristics to identify which class (or group) it belongs to.

Have labels for some points

Supervised learning

Feature X

Feat

ure

Y

Genes

Proteins

DIFFERENCE B/W CLASSIFICATIONAND CLUSTERING

In general, in classification you have a set of predefined classes and want to know which class a new object belongs to.

Clustering tries to group a set of object.

In the context of machine learning, classification is supervised learning and clustering is unsupervised learning.

WHAT IS SVM?

Support Vector Machines are based on the concept of decision planes that define decision boundaries.

A decision plane is one that separates between a set of objects having different class memberships.

the objects belong either to class GREEN or RED.

The separating line defines a boundary on the right side of which all objects are GREEN and to the left of which all objects are RED. Any new object (white circle) falling to the right is labeled, i.e., classified, as GREEN (or classified as RED should it fall to the left of the separating line).

This is a classic example of a linear classifier

Most classification tasks are not as simple, as we have seen in previous example

More complex structures are needed in order to make an optimal separation

Full separation of the GREEN and RED objects would require a curve (which is more complex than a line).

In fig. we can see the original objects (left side of the schematic) mapped, i.e., rearranged, using a set of mathematical functions, known as kernels.

The process of rearranging the objects is known as mapping (transformation). Note that in this new setting, the mapped objects (right side of the schematic) is linearly separable and, thus, instead of constructing the complex curve (left schematic), all we have to do is to find an optimal line that can separate the GREEN and the RED objects.

Support Vector Machine (SVM) is primarily a classier method that performs classification tasks by constructing hyperplanes in a multidimensional space that separates cases of different class labels.

SVM supports both regression and classification tasks and can handle multiple continuous and categorical variables. For categorical variables a dummy variable is created with case values as either 0 or 1. Thus, a categorical dependent variable consisting of three levels, say (A, B, C), is represented by a set of three dummy variables:

A: {1 0 0}, B: {0 1 0}, C: {0 0 1}

SUPPORT VECTOR MACHINES

SUPERVISED LEARNING

Training set: a number of expression profiles with known labels which represent the true population.

Difference to clustering: there you don't know the labels,you have to find a structure on your own.

Learning/Training:find a decision rule which explains the training set well.

This is the easy part, because we know the labels of the training set!

Generalisation ability: how does the decision rule learned from the training set generalize to new specimen?

Goal: find a decision rule with high generalisation ability.

LINEAR SEPARATORS

Binary classification can be viewed as the task of separating classes in feature space:

wTx + b = 0

wTx + b < 0wTx + b > 0

f(x) = sign(wTx + b)

LINEAR SEPARATION OF THE TRAINING SET

A separating hyperplane is defined by

- the normal vector w and

- the offset b:

hyperplane = {x |<w,x>+ b = 0}

<.,.> is called inner product, scalar product or dot product.

Training: Choose w and b from the labelled examples in the training set.

PREDICT THE LABEL OF A NEW POINT

Prediction: On which side of the hyper-plane does the new point lie?

Points in the direction of the normal vector are classified as POSITIVE.

Points in the opposite direction are classified as NEGATIVE.

WHICH OF THE LINEAR SEPARATORS IS OPTIMAL?

CLASSIFICATION MARGIN

Distance from example xi to the separator is

Examples closest to the hyperplane are support vectors.

Margin ρ of the

separator is the

distance between

support vectors.

w

xw br i

T

r

ρ

MAXIMUM MARGIN CLASSIFICATION

Maximizing the margin is good according to intuition and PAC theory.

Implies that only support vectors matter; other training examples are ignorable.

LINEAR SVM MATHEMATICALLY

Let training set {(xi, yi)}i=1..n, xiRd, yi {-1, 1} be separated by a hyperplane with margin ρ. Then for each training example (xi, yi):

For every support vector xs the above inequality is an equality. After rescaling w and b by ρ/2 in the equality, we obtain that distance between each xs and the hyperplane is

Then the margin can be expressed through (rescaled) w and b as:

wTxi + b ≤ - ρ/2 if yi = -1wTxi + b ≥ ρ/2 if yi = 1

w

22 r

ww

xw 1)(y

br s

Ts

yi(wTxi + b) ≥ ρ/2

LINEAR SVMS MATHEMATICALLY (CONT.)

Then we can formulate the quadratic optimization problem:

Which can be reformulated as:

Find w and b such that

is maximized

and for all (xi, yi), i=1..n : yi(wTxi + b) ≥ 1w

2

Find w and b such that

Φ(w) = ||w||2=wTw is minimized

and for all (xi, yi), i=1..n : yi (wTxi + b) ≥ 1

SOLVING THE OPTIMIZATION PROBLEM

Need to optimize a quadratic function subject to linear constraints.

Quadratic optimization problems are a well-known class of mathematical programming problems for which several (non-trivial) algorithms exist.

The solution involves constructing a dual problem where a Lagrange multiplier αi is associated with every inequality constraint in the primal (original) problem:

Find w and b such thatΦ(w) =wTw is minimized and for all (xi, yi), i=1..n : yi (wTxi + b) ≥ 1

Find α1…αn such thatQ(α) =Σαi - ½ΣΣαiαjyiyjxi

Txj is maximized and (1) Σαiyi = 0(2) αi ≥ 0 for all αi

THE OPTIMIZATION PROBLEM SOLUTION

Given a solution α1…αn to the dual problem, solution to the primal is:

Each non-zero αi indicates that corresponding xi is a support vector.

Then the classifying function is (note that we don’t need w explicitly):

Notice that it relies on an inner product between the test point x and the support vectors xi – we will return to this later.

Also keep in mind that solving the optimization problem involved computing the inner products xi

Txj between all training points.

w =Σαiyixi b = yk - Σαiyixi Txk for any αk > 0

f(x) = ΣαiyixiTx + b

SOFT MARGIN CLASSIFICATION

What if the training set is not linearly separable?

Slack variables ξi can be added to allow misclassification of difficult or noisy examples, resulting margin called soft.

ξi

ξi

SOFT MARGIN CLASSIFICATION MATHEMATICALLY

The old formulation:

Modified formulation incorporates slack variables:

Parameter C can be viewed as a way to control overfitting: it “trades off” the relative importance of maximizing the margin and fitting the training data.

Find w and b such that Φ(w) =wTw is minimized and for all (xi ,yi), i=1..n : yi (wTxi + b) ≥ 1

Find w and b such thatΦ(w) =wTw + CΣξi is minimized and for all (xi ,yi), i=1..n : yi (wTxi + b) ≥ 1 – ξi, , ξi ≥ 0

SOFT MARGIN CLASSIFICATION – SOLUTION

Dual problem is identical to separable case (would not be identical if the 2-norm penalty for slack variables CΣξi

2 was used in primal objective, we would need additional Lagrange multipliers for slack variables):

Again, xi with non-zero αi will be support vectors.

Solution to the dual problem is:

Find α1…αN such thatQ(α) =Σαi - ½ΣΣαiαjyiyjxi

Txj is maximized and (1) Σαiyi = 0(2) 0 ≤ αi ≤ C for all αi

w =Σαiyixi

b= yk(1- ξk) - ΣαiyixiTxk for any k s.t. αk>0 f(x) = Σαiyixi

Tx + b

Again, we don’t need to compute w explicitly for classification:

THEORETICAL JUSTIFICATION FOR MAXIMUM MARGINS

Vapnik has proved the following:

The class of optimal linear separators has VC dimension h bounded from above as

where ρ is the margin, D is the diameter of the smallest sphere that can enclose all of the training examples, and m0 is the dimensionality.

Intuitively, this implies that regardless of dimensionality m0 we can minimize the VC dimension by maximizing the margin ρ.

Thus, complexity of the classifier is kept small regardless of dimensionality.

1,min 02

2

m

Dh

LINEAR SVMS: OVERVIEW

The classifier is a separating hyperplane.

Most “important” training points are support vectors; they define the hyperplane.

Quadratic optimization algorithms can identify which training points xi are support vectors with non-zero Lagrangian multipliers αi.

Both in the dual formulation of the problem and in the solution training points appear only inside inner products:

Find α1…αN such thatQ(α) =Σαi - ½ΣΣαiαjyiyjxi

Txj is maximized and (1) Σαiyi = 0(2) 0 ≤ αi ≤ C for all αi

f(x) = ΣαiyixiTx + b

NON-LINEAR SVMS Datasets that are linearly separable with some noise

work out great:

But what are we going to do if the dataset is just too hard?

How about… mapping data to a higher-dimensional space:

0

0

0

x2

x

x

x

NON-LINEAR SVMS: FEATURE SPACES

General idea: the original feature space can always be mapped to some higher-dimensional feature space where the training set is separable:

Φ: x → φ(x)

PROPERTIES OF SVM

Flexibility in choosing a similarity function Sparseness of solution when dealing with

large data sets - only support vectors are used to specify the

separating hyperplane

Ability to handle large feature spaces

- complexity does not depend on the dimensionality of the feature space

Guaranteed to converge to a single global solution

SVM APPLICATIONS

SVM has been used successfully in many real-world problems

- text (and hypertext) categorization

- image classification

- bioinformatics (Protein classification,

Cancer classification)

- hand-written character recognition

THANK YOU