Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics...
Transcript of Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics...
![Page 1: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/1.jpg)
Multiclass SVM and Applications in Object
Classification
Yuval Kaminka, Einat GranotAdvanced Topics in Computer Vision Seminar
Faculty of Mathematics and Computer ScienceWeizmann Institute
May 2007
![Page 2: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/2.jpg)
Outline
Motivation and Introduction
Classification Algorithms K-Nearest neighbors (KNN) SVM
Multiclass SVM
DAGSVM
SVM-KNN
Results - A taste of the distance Shape distance (shape context, tangent) Texture (texton histograms)
![Page 3: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/3.jpg)
Object Classification
?
![Page 4: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/4.jpg)
Motivation – Human Visual System
Large Number of Categories (~30,000)
Discriminative Process
Small Set of Examples
Invariance to transformation
Similarity to Prototype instead of Features
![Page 5: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/5.jpg)
Similarity to Prototypes Vs Features
No need for Feature Space
Easy to enlarge number of categories
Includes spatial relation between features
![Page 6: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/6.jpg)
Similarity is defined by Distance Function
Easy to adjust to different types (Shape, Texture)
Can include invariance to intra-class transformations
Distance Function
D( , (
![Page 7: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/7.jpg)
Distance Function – simple example
2.12731 . . . .
D( , ( = || 2.1, 27, 31, 15, 8 . . .
- ||13,45,22.5, 78, 91 . . .
D( , ( =?
MNR
![Page 8: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/8.jpg)
Outline
Motivation and Introduction
Classification Algorithms K-Nearest neighbors (KNN) SVM
Multiclass SVM
DAGSVM
SVM-KNN
Results - A taste of the distance Shape distance (shape context, tangent) Texture (texton histograms)
![Page 9: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/9.jpg)
A Classic Classification Problem
X4
X2
X5
X6
X7
X3
q
X1
Training Set S: (X1..Xn), with class label (Y1.. Yn)
Given a query image q, determine its label
![Page 10: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/10.jpg)
Nearest Neighbor (NN)
?
![Page 11: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/11.jpg)
K-Nearest Neighbor (KNN)
?
K = 3
![Page 12: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/12.jpg)
K-NN Pros
Simple, yet outperforms other methods
Low Complexity: O(D ּn) D - the cost per one distance function calculation
No need for Feature Space definition
No computational cost for adding new categories
n ∞ ==> Error Rate Bayes optimal
![Page 13: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/13.jpg)
K-NN Cons
P. Vincent et al., K-local hyperplane and convex distance nearest neighbor algorithms, NIPS 2001
Complete Set Missing Set
NN SVM
![Page 14: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/14.jpg)
Outline
Motivation and Introduction
Classification Algorithms K-Nearest neighbors (KNN) SVM
Multiclass SVM
DAGSVM
SVM-KNN
Results - A taste of the distance Shape distance (shape context, tangent) Texture (texton histograms)
![Page 15: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/15.jpg)
SVM
Two class classification algorithm
We’re looking for a hyperplane that best separates the classes
Class 1
Class 2
Some of the slides on SVM are adapted with permission from Martin Law’s presentation on SVM
![Page 16: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/16.jpg)
SVM - Motivation
Class 1
Class 2
Class 1
Class 2
As far away as possible from the data of both classes
![Page 17: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/17.jpg)
SVM – A learning algorithm
KNN – simple classification, no training
Class 1
Class 2
SVM – a learning algorithm
Two Phases:1. Training – find the hyperplane2. Classification – label a new query
![Page 18: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/18.jpg)
SVM – Training Phase
Class 1
Class 2
W
wTx+b=0
~b
1. Classify correctly the classes2. Give maximum margins
We’re looking for (w,b) that will:
![Page 19: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/19.jpg)
1 .Correct classification
Class 1
Class 2
wTx+b=0
Correct classification: wTxi+b>0 for green, and wTxi+b<0 for red
Assume the labels {y1.. yn} are from the set {-1,1}:
}x1, ..., xn{our training
set
0)( bxwy iT
i
![Page 20: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/20.jpg)
2 .Margin maximization
Class 1
Class 2
m
m? =
![Page 21: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/21.jpg)
2 .Margin maximization
w
1z
Class 1
Class 2|wTz+b|
||w||
We can scale (w,b) (w,b), >0
Won’t change classification: wTx+b>0 wTx+b>0
Get a desired distance: |wTz+b|=a =1/a, |wTz+b|=1
w
1w
m2
m
![Page 22: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/22.jpg)
SVM as an Optimization Problem
Maximize margins
Correct Classification
We can find n, such that:
wm
2
Solve optimization problem with constraints
Langrangian multipliers
C.J.C. Burges. A Tutorial on Support Vector Machines for Pattern Recognition, 1998.
![Page 23: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/23.jpg)
Support Vectors
Class 1
Class 2
xi with i>0 are called support vectors (SV)
w is determined only by the SV
=0
=0
=0
=0
=0
=0 =0
>0
>0
>0
SVi
iii xyw
n
iiii xyw
1
![Page 24: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/24.jpg)
SVM – Classification phase
Class 1
Compute wTq+b
SVi
Tiii bqxy
Class 2
q
SVi
iii xyw
Classify as class 1 if positive, and class 2 otherwise
![Page 25: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/25.jpg)
Upgrade SVM
1. In order to find 1.. n we need to calculate xiTxj i,j
2. In order to classify a query q we need to calculate:
We only need to calculate inner products
SVi
Tiii
T bqxybqw
![Page 26: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/26.jpg)
Feature Expansion
)(NOR
( )
( )
( )( )( )
( )
( )( )
(.)( )
( )
( )( )
( )
( )
( )( )
( )
Extended spaceInput space
)( kNOR
(.)( 1 , x , y , xy , x2 , y2 )(x , y)
Problem: too expensive!
![Page 27: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/27.jpg)
Solution: The Kernel Trick
)()(),( jT
iji xxxxK
Find a kernel function K such that:
We only need to calculate inner products
( )
( )
( )( )( )
( )
( )( )
(.)( )
( )
( )( )
( )
( )
( )( )
( )
2)1(),( jiji xxxxK
![Page 28: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/28.jpg)
The Kernel Trick
1. In order to find 1.. n we need to calculate xiTxj i,j
Build a kernel matrix MnXn: M[i,j]= (xi)T(xj)=K(xi,xj)
2. In order to classify a query q we need to calculate wTq+b:
SVi
iiiSVi
Tiii
T bqxKybqxybqw ),()()(
We only need to calculate inner products
![Page 29: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/29.jpg)
Inner product Distance Function
yxyxyyxxyx ,,,2
1,
22200
2
1yxyx
),()0,()0,(2
1yxdydxd
From “origin” Pairwise distance
In our case: convert to distance function
We only need to calculate inner products
![Page 30: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/30.jpg)
SVM Pros and Cons
Pros: Easy to integrate different distance functions
Fast classification of new objects (depends on SV)
Good performance even with small set of examples
Cons: Slow training ( O(n2), n=# of vectors in training set )
Separates only 2 classes
![Page 31: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/31.jpg)
Outline
Motivation and Introduction
Classification Algorithms K-Nearest neighbors (KNN) SVM
Multiclass SVM
DAGSVM
SVM-KNN
Results - A taste of the distance Shape distance (shape context, tangent) Texture (texton histograms)
![Page 32: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/32.jpg)
Multiclass SVM
Class 1Class 2
Class 3
Class 5
Class 4
Extend SVM for multi-classes separationNc = number of classes
![Page 33: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/33.jpg)
Two approaches
Class 1Class 2
Class 3
Class 4
Combine multi-binary-classifiers
Generate one function based on single optimization problem
1-vs-rest 1-vs-1 DAGSVM
![Page 34: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/34.jpg)
1-vs-rest
Class 1Class 2
Class 3Class 4
![Page 35: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/35.jpg)
1-vs-rest
Class 1 Class 2
Class 3 Class 4
w1
w3
w4
w2
Nc classifiers
![Page 36: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/36.jpg)
1-vs-rest
Class 1 Class 2
Class 3 Class 4
q
w1
w3
w4
w2
w1Tq+b1 ~ Similarity(q,SV1)
~ Similarity(q,SV3)~ Similarity(q,SV2)
~ Similarity(q,SV4)
![Page 37: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/37.jpg)
1-vs-rest
Class 1 Class 2
Class 3 Class 4
q
w1
w3
w4
w2
Label(q)=
argmax1≤i ≤Nc{Sim(q,SVi)}
![Page 38: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/38.jpg)
1-vs-1
Class 1Class 2
Class 3Class 4
![Page 39: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/39.jpg)
1-vs-1
Class 1Class 2
Class 3
Class 4
W1,2
W1,3
W1,4
W2,3
W3,4
W2,4
Nc(Nc-1)/2 classifiers
![Page 40: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/40.jpg)
1-vs-1 with Max Wins
Class 1Class 2
Class 3
Class 4
W1,2
W1,3
W1,4
W2,3
W3,4
W2,4
q
Sign(w1,2Tq+b1,2) ~ 1 or 2 ?
☺☺☺ ☺
☺☺
~ 1 or 3 ?
~ 1 or 4 ?
~ 2 or 3 ?
~ 3 or 4 ?
~ 2 or 4 ?
![Page 41: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/41.jpg)
1-vs-1 with Max Wins
Class 1Class 2
Class 3
Class 4
W1,2
W1,3
W1,4
W2,3
W3,4
W2,4
q
☺☺☺ ☺
☺☺
![Page 42: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/42.jpg)
What did we have so far?
1-vs-rest1-vs-1
# of classifiers (each need to be
trained and tested)NcNc(Nc-1)/2
# of vectors for training
(per classifier)
n(all vectors)
~2n/Nc
No bound on generalization error
Class 1Class 2
Class 3Class 4
Class 1 Class 2
Class 3Class 4
![Page 43: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/43.jpg)
DAGSVM
Class 1Class 2
Class 3
Class 4
W1,2
W1,3
W1,4
W2,3
W3,4
W2,4
1-vs-1 Decision DAG (DDAG)
1 vs 4
3 vs 4
2 vs 4 1 vs 3
2 vs 3 1 vs 2
1 2 3 4
3 4
2 3 4
1 2
1 2 3
2 3
not 1
not 1not 2 not 3
not 4
not 4
4 123
J. C. Platt et al., Large margin DAGs for multiclass classification. NIPS, 1999.
![Page 44: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/44.jpg)
DDAG on Nc Classes
DAG
Nc leaves, one per class
Single root node
In every node: Binary decision
function
Nc(Nc-1)/2 internal nodes
1 vs 4
3 vs 4
2 vs 4 1 vs 3
2 vs 3 1 vs 2
1 2 3 4
3 4
2 3 4
1 2
1 2 3
2 3
not 1
not 1not 2 not 3
not 4
not 4
4 123
![Page 45: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/45.jpg)
Classification using DDAG
Class 1Class 2
Class 3
Class 4
W1,2
W1,3
W1,4
W2,3
W3,4
W2,4
q1 vs 4
3 vs 4
2 vs 4 1 vs 3
2 vs 3 1 vs 2
1 2 3 4
3 4
2 3 4
1 2
1 2 3
2 3
not 1
not 1not 2 not 3
not 4
not 4
4 123
~ 1 or 4 ?
~ 1 or 3 ?
~ 1 or 2 ?
![Page 46: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/46.jpg)
DAGSVM
Pros: Only Nc-1 classifiers to be tested
Every classifier uses a small set of vectors for training
Bound on generalization error (~margins size)
Cons: Less vectors for training worse classifier?
Nc(Nc-1)/2 classifiers to be trained
![Page 47: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/47.jpg)
DAGSVM Complexity
For training: Assume that every class contains ~n/Nc instances Nc(Nc-1)/2 classifiers, each using ~2n/Nc vectors:
For classifying new objects: Nc-1 classifiers, each is tested once M = max number of SV
)( 2nDO 22 2)2
(2
)1(n
N
nNN
c
cc
)( cNMDO
![Page 48: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/48.jpg)
Multiclass SVM - Summary Training:
Classification:
Error rates: Bound of generalization error - only on DAGSVM In practice – 1-vs-1 and DAGSVM
The “one big optimization” methods Similar error rates Very slow training – limited to small data sets
DAGSVM / 1-vs-11-vs-rest
O(D ּn2)O(D ּNcּn2)
DAGSVM / 1-vs-rest1-vs-1
O(D ּM ּNc)O(D ּM ּNc2)
![Page 49: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/49.jpg)
So what do we have?
Nearest Neighbor (KNN) Fast Suitable for multi-class Easy to integrate different distance functions Problematic with few samples
SVM Good performance even with small set of examples Easy to integrate different distance functions No natural extension to multi-class Slow to train
Class 1
Class 2
![Page 50: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/50.jpg)
SVM KNN - From coarse to fine
Suggestion Hybrid system
KNN SVM
Zhang et al, SVM-KNN: Discriminative Nearest Neighbor Classification for Visual Category Recognition, 2006
![Page 51: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/51.jpg)
Outline
Motivation and Introduction
Classification Algorithms K-Nearest neighbors (KNN) SVM
Multiclass SVM
DAGSVM
SVM-KNN
Results - A taste of the distance Shape distance (shape context, tangent) Texture (texton histograms)
![Page 52: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/52.jpg)
SVM KNN – General Algorithm
1. Calculate distance from query to training images
Training images and query
Query image
KNN
Class 1Class 2
Class 3
![Page 53: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/53.jpg)
SVM KNN – General Algorithm
1. Calculate distance from query to training images
2. Pick K nearest neighbors
Training images and query
Query image
KNN
Class 1Class 2
Class 3
![Page 54: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/54.jpg)
SVM KNN – General Algorithm
1. Calculate distance from query to training images
2. Pick K nearest neighbors3. Run SVM
Training images and query
Query image
SVM
Class 1Class 2
Class 3
SVM works well with few samples
![Page 55: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/55.jpg)
SVM KNN – General Algorithm
1. Calculate distance from query to training images
2. Pick K nearest neighbors3. Run SVM4. Label !
Training images and query
Query image
Query image Class 2
SVM
Class 1Class 2
Class 3
![Page 56: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/56.jpg)
Training + Classification
1. Calculate distance from query to training images
2. Pick K nearest neighbors3. Run SVM4. Label !
KNN
SVM
Classic process: Training Classification
SVM-KNN Coarse Classification Training Final classification
![Page 57: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/57.jpg)
Details Details Details
1. Calculate distance from query to training images
2. Pick K nearest neighbors3. Run SVM4. Label !
KNN
SVM
Kpotential
Calculating distance is a heavy task Compute crude distance – faster
Finding Kpotential images Ignore all other images
Compute accurate distance Only relative to the Kpotential images
Accurate
L2
![Page 58: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/58.jpg)
Details Details Details
1. Calculate distance from query to training images
2. Pick K nearest neighbors3. Run SVM4. Label !
Complexity: Crude distance
Accurate distance
)( potentialAccurate KDO
KNN
SVM
)( nDO Crude
Kpotential
Accurate
L2
![Page 59: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/59.jpg)
Details Details Details
1. Calculate distance from query to training images
2. Pick K nearest neighbors3. Run SVM4. Label !
If K neighbors are from the same class Done
KNN
SVM
![Page 60: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/60.jpg)
Details Details Details
1. Calculate distance from query to training images
2. Pick K nearest neighbors3. Run SVM4. Label !
Construct pairwise inner product matrix
Improvement – cache distance calculation
),()0,()0,(2
1, yxdydxdyx
KNN
SVM
![Page 61: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/61.jpg)
Details Details Details
1. Calculate distance from query to training images
2. Pick K nearest neighbors3. Run SVM4. Label !
Selected SVM: DAGSVM (faster)
Complexity:
)( 2KDO Accurate
KNN
SVM
1 vs 4
3 vs 4
2 vs 4 1 vs 3
2 vs 3 1 vs 2
![Page 62: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/62.jpg)
Complexity
1. Calculate distance from query to training images
2. Pick K nearest neighbors3. Run SVM4. Label !
Total complexity
DAGSVM training complexity
)( 2KDKDnDO AccuratepotentialAccuratecrude
)( 2nDO Accurate
KNN
SVM
![Page 63: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/63.jpg)
SVM KNN – continuum
Defining an SVM-KNN continuum:
K = 1 K = n (#images)
NN SVMSVMKNN
Biological motivation Human visual system
More than MAJ
![Page 64: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/64.jpg)
![Page 65: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/65.jpg)
![Page 66: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/66.jpg)
SVM KNN Summary
Similarity to prototypes
Combining Advantages from both methods NN – Fast, suitable for multiclass SVM – performs well with few samples and classes
Compatible with many types of distance functions
Biological motivation: Human visual system Discriminative process
![Page 67: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/67.jpg)
Outline
Motivation and Introduction
Classification Algorithms K-Nearest neighbors (KNN) SVM
Multiclass SVM
DAGSVM
SVM-KNN
Results - A taste of the distance Shape distance (shape context, tangent) Texture (texton histograms)
![Page 68: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/68.jpg)
Distance functions
ShapeTexture
D( , ( = ??
Training images and query
Query imageClass 1Class 2
Class 3
![Page 69: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/69.jpg)
Understanding the need - Shape
Well, which is it??
Capturing the shape Distance 1: Shape context Distance 2: Tangent distance
query
![Page 70: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/70.jpg)
Distance 1: Shape context
1. Find point correspondences2. Estimate transformation3. Distance
correspondence quality transformation quality
prototype query
Belongie et al., Shape matching and object recognition using shape contexts, IEEE Trans. (2002(
![Page 71: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/71.jpg)
Find correspondences
Detector - Use edge pointsDescriptor - Create “Landscape”
Relationship to other edge points Histogram of orientations and distances
Count = 5
Count = 6
prototype query
![Page 72: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/72.jpg)
Find correspondence
Detector - Use edge pointsDescriptor - Create “Landscape”
Relationship to other edge points Histogram of orientations and distances
Matching compare histograms ( )
prototype query
2
![Page 73: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/73.jpg)
Distance 1: Shape context
1. Find point correspondences2. Estimate transformation3. Distance
correspondence quality transformation (quality, magnitude)
prototype query
![Page 74: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/74.jpg)
MNIST – Digit DB
70,000 handwritten digitsEach image 28x28
![Page 75: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/75.jpg)
MNIST results
Human error rate – 0.2%Better methods exist < 1%
Error rate (%)
![Page 76: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/76.jpg)
Distance 2: Tangent distance
Distance includes invariance to small changes small rotations translations thickening
Simard et al., Transformation invariance in pattern recognition-tangent distance and tangent propagation . Neural Networks (1998(
Prototype query
![Page 77: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/77.jpg)
Space induced by rotation
α=0
α= -1α= -2
α=1
Pixel space
Dimension = 1
Rotation function
![Page 78: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/78.jpg)
Tangent distance – Visual intuition
Pixel space
Desired distance
P
Q
Prototype Image
Query Image
SP
SQ
But – calculating distance between non linear curves can be difficult
Solution:Use linear approximation
The Tangent
The Tangent
Euclidian distance (L2(
![Page 79: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/79.jpg)
Tangent Distance - General
For every image, create surface allowing transformations
Rotations Translations Thickness, etc.
Find a linear approximation - the tangent plane
Distance Calculate distance between linear planes
Has efficient solutions
7 dimensions
![Page 80: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/80.jpg)
9298 handwritten digits taken from mail envelopes
Each image 16x16
USPS – digit DB
![Page 81: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/81.jpg)
USPS results
Human error rate – 2.5%For L2 –
not optimal DAGSVM has similar results
For tangent NN similar results
PQ
![Page 82: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/82.jpg)
Understanding Texture
How to represent Texture??
Texture samples
![Page 83: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/83.jpg)
Filter responses
for pixel
P1
Texture representation
Represent using responses to a filter bank
0.6
-0.2
.
.
.
0.4
Filter bank – 48 filters
48 Filter responses
for pixel
0.1
0.8
.
.
.
0.3
P2
Filter responses
for pixel
-0.4
-0.7
.
.
.
0.17
P3
….
Texture patch
![Page 84: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/84.jpg)
P1P2
P3
Filter responses in
48-dimensional space
Introducing Textons
Filter responses – points in 48 dimensional spaceA texture patch – spatially repeating
Representation is redundant Select representative responses (K-means)
Textons !
Texture patch
T. Leung, J. Malik Representing and recognizing the visual appearance of materials using three-dimensional textons (2001(
Correspond to pixels
of one image
3.0
...
1.0
4.0
...
6.0
![Page 85: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/85.jpg)
Universal textons
Texton Filter responses in 48-dim space
T1 T2
T3T4
Prototype textures Filter bank
“Building blocks“ for all textures
![Page 86: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/86.jpg)
Filter bank
Distance 3: of Texton histograms
For a query texture1. Create filter responses2. Build texton histogram (using universal textons)
2
Filter responses in 48-dim space
T1 T2
T3T4
Query texture
T1 T2 T3 T4
Query Texton histogram
![Page 87: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/87.jpg)
Distance 3: of Texton histograms
For a query texture1. Create texton histogram2. Build texton histogram (using universal textons)3. Distance compare histograms ( )
2
Prototype Texton histogram
T
1
T
2
T
3
T
4
T
1
T
2
T
3
T
4
T
1
T
2
T
3
T
4
T
1
T
2
T
3
T
4
T
1
T
2
T
3
T
4
T
1
T
2
T
3
T
4
2
T1 T2 T3 T4
Query Texton histogram
Prototype texturesQuery texture
![Page 88: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/88.jpg)
CUReT – texture DB
61 texturesDifferent view pointsDifferent illuminations
![Page 89: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/89.jpg)
CUReT Results
(comparing texton histograms) T
1
T
2
T
3
T
4
![Page 90: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/90.jpg)
Caltech-101 DB
102 categories variations in color, pose, illumination
Distance function combination of texture and shape 2 algorithms Algo. A, Algo. B
Samples from the Caltech-101 DB
![Page 91: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/91.jpg)
Caltech-101 Results
(15 training images(
Still a long way to go…
Algo. B:Using only DAGSVM
(no KNN(
Correct rate(%)
66% correct
![Page 92: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/92.jpg)
Motivation – Human Visual System
Large Number of Categories (~30,000)
Discriminative Process
Small Set of Examples
Invariance to transformation
Similarity to Prototype instead of Features
![Page 93: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/93.jpg)
Summary
Popular methods NN SVM DAGSVM - extension to multi-class SVM
The hybrid method – SVM KNN Motivated by human perception (??) Improved complexity Better methods exist?
A taste of the distance Shape, Texture
Results classification method distance function
T
1
T
2
T
3
T
4PQ
Class 1
Class 2
1 vs 4
3 vs 4
2 vs 4 1 vs 3
2 vs 3 1 vs 2
![Page 94: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/94.jpg)
References
H. Zhang, A. C. Berg, M. Maire and J. Malik. SVM-KNN: Discriminative Nearest Neighbor Classification for Visual Category Recognition. IEEE, Vol. 2, pages 2126-2136, 2006.
P. Vincent and Y. Bengio. K-local hyperplane and convex distance nearest neighbor algorithms. NIPS, pages 985-992, 2001.
J. C. Platt, N. Cristianini, and J. Shawe-Taylor. Large margin DAGs for multiclass classification. NIPS, pages 547-553, 1999.
C. Hsu and C. Lin. A comparison of methods for multiclass support vector machines. IEEE, Vol. 13, pages 415-425, 2002.
T. Leung and J. Malik. Representing and recognizing the visual appearance of materials using three-dimensional textons. Int. J. Computation Vision, 43(1):29-44, 2001.
P. Simard, Y. LeCun, J. S. Denker, and B. Victorri. Transformation invariance in pattern recognition-tangent distance and tangent propagation. Neural Networks: Tricks of the Trade, pages 239-274, 1998.
S. Belongie, J. Malik, and J. Puzicha. Shape matching and object recognition using shape contexts. IEEE, Vol. 24, pages 509-522, 2002.
![Page 95: Multiclass SVM and Applications in Object Classification Yuval Kaminka, Einat Granot Advanced Topics in Computer Vision Seminar Faculty of Mathematics.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649c785503460f9492dc91/html5/thumbnails/95.jpg)
Thank You!