ENTREPRENEURIAL ORIENTATION, MARKET ORIENTATION, LEARNING ...
Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke...
Transcript of Statistical Learning for Image Orientation · Statistical Learning for Image Orientation Luke...
Statistical Learning for Image Orientation
Luke Barrington
Mentors: Nuno Vasconcelos, UCSD ECE Dept.Babak Jafarian, Cal-(IT)2
Agenda
Project OverviewSVM BackgroundParameter SelectionFirst ExperimentFeature Selection MethodsFuture Work / Wrap Up / Questions
Project OverviewDevelop an image classification system for MMS-enabled cell phonesMotivation is for use by spectators at Athens 2004 Olympic gamesUse modern classification technologies to improve server-side organization and provide useful user output
Goal for my project: image orientation
Sports Photographers
Image OrientationCamera phones, scanners, digital cameras, online image databases, …Device output has standard dimension
At least 50% will be incorrect
Bulk, automated, corrective processing before display to human users
System Block Diagram
MMS/Internet server
.
.
.
.
.
.
UnorderedDirectory
OrderedOutputUser Input
Classifier
Classifier Block DiagramUser’s Parameters
+1-1…+1
KernelErrorWeights
TRAIN ModelParameters CLASSIFY
+1
-1
Training DataTraining Data
Data Labels
Class Label
Unlabeled Data
Agenda
Project OverviewSVM BackgroundParameter SelectionFirst ExperimentFeature Selection MethodsFuture Work / Wrap Up / Questions
Decision Boundary
Margin
Training = Learn Decision Boundary
-1 training examples
+1training examples
Support Vectors
Decision Boundary
Classification
Support Vectors
-1
+1
B
Test B = +1
A
Test A = -1
Extension to Multi-Class Problems
SVM is a binary decision classifierOne solution is a decision tree format
n-class problem requires ½n(n-1) classifiers
A B C D
A/B C/D
A/C A/D B/C B/D
Multi-Class: 1-vs-All ClassifierAlternative is a 1-vs-all classifierTrain each class with all training dataLabel: +1 (in the class) or
–1 (in any other class)Requires n classifiers for an n-class problemOutput may be undetermined (all classifiers return –1) Class +1 -1
a a {b, …, n}b b {a, c, …, n}
n n {a, …, n-1}M MM
Linear Support Vector Machine for separable dataAssume the training points can be divided by a linear hyperplane:
xi●w + b = 0 w is the normal vector
is perpendicular distance to origin
With appropriate scaling, the training data satisfy:xi●w + b ≥ +1 for yi = +1xi●w + b ≤ -1 for yi = -1
Combine this into a single condition: yi(xi●w + b) – 1 ≥ 0 for all i
The width of margin is: 2/||w||Robust classifier has widest margin => minimize ||w||2
wb−
Linear SVM, separable data contd.
Primal Lagrangian formulation:
Setting the gradient of LP w.r.t. w & b = 0 gives the conditions:
Combining these equality constraints in the dual formulation gives:
SVM training now amounts to maximizing LD with respect to the weights αi, subject to the constraints above and αi ≥ 0
Support vectors are all the training points where αi > 0
For classifier’s purposes, all other points can be ignored
∑∑ ⋅−== ji
jijiji
l
iiD xxyyL
,1
ααα
( ) ∑∑==
++⋅−=l
ii
l
iiiiP bwxywL
11
2
21 αα
011
== ∑∑==
l
iii
l
iiii yandxyw αα
Linear SVM for unseparable dataWe modify the equations from linear and separable case by the addition of a “slack” variable ξ. Now the equations become
xi●w + b ≥ +1 – ξi for yi = +1xi●w + b ≤ -1 + ξi for yi = -1 ξi ≥ 0if ξi > 1 => an error occurred Σiξi is an upper-bound on the training error
Now we want to minimize:
C is a user-specified cost parameter
This is again a constrained optimization problem where we want to maximize:
Subject to:
The data which satisfy this maximization are the Support Vectors
∑∑ ⋅−== ji
jijiji
l
iiD xxyyL
,1ααα
k
iiC
w⎟⎠
⎞⎜⎝
⎛− ∑ξ
2
2
001
=≤≤ ∑=
l
iiii yandC αα
Non-linear SVMsGeneralize to non-linear decision function for the classifierMap the data to some other (higher-dimension) Euclidean space H:
Now the problem depends only on
Find a kernel function such that
Use this (non-infinite) result to get classifier output
HRd →Φ :
)()( ji xx Φ⋅Φ
)()(),( jiji xxxxK Φ⋅Φ=
( ) ∑∑==
+=+Φ⋅Φ=SS N
iiii
N
iiii bxsKybxsyxf
11),()()( αα
Agenda
Project OverviewSVM BackgroundParameter SelectionFirst ExperimentFeature Selection MethodsFuture Work / Wrap Up / Questions
SVM ParametersKernel:
Linear (dot product) K(xi,xj) = (xi.xj)Polynomial: K(xi,xj) = (xi.xj + 1)p
RBF: K(xi,xj) = e-|| xi-xj ||²/2σ²
Sigmoid neural network: K(xi,xj) = tanh(κxi.xj -δ)
C (Error penalty) :Sets an upper bound on the alpha coefficients Higher C gives higher weight to outliersLow C assumes many data points are outliersAssign different C for each class based on number training data points
Parameter Selection“Research” = correct choice of parametersRadial Basis Function generalizes bestParameter space is 2-DFind optimum set using grid search
σ
C
Agenda
Project OverviewSVM BackgroundParameter SelectionFirst ExperimentFeature Selection MethodsFuture Work / Wrap Up / Questions
Sample Problem: Car Orientation
= +1
= -1
Feature Space
Grayscale pixel values (0 to 255)No input scalingDown-sample image to 128x128 and row scan
Input data is 16384-dimensional vector
Data SetFrom MIT labs516 imagesTraining set: 400Validation set: 50RBF kernel, σ = 4 C = 25
Car Orientation ResultsClassifiers for {0º, 30º, 60º, 90º, 120º, 150º, 180º}
In-class detection accuracy = 97.5 ± 2%False Positive < 2%
Initial ResultsSVM solves this problem easily on a limited domainTraining time can be significant (large feature size)Classification is almost instantaneous
Agenda
Project OverviewSVM BackgroundParameter SelectionFirst ExperimentFeature Selection MethodsFuture Work / Wrap Up / Questions
Expansion of Training SetEfficacy of classifier depends on training data
Corel Database
Sports Images
Crawl web sites
Use subclasses to choose feature spaceTrain “final” classifier on everything
Feature SpaceGrayscale pixel valuesSquare images
Each image provides 4 training examples
Down-sample image to 112x112Input data is 12544 dimensional vector
= +1 = -1
ResultsBest Corel class
70% detection rate, 3% false positiveWorst Corel class
0% detection rate, 30% false positiveSports images
63% detection rate, 14% false positive
Still only working on one specific class at a time
New Feature SpacePixel data is over-sized and under-poweredNeed to capture edge information in more succinct and compact formatEdge orientation histogram
Eigenvalues of gradient matrix
Gradient Matrix
comes from edge detection
⎟⎟⎠
⎞⎜⎜⎝
⎛=0k
⎟⎟⎠
⎞⎜⎜⎝
⎛=
k0
for vertical edges for horizontal edges
•Construct a gradient field on N x N image sub-blocks
•Sub-block size depends on image size
•Eigenvectors of H are features
•Use N = 16, 8, 4, 2, 1 to capture both local and global features
•Feature size = 2(162 + 82 + 42 + 22 + 1) = 682
Using Color DataPixel intensity alone requires 12544 dimensional input spaceAdding color => increase by 200%HSV space mean & varianceat image borders (Wang & Zhang 2003)
Current WorkApply new feature space to experiments reviewedExplore color feature representationsTrain general system across all classesOptimize system to work with camera phone outputDevelop web site / application for digital photos
Add external modules (e.g. face detection, scene classification) to improve performance
Agenda
Project OverviewSVM BackgroundParameter SelectionFirst ExperimentFeature Selection MethodsFuture Work / Wrap Up / Questions