Face Recognition and Feature Subspaces
description
Transcript of Face Recognition and Feature Subspaces
![Page 1: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/1.jpg)
Face Recognition and Feature Subspaces
Computer Vision (CS 543 / ECE 549)University of Illinois
Derek Hoiem
03/29/12
Some slides from Lana Lazebnik, Silvio Savarese, Fei-Fei Li
Chuck Close, self portraitLucas by Chuck Close
![Page 2: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/2.jpg)
This class: face recognition
• Two methods: “Eigenfaces” and “Fisherfaces”• Feature subspaces: PCA and FLD
• Look at results from recent vendor test
• Look at interesting findings about human face recognition
![Page 3: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/3.jpg)
Applications of Face Recognition• Surveillance
![Page 4: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/4.jpg)
Applications of Face Recognition• Album organization: iPhoto 2009
http://www.apple.com/ilife/iphoto/
![Page 5: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/5.jpg)
• Can be trained to recognize pets!
http://www.maclife.com/article/news/iphotos_faces_recognizes_cats
![Page 6: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/6.jpg)
Facebook friend-tagging with auto-suggest
![Page 7: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/7.jpg)
Face recognition: once you’ve detected and cropped a face, try to recognize it
Detection Recognition “Sally”
![Page 8: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/8.jpg)
Face recognition: overview
• Typical scenario: few examples per face, identify or verify test example
• What’s hard: changes in expression, lighting, age, occlusion, viewpoint
• Basic approaches (all nearest neighbor)1. Project into a new subspace (or kernel space)
(e.g., “Eigenfaces”=PCA)2. Measure face features3. Make 3d face model, compare
shape+appearance (e.g., AAM)
![Page 9: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/9.jpg)
Typical face recognition scenarios
• Verification: a person is claiming a particular identity; verify whether that is true– E.g., security
• Closed-world identification: assign a face to one person from among a known set
• General identification: assign a face to a known person or to “unknown”
![Page 10: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/10.jpg)
What makes face recognition hard?
Expression
![Page 11: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/11.jpg)
What makes face recognition hard?
Lighting
![Page 12: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/12.jpg)
What makes face recognition hard?
Occlusion
![Page 13: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/13.jpg)
What makes face recognition hard?
Viewpoint
![Page 14: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/14.jpg)
Simple idea for face recognition1. Treat face image as a vector of intensities
2. Recognize face by nearest neighbor in database
x
nyy ...1
xy kk
k argmin
![Page 15: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/15.jpg)
The space of all face images• When viewed as vectors of pixel values, face images are
extremely high-dimensional– 100x100 image = 10,000 dimensions– Slow and lots of storage
• But very few 10,000-dimensional vectors are valid face images
• We want to effectively model the subspace of face images
![Page 16: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/16.jpg)
The space of all face images• Eigenface idea: construct a low-dimensional linear
subspace that best explains the variation in the set of face images
![Page 17: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/17.jpg)
Principal Component Analysis (PCA)• Given: N data points x1, … ,xN in Rd
• We want to find a new set of features that are linear combinations of original ones: u(xi) = uT(xi – µ)
(µ: mean of data points)
• Choose unit vector u in Rd that captures the most data variance
Forsyth & Ponce, Sec. 22.3.1, 22.3.2
![Page 18: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/18.jpg)
Principal Component Analysis• Direction that maximizes the variance of the projected data:
Projection of data point
Covariance matrix of data
The direction that maximizes the variance is the eigenvector associated with the largest eigenvalue of Σ
N
N
1/N
Maximizesubject to ||u||=1
![Page 19: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/19.jpg)
Implementation issue
• Covariance matrix is huge (M2 for M pixels)
• But typically # examples << M
• Simple trick– X is MxN matrix of normalized training data– Solve for eigenvectors u of XTX instead of XXT – Then Xu is eigenvector of covariance XXT
– Need to normalize each vector of Xu into unit length
![Page 20: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/20.jpg)
Eigenfaces (PCA on face images)1. Compute the principal components (“eigenfaces”)
of the covariance matrix
2. Keep K eigenvectors with largest eigenvalues
3. Represent all face images in the dataset as linear combinations of eigenfaces
– Perform nearest neighbor on these coefficients
M. Turk and A. Pentland, Face Recognition using Eigenfaces, CVPR 1991
𝑽=𝑽 (: ,largest eig)
𝑿 𝒑𝒄𝒂=𝑽 (: ,largest eig )𝑻 𝑿
![Page 21: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/21.jpg)
Eigenfaces example• Training images• x1,…,xN
![Page 22: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/22.jpg)
Eigenfaces exampleTop eigenvectors: u1,…uk
Mean: μ
![Page 23: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/23.jpg)
Visualization of eigenfacesPrincipal component (eigenvector) uk
μ + 3σkuk
μ – 3σkuk
![Page 24: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/24.jpg)
Representation and reconstruction• Face x in “face space” coordinates:
=
![Page 25: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/25.jpg)
Representation and reconstruction• Face x in “face space” coordinates:
• Reconstruction:
= +
µ + w1u1+w2u2+w3u3+w4u4+ …
=
^x =
![Page 26: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/26.jpg)
P = 4
P = 200
P = 400
Reconstruction
After computing eigenfaces using 400 face images from ORL face database
![Page 27: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/27.jpg)
Eigenvalues (variance along eigenvectors)
![Page 28: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/28.jpg)
NotePreserving variance (minimizing MSE) does not necessarily lead to qualitatively good reconstruction.
P = 200
![Page 29: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/29.jpg)
Recognition with eigenfacesProcess labeled training images• Find mean µ and covariance matrix Σ• Find k principal components (eigenvectors of Σ) u1,…uk
• Project each training image xi onto subspace spanned by principal components:(wi1,…,wik) = (u1
T(xi – µ), … , ukT(xi – µ))
Given novel image x• Project onto subspace:
(w1,…,wk) = (u1T(x – µ), … , uk
T(x – µ))• Optional: check reconstruction error x – x to determine
whether image is really a face• Classify as closest training face in k-dimensional subspace
^
M. Turk and A. Pentland, Face Recognition using Eigenfaces, CVPR 1991
![Page 30: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/30.jpg)
PCA
• General dimensionality reduction technique
• Preserves most of variance with a much more compact representation– Lower storage requirements (eigenvectors + a few
numbers per face)– Faster matching
• What are the problems for face recognition?
![Page 31: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/31.jpg)
Limitations
Global appearance method: not robust to misalignment, background variation
![Page 32: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/32.jpg)
Limitations• The direction of maximum variance is not always
good for classification
![Page 33: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/33.jpg)
A more discriminative subspace: FLD• Fisher Linear Discriminants “Fisher Faces”
• PCA preserves maximum variance
• FLD preserves discrimination– Find projection that maximizes scatter between
classes and minimizes scatter within classes
Reference: Eigenfaces vs. Fisherfaces, Belheumer et al., PAMI 1997
![Page 34: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/34.jpg)
Comparing with PCA
![Page 35: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/35.jpg)
Variables
• N Sample images: • c classes:
• Average of each class: • Average of all data:
Nxx ,,1
c ,,1
ikx
ki
i xN
1
N
kkxN 1
1
![Page 36: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/36.jpg)
Scatter Matrices
• Scatter of class i: Tikx
iki xxSik
c
iiW SS
1
c
i
TiiiB NS
1
• Within class scatter:
• Between class scatter:
![Page 37: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/37.jpg)
Illustration
2S
1S
BS
21 SSSW
x1
x2Within class scatter
Between class scatter
![Page 38: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/38.jpg)
Mathematical Formulation• After projection
– Between class scatter– Within class scatter
• Objective
• Solution: Generalized Eigenvectors
• Rank of Wopt is limited
– Rank(SB) <= |C|-1
– Rank(SW) <= N-C
kT
k xWy
WSWS BT
B ~
WSWS WT
W ~
WSW
WSW
SS
WW
TB
T
W
Bopt WW
max arg~~
max arg
miwSwS iWiiB ,,1
![Page 39: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/39.jpg)
Illustration
2S
1S
BS
21 SSSW
x1
x2
![Page 40: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/40.jpg)
Recognition with FLD• Use PCA to reduce dimensions to N-C
• Compute within-class and between-class scatter matrices for PCA coefficients
• Solve generalized eigenvector problem
• Project to FLD subspace (c-1 dimensions)
• Classify by nearest neighbor
WSW
WSWW
WT
BT
fld Wmax arg miwSwS iWiiB ,,1
Tikx
iki xxSik
c
iiW SS
1
c
i
TiiiB NS
1
xWx Toptˆ
)pca( XWpca
𝑊 𝑇𝑜𝑝𝑡=𝑊𝑇 𝑓𝑙𝑑𝑊𝑇𝑝𝑐𝑎Note: x in step 2 refers to PCA coef; x in step 4 refers to original data
![Page 41: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/41.jpg)
Results: Eigenface vs. Fisherface
• Variation in Facial Expression, Eyewear, and Lighting
• Input:160 images of 16 people• Train:159 images• Test: 1 image
With glasses
Without glasses
3 Lighting conditions 5
expressions
Reference: Eigenfaces vs. Fisherfaces, Belheumer et al., PAMI 1997
![Page 42: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/42.jpg)
Eigenfaces vs. Fisherfaces
Reference: Eigenfaces vs. Fisherfaces, Belheumer et al., PAMI 1997
![Page 43: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/43.jpg)
Large scale comparison of methods• FRVT 2006 Report• Not much (or any) information available about
methods, but gives idea of what is doable
![Page 44: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/44.jpg)
FVRT Challenge• Frontal faces
– FVRT2006 evaluation
False Rejection Rate at False Acceptance Rate = 0.001
![Page 45: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/45.jpg)
FVRT Challenge• Frontal faces
– FVRT2006 evaluation: controlled illumination
![Page 46: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/46.jpg)
FVRT Challenge• Frontal faces
– FVRT2006 evaluation: uncontrolled illumination
![Page 47: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/47.jpg)
FVRT Challenge• Frontal faces
– FVRT2006 evaluation: computers win!
![Page 48: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/48.jpg)
Face recognition by humans
Face recognition by humans: 20 results (2005)
Slides by Jianchao Yang
![Page 49: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/49.jpg)
![Page 50: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/50.jpg)
![Page 51: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/51.jpg)
![Page 52: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/52.jpg)
![Page 53: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/53.jpg)
![Page 54: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/54.jpg)
![Page 55: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/55.jpg)
![Page 56: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/56.jpg)
![Page 57: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/57.jpg)
![Page 58: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/58.jpg)
Things to remember
• PCA is a generally useful dimensionality reduction technique– But not ideal for discrimination
• FLD better for discrimination, though only ideal under Gaussian data assumptions
• Computer face recognition works very well under controlled environments – still room for improvement in general conditions
![Page 59: Face Recognition and Feature Subspaces](https://reader035.fdocuments.in/reader035/viewer/2022081503/56815bd4550346895dc9c507/html5/thumbnails/59.jpg)
Next class
• Image categorization: features and classifiers– Ruiqi is teaching on Tues