Face Recognition from Face Motion Face Recognition from Face Motion Manifolds using Robust Kernel RADManifolds using Robust Kernel RAD

Ognjen Arandjelović

Roberto Cipolla

Funded by Toshiba Corp. and Trinity College, Cambridge

Face RecognitionFace Recognition

• Single-shot recognition – a popular area of research since 1970s

• Many methods have been developed

• Bad performance in presence of:

– Illumination variation

– Pose variation

– Facial expression

– Occlusions (glasses, hair etc.)

Eigenfaces

Wavelet methods3D MorphableModels

Face Recognition from VideoFace Recognition from Video

• Face motion helps resolve ambiguities of single shot recognition – implicit 3D

• Video information often available (surveillance, authentication etc.)

Recognition setup Training stream Novel stream

Face ManifoldsFace Manifolds• Face patterns describe manifolds which are:

– Highly nonlinear, and

– Noisy, but

– Smooth

Facial features Face pattern manifold Face region

Limitations of Previous WorkLimitations of Previous Work

• In this work we address 3 fundamental questions:

– How to model nonlinear manifolds of face motion

– How to choose the distance measure

– How and what noise sources to model

?

Comparing Nonlinear ManifoldsComparing Nonlinear Manifolds

Closest-neighbour

Majority vote + Eigen/Fisherfaces

Mutual Subspace Method

Principal angles

Our method, KLD method of Shakhnarovich et al.

Information-theoretic measures

RAD: How well can we distinguish between P(x) and Q(x)?

1 1 1( , ) ( ( || ) ( || ) )RAD KL KLD P Q D P Q D Q P

KLD vs. RAD vs. …KLD vs. RAD vs. …

KLD: How well does P(x) explain Q(x)?

P(x) Q(x)

( || )KLD P Q

Q(x)

P(x)

Nonlinear RADNonlinear RAD

• Use closed form expression for KLD between Gaussians in KPCA space

Kernel PCA

Approximately linear manifolds

Highly nonlinear manifolds

Input space KPCA space

1 12|| log

Tqq p q pKL p q q

p

D p q Tr N

ΣΣ Σ Σ x x x x

Σ

0.6 ,( , ) ( , ) i j

i j i jk e x xx x x x

RBF Kernel

RegistrationRegistration

• Linear operations on images are highly nonlinear in the pattern space

• Translation/rotation and weak perspective can be easily corrected for directly from point correspondences

– We use the locations of pupils and nostrils to robustly estimate the optimal affine registration parameters

Translationmanifold

Skewmanifold

Rotationmanifold

Registration Method UsedRegistration Method Used

• Feature localization based on the combination of shape and pattern matching (Fukui et al. 1998)

Detect features

Crop & affineregister faces

Feature Tracking ErrorsFeature Tracking Errors

• We recognize two sources of registration noise:

– Low-energy noise due to the imprecision feature detector

– High-energy noise due to incorrectly localized features

20 automatically cropped and registered faces from a video sequence

Outliers – highenergy noise

Imperfect alignment offacial features – low energy noise

Low Energy NoiseLow Energy Noise

• Estimate misregistration manifold noise energy

• Augment data with synthetically perturbed samples = thickening of the motion manifold

• Synthetic data explicitly models the variation

Original data Original + synthetic data

Outliers – High Energy NoiseOutliers – High Energy Noise

• Outliers are due to incorrect feature localization

• High energy noise – far from the ‘correct’ data mean in KPCA space

Outliers

Manifold of correctly registered faces (+low energy noise)

RANSAC for Robust KPCARANSAC for Robust KPCA

Minimal, randomsample

Kernel PCAprojection

Valid datacount

IterationOutliers

Algorithm: The Big PictureAlgorithm: The Big Picture

Input frames

Distance

Original + syntheticdata

Valid data inKPCA space

Face Video DatabaseFace Video Database

• No standard database exists – we collected our own data

• 160 people, 10 different lighting conditions (each condition twice i.e. 20 video sequences per person)

Evaluation ResultsEvaluation Results

• Robust Kernel RAD outperformed other methods on all databases

• Average recognition rate of 98%

Method Limitations / Future WorkMethod Limitations / Future Work

• Less pose sensitivity (why should input and reference distributions be similar?)

• Illumination invariance is not addressed

Same person,different illumination

Novel personSee Arandjelović et al., BMVC 2004

For suggestions, questions etc. please contact me at: oa214@cam.ac.uk