1 M. Bronstein | Expression-invariant representation of faces and its applications for face...

1M. Bronstein | Expression-invariant representation of faces and its applications for face recognition

Expression-invariant representation of facesand its applications for 3D face recognition

Michael M. Bronstein

Department of Computer ScienceTechnion – Israel Institute of Technology

M.Sc. Seminar 2 November 2004


Biometrics in the age of patriarchs

And Jacob went near unto Isaac

his father; and he felt him, and

said,

‘The voice is Jacob’s voice, but

the hands are the hands of Esau’.

And he recognized him not,

because his hands were hairy, as

his brother Esau’s hands.

Genesis XXVII:22


Biometrics today

IRIS

FINGERPRINT

PALM

VOICE

RETINA

DNASIGNATURE

FACE


ILLUMINATION+ EXPRESSION

ILLUMINATION+ POSE

ILLUMINATION+ EXPRESSION

+ POSE

A. Bronstein, M. Bronstein and R. Kimmel, “Expression-invariant 3D face recognition”, chapter in Face processing: advanced modeling and methods, Chellapa and Zhao (Eds.) to appear

Problems of 2D face recognition


Approaches in 2D face recognition

Few feature points can be reliably extracted

Such features are usually insufficient for good recognition

Appropriate model is a problem

No good model for facial expressions

INVARIANT REPRESENTATION GENERATIVE APPROACH


Intrinsic problem of 2D face recognition



2D vs 3D in face recognition

Simple acquisition – legacy hardware and databases

Sensitive to everything (lighting, pose, makeup, expressions)

Insensitive to lighting, pose, make up

Requires special hardware

Sensitive to expressions

Sensitive to aging and plastic surgery (?)


Face is a smooth, connected, compact 2D Riemannian manifold

Parametrization

Metric tensor

Riemannian geometry basics

2 1 1 2 3 1 2 3: , ,..., ,x x x R R

Geodesic distances

ij i jg x x

infCd C

x1

2

S

S

1x

3x

2x

x 1 x

2 x


ISOMETRIC NON-ISOMETRIC

Isometry is a transformation that preserves geodesic distances

Isometric model


Isometric model: Facial expressions = isometries of some initial facial surface (neutral expression)


The open mouth problem

WITHOUTH TOPOLOGICAL CONSTRIANT

Open mouth is not an isometry

Isometric model is true for expressions with closed or open mouth

Extension: topologically-constrained isometric model

WITH TOPOLOGICAL CONSTRIANT

M. Bronstein, A. Bronstein and R. Kimmel, “Expression-invariant representation for human faces”, submitted


RIEMANNIAN EUCLIDEAN

Geodesic distances are an invariant description of the surface…

…but are inconvenient to work with

A. Elad and R. Kimmel, CVPR 2001A. Elad and R. Kimmel, IEEE Trans. PAMI, 2003

[Elad & Kimmel 2001]: embed isometric surfaces into a low-dimensional Euclidean space and treat them as rigid objects

Canonical forms

i

jix

jx

,ij i j 2ij i jd x x


- (m > 2)-dimensional manifold (embedding space)

Isometric embedding

1 1: ,..., , ,..., ,Nm

Nx x D SS

, 1,..., .ij ij id j N

mS

- NN matrix of original geodesic distances ij

- NN matrix of distances in the embedding space ijdD

A mapping between finite metric spaces

such that


Embedding problem in cartography

A. Bronstein, M. Bronstein and R. Kimmel, “Three-dimensional face recognition”, submitted to IJCV

The globe cannot be embedded into a plane according to Gauss Theorema Egregium.

GLOBE (HEMISPHERE) PLANAR MAP


Does an isometric embedding always exist ?

N. Linial

Example of 4 points on a sphere that cannot be isometrically embedded into an Euclidean space of ay finite dimension.

4 POINTS ON A SPHERE A NEAR-ISOMETRIC EMBEDDING


- Nm matrix of canonical form parametric coordinates

Embedding error

2

rawi

ijijj

d

X X

2

norm 2

iji j

ij

ij

ji

d

d

X

XX

1,..., NX x x

RAW STRESS:

NORMALIZED STRESS:

I. Borg and P. Grönen, Modern multidimensional scaling, Springer, 1997


Multidimensional scaling

Nm optimization variables


min X X

Multidimensional scaling (MDS) problem:

Optimum defined up to an isometry in mS

Non-convex optimization problem

Optimum = canonical form


Euclidean embedding – LS MDS

where

1 if

1 if ij

i ju

N i j

1 if and 0

0 if and 0

if ij

ij

ij iij j

j i

ij

d d

d

i j

b i j

b i j

X

Choose 3m RS


2 2raw

T T

2

trace 2trace

ij ii j i j i j

ji j

A

j id d

A

X X

XUX XB

X

X X

raw 2 2 0 X UX X X X

For require rawmin X X

Use


Alignment

Eliminate first-order moments

Reorder the axes

Eliminate mixed second-order moments

Force the sign

100 010 001 0

200 110 101 200

110 020 011 020 200 020 002

101 011 002 002

;

1

sign 0, 1,...,3N

i

ki k

where 1 2 2

1

N p q

i i i

r

pqri

x x x


Canonical forms of faces (closed mouth)

FACIAL SURFACES

CANONICAL FORMS



FACIAL SURFACES

CANONICAL FORMS

Canonical forms of faces (open mouth)



Spherical embedding

A. Bronstein, M. Bronstein and R. Kimmel, “On isometric embedding of facial surfaces into S 3”, subm. ScaleSpace

Choose as a sphere immersed into

3 4 : 1 x xS R

Geodesic distances (arcs of great circles)

31, cos ,i j i jd x x x x

S

3S 4RmS

ix

jx

Minimize the normalized stress w.r.t. the parametric coordinates

2

2normi j

i

ijj

jj

i

i

d

d

X

X

Alignment performed using Euclidean moments in 4R


Spherical embedding example

Maximum-variance projection onto R3 of a facial surface embedded into S3 with different radii

R = 5 cm


R = 7.5 cm R = 15 cm


Spherical embedding vs Euclidean embedding


SPHERE RADIUS R [cm]

EM

BE

DD

ING

ER

RO

R


3DFACE system – hardware

PROJECTOR

CAMERA

MONITOR

CARDMOUNTING


3DFACE system – user interface


3DFACE system – pipeline


Isometric model validation

133 toothpaste markers placed on the face as invariant points

Track how geodesic / Euclidean distances change due to expressions

Lips cropped

16 different expressions



-60 -40 -20 0 20 40 600

0.2

0.4

0.6

0.8

1

-150 -100 -50 0 50 100 1500

0.2

0.4

0.6

0.8

1

Isometric model validation (cont.)

ABSOLUTE CHANGE mm RELATIVE CHANGE %

EUCLIDEANGEODESIC



Sensitivity to facial expressions

A. Bronstein, M. Bronstein and R. Kimmel, “Three-dimensional face recognition”, submitted to IJCVM. Bronstein, A. Bronstein and R. Kimmel, “Expression-invariant representation for human faces”, submitted


Sensitivity to facial expressions (closed mouth)

Visualization of the distances between faces in the facial expressions experiment. Each point represents a subject in the database.

-1000 -800 -600 -400 -200 0 200 400 600 800 1000-1000

-800

-600

-400

-200

0

200

400

600

800

1000

-1000 -800 -600 -400 -200 0 200 400 600 800 1000-1000

-800

-600

-400

-200

0

200

400

600

800

1000

RIGID SURFACES CANONICAL FORMS



Sensitivity to facial expressions (open mouth)

Visualization of the distances between faces in the facial expressions experiment. Each point represents a subject in the database.

RIGID SURFACES TOPOL. CONSTR. CANONICAL FORMS-1000 -800 -600 -400 -200 0 200 400 600

-100

-50

0

50

100

150

-1000 -500 0 500 1000

-400

-300

-200

-100

0

100

200

300

400



Benchmarks


Benchmark of face recognition algorithms on a database containing 220 instances of 30 faces with extreme facial expressions.

RIGID

CANONICAL

EIGENFACES

FALSE ACCEPTANCE RATE %

FA

LS

E R

EJE

CT

ION

RA

TE

%

RECOGNITION RANKR

EC

OG

NIT

ION

AC

CU

RA

CY

%


Recognition example


PROBE EIGENFACES RIGID SURFACE CANONICAL FORM

MORAN 129 ORI 188 SUSY 276 MORAN 114

MICHAEL 17 ALEX 40 ALEX 39 MICHAEL 2


Identical twins – find the differences

The difference between Michael and Alex obtained by comparing the canonical forms reveals a slight difference in the geometry of their nose.

MICHAEL ALEXDIFFERENCE MAP


Bibliography

A. Elad and R. Kimmel, On bending invariant signatures for surfaces, Trans. IEEE PAMI 25(10): 1285-1295, 2003

A. Bronstein, M. Bronstein and R. Kimmel, Expression-invariant 3D face recognition, Proc. AVBPA 2003, LNCS 2688, 62-69, Springer

A. Bronstein, M. Bronstein, R. Kimmel and A. Spira, Face recognition from facial surface metric, Proc. ECCV 2004, 225-237

A. Bronstein, M. Bronstein, R. Kimmel and E. Gordon, Fusion of 2D and 3D information in 3D face recognition, Proc. ICIP 2004

M. Bronstein, A. Bronstein and R. Kimmel, Three-dimensional face recognition, CIS Tech. Report 04, 2004, submitted to IJCV

M. Bronstein, A. Bronstein and R. Kimmel, Expression-invariant representation of faces, submitted to PNAS

A. Bronstein, M. Bronstein and R. Kimmel, On isometric embedding of facial surfaces into S3, submitted to ScaleSpace


Isometric model for facial expressions

NEUTRAL AVERAGE MIN. MAX

The uncertainty region around the face in the presence of facial expressions is so large that many other faces can fit in.



Parametrization of S3

1 1 2 3

2 1 2 3

3 1 3

4 3

cos cos cos ;

cos sin cos ;

sin cos ;

sin ;

0, 0,2 0,

x

x

x

x


3 4 : 1 x xS R

Geodesic distances

31

1 1 3 1 3 2 2

3 3 1 1 1 3

, cos ,

cos [cos cos cos cos cos

cos cos sin sin sin sin ]

i j i j

i i j j i j

i j i j i j

d

x xS

1 M. Bronstein | Expression-invariant representation of faces and its applications for face...

Documents

Transcript of 1 M. Bronstein | Expression-invariant representation of faces and its applications for face...