Learning the Appearance of Faces:A Unifying Approach for the Analysis and Synthesis of Images.

Thomas VetterThomas Vetter

Germany GermanyUniversity of FreiburgUniversity of Freiburg

http://graphics.informatik.uni-freiburg.dehttp://graphics.informatik.uni-freiburg.de

Computer Vision & Computer Graphics

Computer Graphics can help to solve Computer Vision!

| G (p) - I |2 = min Parameters

G ( image ) Parameters-1

Vision ( image ) parameters

image Graphics ( parameters )

Analysis by Synthesis

3DWorld Image

Analysis

Synthesis

Image Model

Image Description

model parameter

Synthesis of Faces

Input ImageInput Image

Modeler

ResultResult

Database

FaceAnalyzer

3D Head3D Head

Morphable

Face Model

Approach: Example based modeling of faces

2D Image 3D Face Models2D Image 3D Face Models

= w1 * + w2 * + w3 * + w4 * +. . .

Cylindrical Coordinates

red(h,)green(h,)

blue(h,)

red(h,)green(h,)

blue(h,)

h

radius(h,)radius(h,)

h

Morphing 3D Faces

3D Blend

3D Morph

1__2

1__2

+ =

Correspondence: A two step process!

Correspondence between

1. two examples ( Optical Flow like algorithms).

2. many examples ( Morphable Model )

Reference

Example

2nd Example

=

a1 * + a2 * + a3 * + a4 * +. . .

b1 * + b2 * + b3 * + b4 * +. . .

Vector space of 3D faces.

A Morphable Model can generate new faces.

Manipulation of Faces

Modeler

Modelling in Face Space

Caricatur

OriginalOriginal

AverageAverage

Modelling the Appearance of Faces

A face is represented as a point in face space.Which directions code for specific attributes ?

Learning from Labeled Example Faces

Fitting a (linear) regression function

Facial Attributes

WeightWeightWeightWeight

OriginalOriginalOriginalOriginal

Subjective Subjective AttractivenessAttractiveness

Subjective Subjective AttractivenessAttractiveness

Transfer of Facial Expressions

= Smile= Smile--

Originals:Originals:

+ Smile =+ Smile =Novel Face:Novel Face:

Facial Expressions

OriginalOriginalOriginalOriginal

3D Shape from Images

FaceAnalyzer

3D Head3D HeadInput ImageInput Image

Matching a Morphable 3D-Face-Model

= R

Optimization problem!Optimization problem!Optimization problem!Optimization problem!

a1 * + a2 * + a3 * + a4 * +. .

b1 * + b2 * + b3 * + b4 * +. .

Error Function

• Image difference

• Plausible parameters

• Minimize

priorimage EEE priorimage EEE

2

,inputmodelImage ),(),(

yx

yxyxE II 2

,inputmodelImage ),(),(

yx

yxyxE II

),...),(log(prior ii bapE ),...),(log(prior ii bapE

Optimization Strategies

• Stochastic Gradient Decent

new old

old

i i ci

dEc c i

dc

new old

old

i i ci

dEc c i

dc

I c A I c A

• Difference Decomposition • Difference Decomposition

Future Challenges

• Which Object Classes are linear ?

• How to built them automatically?