3D Face Modeling

Michaël De Smet

Topics to Discuss

3D Morphable Models 3D face reconstruction Face recognition Lip synchronization

Topics to Discuss

3D Morphable Models 3D face reconstruction Face recognition Lip synchronization

3D Morphable Models

Statistical model of shape and texture Derived from laser scans

USF DARPA HumanID 3D Face Database Processing

Hole filling Surface smoothing Albedo estimation Dense correspondence

3D Morphable Models

-2 +2

…

-2 +2

…

Topics to Discuss

3D Morphable Models 3D face reconstruction Face recognition Lip synchronization

3D Face Reconstruction

Fitting the 3DMM to one or more images of the same face

•Scale•Rotation•Translation•Illumination

•Shape•Texture

?

?

Optimization problem with > 100 parameters

3D Face Reconstruction

Feature points Feature alignment Fitting result

3D Face Reconstruction

Dealing with Occlusions

Dealing with Occlusions

Without occlusionhandling

With occlusionhandling

3D Face Reconstruction

3D Face Reconstruction

3D Face Reconstruction

3D Face Reconstruction

3D Face Reconstruction

3D Face Reconstruction

Topics to Discuss

3D Morphable Models 3D face reconstruction Face recognition Lip synchronization

Face RecognitionFit the 3DMM to an image of an unknown face

•Scale•Rotation•Translation•Illumination

•Shape•Texture

Compare to database Recognition

?

?

Face Recognition

In controlled settings, almost perfect recognition is possible

Pose 1100.0%

Pose 2100.0%

Pose 3N/A

Pose 4100.0%

Pose 595.7%

Training view

Face Recognition

Uncontrolled environments are challenging Face orientation unknown Difficult illumination Facial expressions Occlusions Low resolution Motion blur …

Face Recognition

European parliament video: 21 persons, 86% correct

Face Recognition

VRT news broadcasts: 12 persons

Face Recognition

VRT news broadcasts: 12 persons

82.3% correctrecognition

Topics to Discuss

3D Morphable Models 3D face reconstruction Face recognition Lip synchronization

Lip Synchronization Speech driven animation Texture based, i.e. shape is fixed Strategy:

Extract 3D model of speaker’s face Track rigid motion of the face in video Extract texture for each frame Compute PCA model of texture Train ANN to link phonemes and PCA

coefficients (visemes)

System Overview

AutomaticAutomaticPhonePhone

RecognitionRecognition

NeuralNeuralNetworkNetwork

FaceFaceSynthesisSynthesis

SpeechSpeechfeature vectorsfeature vectors

FacialFacialfeature vectorsfeature vectors

Training Setup

AutomaticAutomaticPhonePhone

RecognitionRecognition

NeuralNeuralNetworkNetworkTrainingTraining

FaceFaceAnalysisAnalysis

SpeechSpeechfeature vectorsfeature vectors

FacialFacialfeature vectorsfeature vectors

Video Processing

3D face model acquisition Rigid motion tracking Normalized texture extraction Texture feature extraction (PCA)

Conclusion

3DMMs are a very powerful tool for face modeling

Many applications in computer vision and computer graphics