Realistic Performance-driven

Facial Animation

M. Sanchez, J. Edge and S. Maddock

Realism in Facial Animation

Traditional concept:

Not necessarily true… The human face is a highly complicated system of which we have

rather limited information to build a mechanical simulation We mainly perceive the visual effects of such processes, not the

mechanisms underneath

Physically-based approaches have issues of their own: Stability is difficult to assess, convergence impossible Most numerical methods scale up worse than linearly wrt the

complexity of the model The problem is connaturally stiff (linear elasticity >> resistance to

bend and shear), making it even more numerically sensitive

Conclusion: it’s questionable whether this concept of realism can lead to more plausible results than mere anatomically-driven geometric systems (eg. Waters)

realism

simulating the biomechanical processes that drive facial motion

Capturing Realism

One sensible alternative:

Intrinsic problems: Only a limited domain of information of actual facial motion is

accessible through capture (spatially discrete / timewise discrete) The captured data is bound to the physiognomy of the actual subject

Geometric deformation techniques can be used to extrapolate spatially discrete motion (eg. markers) to the full extent of facial skin

Can be consistent with large-scale properties of skin deformation Oversee fine detail aspects of skin mechanics (furrows / creases)

Retargeting of facial motion is possible using purely geometric criteria (without an anatomic model)

Small skin deformation can be modelled as a parallel process to large-scale motion

Capturing from actual performances the visual clues that make facial motion perceptually

plausible

System overview

System overview

System overview

System overview

System overview

System overview

1. Large-scale deformation

1.3. Animating the skin

1.3. Animating the skin

Large-scale aspects of skin mechanics to be reproduced:

Motion is propagated across the connectivity of the skin

Underlying layers of tissue dampen the extents of deformation

Local smoothness is generally preserved

Conventional techniques for spatial interpolation of motion do not fully satisfy these criteria

Point-based deformations (Williams, Pelachaud, Kshirsagar)

Volumetric techniques (Kalra, Escher) RBFs (Fidaleo)

Similar approaches could work if interpolation was performed along geodesic curves on the surface, but that would only work for 2-manifolds, and the skin has openings

1.3. Animating the skin

Through surface-to-surface mappings we can consistently interpolate motion across approximations of the mesh to be deformed

BIDS does precisely that… Motion propagates naturally across a

piecewise surface whose patch topology is compatible with that of the face model

Smoothness and locality of deformation can be asserted through the construction and conditioning of such approximating surface

For further details: “Realistic Performance-driven Facial

Animation using HW Acceleration” (submitted to ToG)

Thesis (soon to come ;)

1.2. Adapting facial motion

1.2. Adapting facial motion

The relation between source and target motion is highly non-linear, both with relation to its point of application and its extent and direction

We model this by constructing an RBF-based mapping between the embedding of source and target control surfaces…

capable of representing the two kinds of non-linearities

lacking of boundary separation issues due to the application of BIDS

stable under the usual scale of facial motion

For further details: “Use and re-use of facial MoCap” VVG’03

Different physiognomies Different scale of motion

Key fact:

Conventional MoCap retargeting cannot be applied to soft body deformation

1.1. Labelling the target mesh

Basic idea:

Our approach to this model consists on deforming the ref. mask to fit the target mesh, and then using the resulting control surface

this is solved as an iterative process, minimizing the following energy function:

the elastic energy terms correspond to the classic definition by Terzopoulos, and they have analytic solutions for Bezier Triangles

non-analytic terms require using blind methods (eg simplex downhill) or forward differences for their spatial derivatives

1.1. Labelling the target mesh

Transplanting with minimal distortion the labelling of the

ref. mask onto the target mesh

RR S

bsaverts

Tv dSBkSkAkcvvwc1

1. Large-scale deformation

2. Fine tissue detail

2. Fine tissue detail

2.a. Fitting a model to photographs

Motivation: A model of the face is needed to act as

support of the normal fields captured in 2.b

We need to analyse the ratio of deformation for each of the expressions in 2.c

We proceed by retrieving the position of markers on the face of the subject with reverse stereometry from 3 viewpoints…

and then deform the reference mask using BIDS:

2.a. Fitting a model to photographs

2.b. Shape from shading

Consider the simplified BDRF equation:

without the specular component, it would be linear…

use complementary light polarizers on lights and camera lenses, or

apply translucent makeup on the subject (cheap option)

With a set of 3 exposures to distinct lighting conditions, N is fully determined for any given pose (Rushmeier)

The variation of the normal field (N) is then encoded in the tangent space of the fitted mesh, producing normal maps

2.b. Shape from shading

nsdinout kkII NHNL

2.cd. Deriving a model of wrinkling

2.cd. Deriving a model of wrinkling

The fitted model in every expression not only acts as reference for normal maps, but also allows measuring the ratio of deformation on the face by approximating its strain tensor with that on the control surface

represents both magnitude and direction of infinitesimal strain

compression (negative strain) can be isolated by chopping off negative eigenvalues

Compression is ultimately the cause of facial tissue wrinkling, however…

elastic response is not the same in every region, neither in every direction of strain

linear compression is much more relevant than shearing

We have a sampling representing these properties (normal maps), together with the corresponding compression tensors, so we can build a model relating the two:

it’s a polynomial approximation of N wrt the 3 components of the symmetric form of the tensor

experimentation reveals linear degree suffixes

For any displaced configuration of the control surface, we can evaluate the model and compute the corresponding normal map

Results

Results

Question time…