Use and Re-use of Facial Motion Capture

M. Sanchez, J. Edge, S. King and S. Maddock

0. Motivation Facial Animation in the Computer Graphics industry

is a mainly human-driven process, requiring a lot of time and resources

Other aspects of Character Animation, such as Skeletal Animation, have been successfully automated by the use of human motion capture technology

Applying the same approach to Facial Animation could sensibly reduce the workload involved, and would lead to a corresponding increase in the realism of Facial Animation

However, the differences in the nature of the captured content requires the development of specific techniques to make Facial Motion Capture applicable

1. System overview

1.1 Facial Motion Capture System input:

Problems: It doesn’t analyze the motion

over the full geometry of the face (just at the markers);

The captured face may not correspond with the face to be animated;

Noise and missing data.

3D tracking of a predefined set of markers attached to the skin surface

2. Animating the skin

2. Animating the skin How to reconstruct the deformation

of the complete skin surface when only the movement of a few points is known?

Direct interpolation of the movement of the markers over the skin

(Kshirsagar et al. 00, Pasquariello and Pelachaud. 01)

Dirichlet Free Form Deformations (Escher et al. 98)

Radial Basis Functions (Fidaleo, Noh et al. 00)

We use Planar Bones (Sanchez and Maddock 03)

2.1. Planar Bones Extended formulation of Surface-oriented

Free Form Deformations(Kokkevis and Singh, 00)

Define a parameterisation of every vertex over a control mesh, used to drive a deformation

Preserve a “distance relation” between the control structure and the deformed geometry

Replicate proper transmission of motion across the skin without the need of surface metrics

4. Retargeting Facial Motion Capture

The dimensions of the face are different, and so is the scale of the motion;

Conventional full-body Motion Capture retargeting is not applicable;

The correspondence between different faces is highly non-linear.

4. Building a mapping between faces

4. Building a mapping between faces

Retargeting FMC requires:

1) Adapting the Planar Bones control mesh to the target geometry;

2) Scaling the motion of the markers according to the change of physiognomy.

Ideally, both processes should be performed automatically

In practical terms, we need some user input.

4.1. Fitting the control mesh

4.1. Fitting the control mesh 3 stages:

1) Radial Basis Functions -

produce initial approximation

2) Cylindrical projection –

Constraining the markers to the target surface

3) Mesh fitting – Blind constrained optimisation

Additional parameters of the Planar Bones method are also retargeted by this process:

Extents of the deformation (affection volume) Discontinuity maps

4.1. Fitting the control mesh RBF stage:

Build an interpolant of the offset between the markers labelled on the target face and their equivalents in the reference model

Evaluate this function at the non-hand-labelled markers to obtain their image on the target geometry

Mesh fitting stage: Finding the optimal distribution of

control points that:

a) Minimizes the “distance” between the reference face deformed by the retargeted control mesh and the target geometry

b) Preserves the general shape represented by a deformation energy function

c) Stays on the surface of the target face (enforced through the cylindrical mapping)

Simplex downhill method

4.2. Scaling Facial Motion

4.2. Scaling Facial Motion The two faces are labelled with

the same markers

After fitting the control mesh

We can extend this mapping to the whole space the faces are given in

By interpolating the initial displacement at every control point using RBFs

This interpolant is used to compute the mapping on the target space of the captured markers during the animation

4.2. Scaling Facial Motion

An evaluation the 2-norm of the metric tensor of the mapping shows how infinitesimal displacements are scaled

green: positive scaling (>1)

blue: negative scaling

The Planar Bones algorithm computes the final deformation, driven by the retargeted control mesh

This procedure implicitly scales the movement of the markers in the target space, according to the initial correspondence that is given as reference.

5. Processing Motion Capture Input

Limitations in the marker tracking technology lead to deficiencies in the captured data:

6. Sample results: lip tracking

6. Sample results: lip tracking

7. Conclusions and future work We have introduced a novel method for the retargeting

and animation of faces from motion capture data

Current research: Provide a better model for the tracking of the inner contour of the

lips: Marker-less image processing of the video capture Physical model using a mass-spring system attached to the outer

contour

Introduce furrowing and wrinkling in the skin animation A posteriori deformation analysis on the deformation induced by Planar

Bones

Questions?

m.sanchez @dcs.shef.ac.ukj.edge @dcs.shef.ac.uk

Additional samples: chorus