Simone Croci

1

Pose Space DeformationA unified Approach to Shape Interpolation and Skeleton-Driven

Deformation

J.P. Lewis Matt Cordner Nickson Fong

Presented by Simone Croci

Simone Croci

2

Talk Outline• Character Animation

OverviewProblem Statement

• BackgroundSkeleton-Subspace DeformationShape Interpolation

• Pose Space DeformationDeformation ModelEvaluationRelated Works

Simone Croci

3

Character Animation

OverviewAnimation:

“To give a soul to a lifeless character”

Simone Croci

4

Character Animation

Problem StatementMain Components:

1. Character Model

2. Deformation model

Simone Croci

5

Character Animation

Deformation Model

Complexity:∀ vertex: 3 DoFs

=> Reduction of DoFsx

y

z

Simone Croci

6

Character Animation

Deformation Model

Mapping:Parameters => Deformation

Simone Croci

7

Character Animation

Deformation ModelJoint rotations of a skeleton

Simone Croci

8

Character Animation

Deformation ModelEmotional Axes

X=Sad/Happy Y=Relaxed/ Excited

Simone Croci

9

Character Animation

Deformation ModelReduction of DoFs

ReducedDeformation Freedom

MovementDeficiencies

Lack of realism

Difficulty tocontrol the deformations

PathologicalDefects

Simone Croci

10

Character Animation

Deformation ModelReduction of DoFs

ReducedDeformation Freedom

solves these problems

retaining few DoFs

MovementDeficiencies

Lack of realism

Difficulty tocontrol the deformations

PathologicalDefects

Pose Space Deformation

Simone Croci

11

Skeleton-Subspace Deformation

…also called skinning, enveloping

Deformation ModelSkeleton => Vertex Position

Simone Croci

12

Skeleton-Subspace Deformation

Initial posey

x

v

v’’

v’1. Local

coordinatesv’,v’’

2. Vertex-joint weights: w`, w``

Simone Croci

13

Skeleton-Subspace Deformation

New posey

x

v

v = w` T’(v`)+ w`` T’’(v``)

v’v’’

T from local toworld coordinates

Simone Croci

14

Skeleton-Subspace Deformation

Pros• Simple• Smooth deformations• Low memory requirements• Real-time deformations

Simone Croci

15

Skeleton-Subspace Deformation

Cons• Indirect control on deformations through

weights

• Deformation Subspace is limited=> Pathological Defects (complex Joint

configurations, “Collapsing Elbow”)

Simone Croci

16

Skeleton-Subspace Deformation

Collapsing Elbow

Simone Croci

17

Skeleton-Subspace Deformation

Frame 1 Frame 2

Initial pose

Simone Croci

18

Skeleton-Subspace Deformation

Frame 1

New pose

Frame 2

Simone Croci

19

Skeleton-Subspace Deformation

Cons• Indirect control on deformations through weights

PSD: direct sculpting

• Deformation Subspace is limited=> Pathological Defects (complex Joints, “Collapsing Elbow”)

PSD: interpolation of a vast range of deformations

Simone Croci

20

Shape Interpolation

∑=k

kkSwS

also called shape blending, multi-target morphing

AlgorithmInput: key shapes Sk

(same topology)

SuperpositionAnger Disgust Fear

Joy Sadness Surprise

Simone Croci

21

Shape Interpolation

neutral smirk

+ =>

half smirk

Simone Croci

22

Shape Interpolation

smirk

+ )=½(half smirkneutral

Simone Croci

23

Shape Interpolation

smirk

+ )=½(half smirkneutral

Simone Croci

24

Tony de Peltrie1985

Simone Croci

25

Shape Interpolation

Pros• Direct manipulation of desired expressions• Skin deformation for facial animation

(Used in the movies)

Simone Croci

26

Shape Interpolation

Cons• Not suited for skeleton-based deformations

Simone Croci

27

Shape Interpolation

Cons• Not suited for skeleton-based deformations• Storage expensive

Simone Croci

28

Shape Interpolation

Cons• Not suited for skeleton-based deformations• Storage expensive• Conflicting Key Shapes

Simone Croci

29

Shape Interpolation

Conflicting Key Shapes

Key Shapes are not independent

Superpositiona KSHappy+ b KSSurprise

Simone Croci

30

Shape Interpolation

Conflicting Key Shapes

Key Shapes are not independent

Superpositiona KSHappy+ b KSSurprise + c KSFear

Simone Croci

31

Shape Interpolation

Conflicting Key Shapes

Key Shapes are not independent

Superpositiona’ KSHappy+ b’ KSSurprise + c KSFear

Simone Croci

32

Shape Interpolation

Conflicting Key Shapes

Key Shapes are not independent

Superpositiona’ KSHappy+ b’ KSSurprise + c KSFear

PSD: interpolation between key Shapes

Simone Croci

33

Shape Interpolation

Cons• Not suited for skeleton-based deformations• Storage expensive• Conflicting Key Shapes• Key Shape <=> Animation Parameter

Simone Croci

34

Shape Interpolation

Key Shape Anim. Parameter

If new expression is needed=> New Key shape & New Parameter

∑=k

kkSwS

Simone Croci

35

Shape Interpolation

Key Shape Anim. Parameter

If new expression is needed=> New Key shape & New Parameter

PSD: no additional parameter

freedom in the design of parameter space

∑=k

kkSwS

Simone Croci

36

Comparison TablePose Space Deformation

Low requirementsHigh requirementsMemory

YesNot alwaysSmoothness of Interpolation

NoYesDirect manipulation

Real-timeReal-timePerformance

Body (skeleton-influenced) Deformation

Facial Deformation

Field of application

Skeleton-subspace Deformation

Shape Interpolation

Simone Croci

37

Comparison Table

Real-time

Pose Space Deformation

Low requirementsHigh requirementsMemory

YesNot alwaysSmoothness of Interpolation

NoYesDirect manipulation

Real-timeReal-timePerformance

Body (skeleton-influenced) Deformation

Facial Deformation

Field of application

Skeleton-subspace Deformation

Shape Interpolation

Simone Croci

38

Comparison Table

Low requirements

Real-time

Pose Space Deformation

Low requirementsHigh requirementsMemory

YesNot alwaysSmoothness of Interpolation

NoYesDirect manipulation

Real-timeReal-timePerformance

Body (skeleton-influenced) Deformation

Facial Deformation

Field of application

Skeleton-subspace Deformation

Shape Interpolation

Simone Croci

39

Comparison Table

If needed

Low requirements

Real-time

Pose Space Deformation

Low requirementsHigh requirementsMemory

YesNot alwaysSmoothness of Interpolation

NoYesDirect manipulation

Real-timeReal-timePerformance

Body (skeleton-influenced) Deformation

Facial Deformation

Field of application

Skeleton-subspace Deformation

Shape Interpolation

Simone Croci

40

Comparison Table

Yes

If needed

Low requirements

Real-time

Pose Space Deformation

Low requirementsHigh requirementsMemory

YesNot alwaysSmoothness of Interpolation

NoYesDirect manipulation

Real-timeReal-timePerformance

Body (skeleton-influenced) Deformation

Facial Deformation

Field of application

Skeleton-subspace Deformation

Shape Interpolation

Simone Croci

41

Comparison Table

Facial and Body Deformations

Yes

If needed

Low requirements

Real-time

Pose Space Deformation

Low requirementsHigh requirementsMemory

YesNot alwaysSmoothness of Interpolation

NoYesDirect manipulation

Real-timeReal-timePerformance

Body (skeleton-influenced) Deformation

Facial Deformation

Field of application

Skeleton-subspace Deformation

Shape Interpolation

Simone Croci

42

Character Animation

Deformation Model

Mapping:Parameters => Deformation

Simone Croci

43

Character Animation

Deformation Model

Mapping:Parameters => Deformation

Interpolat from deformation examples

Pose Space Deformation

Simone Croci

44

Character Animation

Deformation Model

Mapping:Parameters => DeformationPose Space => Displacements Δx

Pose Space Deformation

Interpolat from deformation examples

Simone Croci

45

Pose Space Deformation

Face Animation1. Pose Space

Ex: Emotion Space

2. Initial Face Model

3. Sculpting of exampleswith Pose Space Coordinates[Δx: between initial model and examples]

4. Interpolant

Simone Croci

46

Pose Space Deformation

Face Animation

Simone Croci

47

Pose Space Deformation

Body Animation1. Pose Space

Ex: Skeleton Parameters

2. Skeleton & Model

3. Sculpting of exampleswith Pose Space Coordinates[Δx: between SSD and examples]

4. Interpolant

SSD PSD

Simone Croci

48

Pose Space Deformation

Body Animation

Simone Croci

49

Interpolat from ExamplesInterpolant:

Pose Space => Displacements Δxfrom {(PSCoord1, Δx1), ..., (PSCoordN, ΔxN)}

Scattered Data Interpolation:Interpolation of a set of irregularly located data points

Approaches:- Shepard’s Method- Radial Basis Function Interpolation

Simone Croci

50

Shepard’s Method

are the given data points at

0,)( >−= − pw pkk xxx

∑

∑

=

== N

kk

N

kkk

w

dwd

1

1

)(

)()(ˆ

x

xx

Ndd ,,1 K Nxx ,,1 K

Simone Croci

51

Weight function

Simone Croci

52

Flat zones

Simone Croci

53

Average at far points

Simone Croci

54

Shepard’s Method

Properties:• Singular at data points ( )• Infinitely differentiable except at data points• Partial derivatives are zero at data points

=> flat zones around data points• Far from data points converges to the

average value

kx

N

d

w

dwd

N

kk

N

kk

N

kkk ∑

∑

∑=

=

= =∞

∞=∞ 1

1

1

)(

)()(ˆ

0with )( >−= − pw pkk xxx

kx

Simone Croci

55

Radial Basis Functions…also called Hardy’s multiquadratics

from at solving a system of linear equations:

( )∑=

=N

kkkwd

1-)(ˆ xxx ψ

Nww ,,1 K Ndd ,,1 K Nxx ,,1 K

( ) ( )

( ) ( ) ⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛=

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

NNNN

Nk

N w

w

d

dM

L

MOM

L

M1

1

111

--

--

xxxx

xxxx

ψψ

ψψ

wd Ψ=

Simone Croci

56

Choice of Radial Basis Function

( ) rr =ψ

kr xx -=

( ) 0,)( 22 >+= − ασψ αrr

( ) )ln(2 rrr =ψ

( ) 10,)( 22 <<+= βσψ βrr

( ) 3rr =ψ

localized

Thin-plate spline function

linear

cubic

Simone Croci

57

Gaussian Radial Basis

=>

( ) ⎟⎟⎠

⎞⎜⎜⎝

⎛ −= 2

2

2exp

σψ rr

kr xx -=

∑= ⎟

⎟

⎠

⎞

⎜⎜

⎝

⎛ −=

N

k

kkwd

12

2

2-

exp)(ˆσxx

x

Simone Croci

58

Gaussian Basis Function

Simone Croci

59

Gaussian Radial Basis

Simone Croci

60

Gaussian Radial Basis

Properties:- smooth- localized:

=> width of falloff is adjustable- relatively fast to compute (table)

( ) ∞→= rr for0ψσ

( ) ⎟⎟⎠

⎞⎜⎜⎝

⎛ −= 2

2

2exp

σψ rr

Simone Croci

61

Pose Space Deformation[Summary]

∑=

=N

kkwd

1)()(ˆ xx ψ

Deformation model:Interpolant:

Pose Space => Displacements Δx

from {(PSCoord1, Δx1), ..., (PSCoordN, ΔxN)}

Simone Croci

62

Workflow

l

Simone Croci

63

PerformanceFor N poses:

• Preprocessing phase:- NxN Matrix must be inverted- Matrix-vector multiplication

(∀ component of ∀ displaced vertex)

• Animation phase:- Interpolated table lookup

wd Ψ=

dw 1−Ψ=Ψ

( )∑=

=N

kkkwd

1-)(ˆ xxx ψ

( ) ( )

( ) ( )⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛=Ψ

NNN

Nk

xxxx

xxxx

--

--

1

11

ψψ

ψψ

L

MOM

L

Simone Croci

64

Memory Requirements

For N poses:Every vertex stores Nx3 weights

(N weights ∀ component of a vertex)N pose space coordinates of ex.

( )∑=

=N

kkkwd

1-)(ˆ xxx ψ

Simone Croci

65

Memory Requirements

For N poses:Every vertex stores Nx3 weights

(N weights ∀ component of a vertex)N pose space coordinates of ex.

( )∑=

=N

kkkwd

1-)(ˆ xxx ψ

Simone Croci

66

Memory Requirements

For N poses:Every vertex stores Nx3 weights

(N weights ∀ component of a vertex)N pose space coordinates of ex.

( )∑=

=N

kkkwd

1-)(ˆ xxx ψ

Simone Croci

67

Pros & ConsPros:• Wide range of deformations• Arbitrary Pose Space Axes• Real-time synthesis • Relatively simple to implement• Control on interpolation ( )• Direct manipulation

(Iterative layered refinement)

σ

Simone Croci

68

Pros & Cons

Cons:• Accuracy is reliant on the modeler/animator• is manually tuned• Performance• Memory requirements

σ

Simone Croci

69

Critiques

• Least Square Problem ?with regular

• If N poses then 3 NxN matrices must be inverted for each control vertex?

Only one NxN matrix must be inverted• Close coordinates in PS results in a

numerically unstable matrix (regularization, TSVD)

dΨw 1−=( ) dΨΨΨw TT 1−

=Ψ

Ψ

Ψ

Simone Croci

70

Related Papers

Shape from ExamplesSloan, Rose, Cohen (I3D conference 2001)

EigenSkinKry, James, Pai (SIGGRAPH 2002)

Skinning Mesh AnimationJames, Twigg (SIGGRAPH 2005)

Simone Croci

71

Shape by ExamplesSloan, Rose, Cohen

• Paradigm: Design by Examples• Interpolation in Lagrangian form• Radial Basis Functions (B-Splines) and Polynomials• Interpolation & Extrapolation• Reparameterization• Applied also to Textures• Preprocessing phase:

– one linear system per example• Animation phase: twice faster

Simone Croci

72

Shape by ExamplesSloan, Rose, Cohen

Simone Croci

73

EigenSkinKry, James, Pai

• Paradigm: Design by Examples• Articulated characters• Deformation field for each Joint support

– EigenDeformations (Deformation basis)• Mapping: Joint => Eigendeformation coordinates

– 1-D interpolation with Radial Basis Functions– No non-linear joint-joint coupling effects

• Graphical Hardware Optimization– Reduction of Memory Requirements– Real-time rendering

Simone Croci

74

EigenSkinKry, James, Pai

Simone Croci

75

Skinning Mesh AnimationJames, Twigg

• Approximation of sequence of input meshes– No Pose Space

• Skinning algorithm:– Proxy bone Transformations (+ weights)

automatically approximated– Displacement field in rest pose (TSVD)

• Real-time rendering (Graphical HW support)

... more to come

Simone Croci

76

Question Time