Motivation - UMR 5205 CNRS / INSA Lyon / Université ...6/8/2011 1 Laboratoire d'InfoRmatiqueen...
Transcript of Motivation - UMR 5205 CNRS / INSA Lyon / Université ...6/8/2011 1 Laboratoire d'InfoRmatiqueen...
6/8/2011
1
Laboratoire d'InfoRmatique en Image et Systèmes d'information
UMR 5205 CNRS/INSA de Lyon/Université Claude Bernard Lyon
1/Université Lumière Lyon 2/Ecole Centrale de Lyon
http://liris.cnrs.fr
Ludovic Dutreve Alexandre Meyer Saïda Bouakaz
Université Lyon 1
Easy Acquisition and Real-Time Animation of Facial Wrinkles
Motivation
Facial Animation
o Realism of a virtual character
Real-time applications
o Video games
o Interactive systems
Issues
o High level human analysis
o Variety of deformations
o Tissues/muscles complexity
Fine details
o Wrinkles, bulges, …
o Improve expression recognition [Martin09]
May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 2
Related work: capture
Facial motion capture
o Never simple
• Structured light [Zhang04, Na04, …]
• Stereo with often several pair of cameras [Bickel07,…, Beeler10, …]
• Sphere of cameras/polarised lights (Emily project) [Debevec etal 09]
o Hard to apply only the details on other faces
• Capture the face and details in the same huge mesh
May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 3
[Beeler2010]
[Zhang2004]
[Bickel 07][Debevec etal 09]
Related work: synthesis
Dynamic skin effects (wrinkles)
o Hard to get real-time, control, few manual work and realism
• Procedural modeling [Bando02]
• Simulation [Sifakis05], compression [Volino99,Larboulette04]
• Synthesis from examples: wrinkle maps [Oat07]
�Need simple details acquisition methods compatible with real
time animation system
May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 4
[Sifakis 05][Bando 02] [Larboulette 04]
Overview
Footer
…
Session of capture (preprocessing)
Compute the
normal maps
Bones of
the frame i
Generate the normal map
using the reference maps
according to the bones positions
Set of reference poses
…
Reference
pose 0
Reference
pose 1
Reference
pose N
Easy Acquisition and Real-Time Animation of Facial Wrinkles May 2011
Winkles capture system (preprocessing)
1 camera with flash�photos of several expressions
N reference poses (= bones position + normal map)
8 June 2011 6
…Session of capture
Compute the
normal maps
(preprocessing)
…
Reference pose 0
(neutral)
Reference pose 1
Reference pose N
Easy Acquisition and Real-Time Animation of Facial Wrinkles
6/8/2011
2
Winkles capture system (preprocessing)
May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 7
From the photos to the normal maps
Neutral expressionExpression 1
Expression 1 deformed
by RBF to match the
neutral expression
Image ratio Normal map 1
mask
Landmarks (markers
or manually set)
Wrinkles capture: shape from shading
For each pixel
o Flash of the camera � L=(0,01)
o Ratio and assumption 1: each region = diffuse plane
� Declination of N
o Assumption 2: orientation of the gradient of the image ratio
~ orientation of the normal of the surface
� Orientation of N
May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 8
Image ratio Normal map 1
Wrinkles capture: results
May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 9
Image ratio
Normal map
Photos
During the animation: real-time wrinkles
Reference Poses = position of bones + wrinkles map (normal map)
o Position of bones� large-scale deformations
o Wrinkles map � small-scale deformations
May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 10
Input dataNeutral Reference Pose 0 Reference Poses Detail Maps
Runtime
Reference Pose 1 Reference Pose 2
During the animation: real-time wrinkles
May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 11
Input data
Pose EvaluationRuntimeArbitrary Pose
Neutral Reference Pose 0 Reference PosesReference Pose 1 Detail MapsReference Pose 2
During the animation: real-time wrinkles
May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 12
Input data
Arbitrary Pose New Map RenderingRefPose InfluencesPose Evaluation Runtime
Neutral Reference Pose 0 Reference PosesReference Pose 1 Detail MapsReference Pose 2
6/8/2011
3
Real-time wrinkles: influences at pixel level
To generate a new wrinkle map, needo Reference poses influences at a pixel level
From the bones level to the pixel levelo (1)Pose Evaluation Reference Poses Influences � for each bone
o (2)Skinning Bones � Vertices
o (3)Rasterization Vertices � Pixels
May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 13
Reference Poses Influences� Each pixel
α2
α1
α0
Bone i V2
V1
V0
…
Vn
(2)Skinning(1)Pose Evaluation (3)Rasterization
Real-time wrinkles: pose evaluation
During the animation, normal map generation
o For each bone b, compute the influences Ib of each reference pose i
• Orthogonal projection,
� Ib={ Ib0=c/r, …, Ib
i, …}
o For each vertices
• Use the weight of skinning to compute
the reference poses influences
• Vertex program (GPU)
o For each pixel
• Use barycentric interpolation
• Fragment program (GPU)
May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 14
Bone position of the
reference pose i
Bone position of
the neutral
reference pose 0
Current bone
positionc
r
Real-time wrinkles: non linear function
Introduce a non linear term in the normal map generation
to add control on wrinkles apparition
o For each bone b, compute the influences
Ib of each reference pose I
� Ib={ Ib0=F(c/r), …, Ib
i, …}
May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 15
Bone position of the
reference pose i
Bone position of
the neutral
reference pose 0
Current bone
positionc
r
Results : capture and real time synthesis
Tangent space normal map
Linear Blend Skinning
Reference poses
o Rest pose
o Compressed expression
o Stretched expression
May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 16
Examples of normal map captured
and applied to a 3D model Local influence
Results - video
May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 17
Conclusion
Capture• Easy to setup with a simple camera
• Few seconds to compute the normal maps (preprocessing)
Real-Time wrinkles generation• GPU implementation > 100fps
Easy to add to an existing implementation• Widely used techniques (Skinning, Bump-Mapping)
• No additional rendering pass or pipeline modification
Future worko Improve capture/shape from shading assumptions
o Data comparison with stereo-based capture
o Temporal normal maps capture
May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 18
6/8/2011
4
Questions ?
May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 19
Results - video
May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 20
24 mars 2011 21
Results
Pose neutre Pose référence 1 Pose référence 2
Overview of our system(2): animation
Online: during animation for frame i
May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 22
Bones of the frame i
Generate the normal map
using the reference maps
according to the bones positions
Set of reference poses
…
Reference
pose 0
Reference
pose 1
Reference
pose N
Results : real time synthesis
May 2011 Easy Acquisition and Real-Time Animation of Facial Wrinkles 23
3 references poses:
neutral, compressed
and stretched
Détails fins dynamiques
Acquisition
Adapter la carte de normale obtenue au personnage 3Do Espace photo. � Espace texture
• Points repères + Interpolation
�Carte de normale utilisable en 3D (Bump-Mapping)
24 mars 2011 24
Image ratio
Carte de normales
Adaptation